ub[3]rgeek.net

Apple’s Time Machine from a Storage Guy’s perspective

Background:

As a technology guy I’m always interested in how things work under the covers. I recently switched to a MacBook Pro because frankly I was tired of chasing the dragon with Linux on my workstation and I wanted something that Just Worked(tm). One of the very first things I did was get Apple’s Time Machine working. Like many others I decided to not go down the Time Capsule route, because I already have a very nice NAS (3x2T RAID-5 running Debian 6.0 and several virtual machines) and a very nice Wireless Access Point and frankly I don’t need to throw that all out just to get a Time Capsule . Using one of the many sites out there I was able to get netatalk running and all the magical Bonjour services advertised to get the MacBook Pro to believe my Linux box was in fact a valid Time Machine backup destination. Afterwards I started poking around inside Time Machine to see what made it tick.

Time Machine Internals:

Time Machine leverages a technology similar to snapshots. It keeps incremental backups of your filesystem inside a sparse filesystem image on an AFP share that the pretty 3D animated browser lets you navigate through to perform restores. If you mount the disk image outside of Time Machine you will see the various ‘snapshots’ contained in the root directory arranged by date and time of the backup. Underneath the covers the files are stored in 8 MB chunks which are created and deleted as space is required or freed. The ‘snapshots’ are file-based, so if you change a part of a file the entire file is re-transferred. The transfer is triggered by a change in the file’s mtime (UNIX for Modification TIME) attribute so if you were to say take an ISO image and use the UNIX ‘touch(1)‘ command to update the mtime to ‘now’ the file will be re-transferred next time Time Machine runs even though the entire file is identical to the already backed-up copy.

For the skeptical:

aramaki@15:11:03 ~/Downloads (535)$ md5 SW_DVD5_Office_Mac_Standard_2011_English_MLF_X16-99088.ISO
MD5 (SW_DVD5_Office_Mac_Standard_2011_English_MLF_X16-99088.ISO) = cd180db30b9851da6a55545457bc36aa

aramaki@15:11:22 ~/Downloads (536)$ touch SW_DVD5_Office_Mac_Standard_2011_English_MLF_X16-99088.ISO

aramaki@15:11:28 ~/Downloads (537)$ md5 SW_DVD5_Office_Mac_Standard_2011_English_MLF_X16-99088.ISO
MD5 (SW_DVD5_Office_Mac_Standard_2011_English_MLF_X16-99088.ISO) = cd180db30b9851da6a55545457bc36aa


As you can see the mtime has been updated:

aramaki@15:14:14 ~/Downloads (541)$ stat -f "%m" SW_DVD5_Office_Mac_Standard_2011_English_MLF_X16-99088.ISO
1335366688

aramaki@15:14:25 ~/Downloads (542)$ date -r 1335366688
Wed Apr 25 15:11:28 UTC 2012


And there are no extended HFS+ attribs on the file:

aramaki@15:14:33 ~/Downloads (543)$ ls -lO SW_DVD5_Office_Mac_Standard_2011_English_MLF_X16-99088.ISO
-rw-rw-r-- 1 mernisse staff - 972087296 Apr 25 15:11 SW_DVD5_Office_Mac_Standard_2011_English_MLF_X16-99088.ISO


And yet, Time Machine is currently transferring it as if it was changed.

The other caveat to the architecture is that if you (like me) use File Vault 2 on Lion, the backup you make via Time Machine will be unencrypted. So you will want to take precautions to secure your network (wired and wireless) as well as physically secure the device upon which you store your Time Machine backups. In the case of using a Linux server to emulate a Time Capsule (as I am doing) eCryptFS or dm-crypt can provide security for the storage backing the Time Machine volume.

Disaster Recovery:

Time Machine provides a nice way to easily backup your Macintosh workstation. This is probably enough for most people but for the more paranoid of us the obvious flaw is that it provides no disaster recovery options. If your house burns down, or a power surge blows up your electronics you are completely sunk. This is where using a Linux server in lieu of a Time Capsule really shines.

Now because the disk image is comprised of 8MB chunks, you could simply use a file-based replication technology like rsync or duplicity to mirror the Time Machine destination volume offsite. The drawback of course being that your minimum transfer is going to be 8MB/hr and will likely include a LOT of duplicate data.

The alternative is to utilize something like NetApp‘s OSSV technology to make de-duplicated, block-level transfers of the Time Machine volume offsite. This would be much more efficient over the WAN as it will overcome the file-based snapshot technology limitation of the underlying Time Machine engine and will provide a much higher storage efficiency in the destination volume.

Since I already use OSSV to protect my virtual machine infrastructure here at home and in the colocation facility that I host my public-facing IT gear at, I intend (when I find the free time…) to test using OSSV to protect my Time Machine volume and get some real world numbers to show the bandwidth and storage reduction involved. My impression after looking under the covers a bit is that it will likely be significant.

Summary:

Time Machine is a good technology. It truly makes backups easy for the general computing public and I highly recommend it to any Macintosh user (though I imagine, given what I know about the slightly-fanatical nature of the Macintosh user base I don’t have to). I wouldn’t be surprised if we see Apple offering an iCloud bolt-on service in the near future that will allow you to replicate all, or part of your Time Capsule protected data off to some centralized Apple cloud storage environment (hopefully on NetApp storage) but until that happens, and for those of us who rolled our own Time Machine backup destinations it seems to make sense to roll our own DR protection methods.

NFSv4 lessons

I recently have been re-architecting my network at home and one of the big transitions has been back to NFS home directories across my Linux systems. I previously used NFSv3 shares to provide this functionality years ago but now that modern Linux systems (I am running mostly Debian 6.0 (squeeze)) support the more secure, performant and robust NFS version 4 I standardized on that. Thanks to my use of nss_ldap for user authentication it has been pretty straightforward but there have been a couple of interesting gotchas.

I have a NetApp FAS2020 which I am currently using to re-arrange my VM server’s RAID array and have been syncing data to it using rsync(8) to a NFSv4 share on the filer. One of the interesting things is that NFSv4 no longer uses numeric UIDs/GIDs and instead transmits the textual strings and relies on the host’s getpwnam() functions to translate it into a numeric value for the underlying file system. Thankfully Data ONTAP has the ability to use LDAP in a similar fashion to modern Linux hosts[1].

I am running Data ONTAP 7.3.6 and my LDAP configuration looks like this:

toaster*> options ldap
ldap.ADdomain
ldap.base dc=ub3rgeek,dc=net
ldap.base.group ou=groups,dc=ub3rgeek,dc=net
ldap.base.netgroup
ldap.base.passwd ou=users,dc=ub3rgeek,dc=net
ldap.enable on
ldap.minimum_bind_level anonymous
ldap.name xxxxxx
ldap.nssmap.attribute.gecos gecos
ldap.nssmap.attribute.gidNumber gidNumber
ldap.nssmap.attribute.groupname cn
ldap.nssmap.attribute.homeDirectory homeDirectory
ldap.nssmap.attribute.loginShell loginShell
ldap.nssmap.attribute.memberNisNetgroup memberNisNetgroup
ldap.nssmap.attribute.memberUid memberUid
ldap.nssmap.attribute.netgroupname cn
ldap.nssmap.attribute.nisNetgroupTriple nisNetgroupTriple
ldap.nssmap.attribute.uid uid
ldap.nssmap.attribute.uidNumber uidNumber
ldap.nssmap.attribute.userPassword userPassword
ldap.nssmap.objectClass.nisNetgroup nisNetgroup
ldap.nssmap.objectClass.posixAccount posixAccount
ldap.nssmap.objectClass.posixGroup posixGroup
ldap.passwd ******
ldap.port 389
ldap.servers ldap.internal.ub3rgeek.net
ldap.servers.preferred
ldap.ssl.enable off
ldap.timeout 20
ldap.usermap.attribute.unixaccount unixaccount
ldap.usermap.attribute.windowsaccount windowsaccount
ldap.usermap.base
ldap.usermap.enable on


This basically mirrors my nss_ldap configuration and has allowed me to easily copy my RAID array over to the filer so I can wipe and rebuild the underlying RAID layer.

NFSv4 Reference:
RFC-3530
NetApp TR 3085

[1] http://media.netapp.com/documents/tr-3464.pdf

Some interesting lessons…

Unsurprisingly starting a new job has a way of rearranging the priorities in life. One of the more unexpected ways that my perspective changed was in my personal IT infrastructure. I previously relied fairly heavily on work provided resources for development and testing environments, but my new office does not provide the same level of resources as I previously enjoyed. In response to this I have found myself creating flexible infrastructures and frameworks at home to fill that void. I spent a fair bit of time recently implementing a fully virtualized computing infrastructure to support the ever changing demands that pre-sales consulting conjures up. One of the more interesting side effects of that is finally sorting out a reasonable backup strategy for most of my critical data. Being able to easily backup virtual machine images is a huge boon, and really an under appreciated benifit of virtualization.

The other big change has been my further move towards mobile computing. I am actually writing this on my BlackBerry PlayBook while enjoying the delightful weather and watching people come home after a long day celebrating my heritage ;) My fixed computing has been relegated to recreational tasks and only the most serious work related functions. I think more so than any advance since the Internet itself, mobile computing technology is going to drive the future of communications and human interaction. The cloud model may be a big buzzword in marketing these days but for me it is nothing more than an optimization of the already decades old client server model, whereas the platform and location agnostic nature of mobile computing is truly a revolutionary movement.

I still recall how liberating it was the first time I got a BlackBerry smartphone and was able to be effective at working not only while out of the office, but on the other side of the planet. The decoupling of location from the ability to do ones job is worth so much more, not only to businesses but to the individuals who get to enjoy life unchained from a desk.

Now if only I had the freedom to fly more… That would be something.

Sixteen hours, frustration starts to set in.

Thinking back at the first 12 or 13 hours of my flight training I have to say it was mostly an experience of pretty constant enjoyment. My instructor has a mission based syllabus so all the lessons up until now have been go somewhere do something missions which I really enjoied. It means I got to go through the process of a typical flight: plan, brief, pre-flight, and execute the mission. Going somewhere (usually a nearby airport or landmark) means I got to go through the phases of a typical flight (departure, enroute/cruise, arrival) and the differing work loads associated with them. I’d exersize my skills (such as they are at this point) in various tasks such as pilotage (navigating by looking at a map and out the window), radio communication (not with ATC yet, but CTAF certainly), straight and level flight, airport operations, etc…

The last several lessons however are leading up to my first solo which is arguably the most important mile stone in a pilot’s training for any class of certificate. As such the concentration has been on me executing take off, traffic pattern and landing successfully and if I’m honest I’m really starting to struggle and get frustrated by it. I don’t know what the root cause is at the moment, part of it is trouble maintaining alignment with the center-line of the runway, which sounds easy and trivial (especially with a 32′ wide plane on a 60′ wide runway that is almost 3/4 of a mile long) but I can assure you it is not. So it kinda works out that I get overwhelemed, frustrated, and end up over-controlling the plane…

I can drive my car with two fingers, or a thumb simply hooked on the side of the steering wheel and maintain position within a hand-full of inches at 65MPH with ease. I can do the same in a boat, but put me on final towards a runway and I’m ham-fisting the aircraft, complete with sweaty palms and 180bpm heart-rate. Time, which is supposed to be a constant seems to accelerate and all the tasks I need to complete blur together… did I make the radio call for the turn to final, am I high, flaps, air-speed, throttle, descent rate… whereas I can drive a car at over a hundred miles per hour, in traffic, with barely more than a resting heart rate, doing 80MPH in this plane is a whole different story.

And I think that’s part of the frustration. I’m competent in activities that seem similar at least when measured by speed or requirements of precision, but this is still difficult. I am sure that time and practice will bring the confidence and calm that I’m used to feeling in other activities to this and I tihnk admitting this to myself is certainly going to help but at least for the time being… it’s frustrating and difficult.

And deep down inside, that’s part of why it is so great. Things worth doing shouldn’t be too easy.

Trying out Ubuntu 11.04 Beta 1

I just wanted to jot down a couple of first impressions about Ubuntu 11.04 Beta 1, as that seems to be sort of the en vogue thing to do these days.

A brief summary of my configuration:
Lenovo Thinkpad T500
Intel Core 2 Duo P8700 @ 2.53GHz
8GB 1066MHz DDR SDRAM
Intel 5100AGN Wireless
Intel GMA x4500 video (ATI graphics disabled for battery life issues)
WD 160G 5400RPM drive, with dm-crypt /home partition

To start with, I upgraded my 10.10 installation by letting update-manager -d do its thing. The process was mostly painless (a couple of customized config files needed manual intervention but that’s to be expected) and took a little over an hour to complete. Upon reboot the system was up and running without any issues which is a testament to the continued work of the Ubuntu developer community. It continues to astound me how trouble-free the various betas have been at least from the dist-upgrade front.

I’ve been using the new Unity interface for a day or two now and while I’m happy that I don’t hate it as much as I did in the alphas.

The Good:
It is fast and stable at the moment. I think compiz is a little more power-hungry on my laptop than metacity was, but not enough that it is setting my lap on fire. At least from an upgrade perspective I have not found anything that doesn’t work that used to (I maintain a local apt repository of various changes and customizations for my systems, nothing that I had working in 10.10 broke in 11.04 beta). The work on the indicators has been very good for the most part, the Network Manager applet is quite a bit better than the previous versions.

The Bad:
Compared to the new Gnome 3 interface Unity seems a little clunky, probably due to the netbook heratige of the design though I am glad that I still have the minimize buttons. The launcher/dock/whatever thing doesn’t always hide, so it sits there taking up space on screen for no good reason and you can’t move it from the left hand side which is not where I’d put it if I had the option. The Apple-like menu bar is a little irritating, but really only because I think it is highly derivative and I am sad that we couldn’t come up with something better from a design standpoint. I also find the dash to be annoying to use but I don’t really know why. Finally I’m not sure why the new date/time indicator had to lose the weather. I travel a lot and had many different locations defined in the previous applet and having an at-a-glance view of the weather was a nice feature to have.

Conclusion:
I was really worried that I’d either have to skip this release or continue using the Gnome 2 panel interface after looking at the first couple of alphas but overall am pretty happy. I will try Gnome 3 when (if?) it comes to Ubuntu and see how it stacks up. I think it’s a prettier and cleaner interface but only time will tell which I like more. The amount of work that the Ubuntu community put into this release is pretty amazing and the fact that it landed in as good of shape as it did is a really huge accomplishment.

Obligatory screenshot:
Ubuntu 11.04 Beta 1

Update, 18/April/2011 It would be really nice if they would fix this bug before shipping 11.04.

途中に日本に! (I am) Enroute to Japan!

So, if all this technology worked right this should actually post as I am leaving ROC for ORD, my only stop enroute to NRT (Tokyo, Japan). Over the last few weeks people have been asking me if I am still going to Japan on holiday and my answer has been an unwavering yes. Granted things looked a little sketchy for a bit there, the trains from Narita into Tokyo proper were running at reduced rates (and even briefly closed) and I admit I was not really liking the idea of tossing down 23,000円-25,000円 ($280 – $310) right out the gate for a taxi, or rolling around in a bus for 2 to 3 hours after flying for 20 but if that was the price I was going to have to pay I was willing to pay it. Once the trains resumed service last week I felt confident that the trip would be problem-free and the anxiety was left behind. I have full faith in both the Japanese government, people and the international group of scientists that are working on the problems at the Fukushima Daiichi nuclear power plant as well as the relief workers bringing aid to the victims of the massive earthquake and tsunami. Along with donating to the Red Cross, bringing in a little tourism money and showing some faith is about the best I can come up with to do to help.

I enjoy international travel. I think that the more of this world that we can see the better off we are as people and the better we come to understand ourselves. This is certainly not my first time out of the country nor even off the continent but I admit this trip has a special place in my heart because I expect the sheer weight of the culture shock to be unlike anything I’ve ever felt before. I have been to the UK and Australia and as a native of a former British colony it really wasn’t all that different. Yes I had to drive on the other side of the car and road, yes the slang was different, and yes the money was funny colors and had pictures of the Monarchy on it but for all intents in purposes we all shared a common cultural liniage. Thursday evening (Japan Time) when I step out of the airplane I will be for the first time completely out of my element and I am excited beyond words to experience that for the first time.

Surfing Safely, from somewhere in the Universe,
Matt

I finally wrote my first BlackBerry (handheld) App

I spent a couple days over the last couple weeks working on a BlackBerry WebWorks application. The BlackBerry WebWorks API is basically a HTML/JavaScript application compiled into a Java application for use on the handheld. It has access to a fairly large set of the RIM APIs in the device and as I am not interested in learning Java, is about the only option I have.

As a primarily hobbiest developer I found the process to be fairly straightforward, though unfortunatly extremely Windows-centric. The primary debug workflow involves Eclipse and the BlackBerry device simulators, the latter being Windows-only. Trying to debug without the sim is a non-starter as the JavaScript engine will not throw any useful error messages while running live on a device. It is not wholly awful but I really do prefer avoiding Windows if I can reasonably manage it.

There are obvious Enterprise edges sticking out through the process though happily the new, fluffy, comsumer oriented image that RIM is putting on has really improved the developer start up process. Code signing keys no longer cost money, and getting them is basically a process of signing up in a web form and waiting a few days for the files to be delivered to your Inbox.

I ended up writing a small application that takes some of the backend data that powers my personal Status Board (similar to the Panic Status Board) and displays it on the phone along with some other useful data that I can get out of the RIM APIs.

I originally was just using jQuery to fetch the JSON from my status board but was somewhat unsatisfied by the performance of pulling the data over the macro network and loading it into the page every time the application loaded so I replaced that with an on-device cache and push update system that takes advantage of the BlackBerry Enterprise Server I have (RIM offers a free push service over BIS as well but I like the idea of using the BES’s encrypted transport) hosting my e-mail.

The push server side of the equation is somewhat poorly documented, mostly through snippits of Java and .NET code that other people have implemented and some PDFs from RIM describing the format. After some trial and error I was able to pull together a workable Python update script that fetches the JSON from my status board and pushes it to the handheld. This allows fast, bandwidth friendly, scheduled, asynchronous updates and improves the user experience quite a bit. I totally understand why RIM is pimping their push infrastructure as being a vital part of what they are calling “Super Apps”. Having the data come to the user versus making them constantly update it is kinda awesome.

I doubt this application will be useful for anyone else but in the odd case that it might be the code is available in my bzr repository, which you can view at: http://repo.ub3rgeek.net/branches/blackberry/files, the status-app directory contains the handheld application source and the push-server/status-push.py file contains the push application.

Am I growing up or is Moore’s law hiding my obsession?

I’ve been interested in technology and computing for a very long time. I built my first full time Linux server on Jun 13 1998 which I used to do what a $50 Linksys router does today, only for a 14,400kbps dial-up Internet link (which I assure you was impressive at the time).

Over the next several years I endeavored to have more and more computers running for various reasons. Back before laptops became accessable for the average person I solved the portability problem by having computers all over the house so that wherever I went I had a terminal on the network that could access whatever it was I wanted. I think back in the day before AIM allowed multiple sessions I had 5 or 6 AIM names which I left logged in at each workstation. It sure made finding me, even at home, a fun logistical problem.

Sometime around 2003 I received surplus from a company that was moving an enclosed network equipment rack. Part of the problem of having something like this is the desire to stuff more and more computers into it until it is full to bursting.

Part of this desire is augmented by the fact that I tend to do various complex things with computers and sometimes you just need a number of them to get these jobs done. Thankfully computers have been getting better and better and you can cram a lot more things into them. Also since I started working with technology for a living I have started to get more and more interested in being a user while I’m at home as opposed to being a Systems Engineer 24×7. So I guess the answer to the question in this post’s title is this: I am both growing up and at the same time the faster better stronger computers are making it easier and easier to hide from the rest of the world my raging computer problem.

To help illustrate that a quick comparason: in 1998 my very first Linux server was a 486 DX4-100 with about 320MB of hard drive space, and a whopping 48MB of RAM. In 2011 my sole home server is a dual-core Athlon II X250 with 4TB (6TB raw) of disk and 2GB of RAM.

So, for your concideration, the last few years in pictures of my home network, ‘internal.ub3rgeek.net’:

2011

2010

Late 2007

Early 2007

2005

Vacations

The hardest thing about vacations is coming back home and getting used to returning to the daily routines. I had a fantastic time in California as has come to be expected by now. There are a few new pictures over in the gallery. The highlights were the AOPA Summit in Long Beach and the spirit tasting and pairing at the St George Spirits distillery in Alameda. I had a fantastic two weeks visiting with wonderful friends, eating more than my share of awesome food (Izzy’s steakhouse has my vote for best steak ever and Gather in Berkley for best healthy meal ever) drinking my share of delicious wine, beer and spirits and got to take in more of California, which is never a bad thing.

I managed to get away from the AOPA Summit only spending a few hundred dollars (resisting the urge to buy a Bose A20, somehow). I took in two extremely good and interesting panels, one on flying in and around Mexico and the other on FedEx Flight 705 by the co-pilot who was on board that day. It was a harrowing tale and I think I will always remember it. It is another affirmation that the idea of becoming a pilot is a dream that I cannot wait to finally fulfill.

It is always a good feeling getting away from home and spending time in different places. I don’t think that I shall ever lose the joy of traveling, but I am certainly going to try as hard as I can to wear myself out.

Now that I am back home the weather here has of course turned cold and I am having to fight the urge to just curl up and hide from it. I am hoping that the weather stays good enough so that I can continue my flight training.

I have logged 11 hours of dual received so far and nearly 50,000 miles aboard United in the last two years, over 25,000 this year, qualifying me for Elite status. All in all a good time.

Next year my two closest friends will be moving into his first house, and getting married respectively. If all goes well I will be visiting another friend in Japan and of course I will have to find time to fit in another trek to Australia to see my dear sister and brother-in-law, Pennsic, and whatever else comes across my plate.

Surfing Safety, traveling lightly.

The dangerous part of flying

Statistically speaking flying (even General Aviation) is pretty darn safe to life, limb, and property (AOPA has some nice statistics here) however it does pose a significant risk to your wallet. Flying is of course not a cheap hobby to have in the first place but it is all the ancillary costs that you don’t factor in. Yes, you DO need that $1100 headset, yes you DO need that ‘I fly’ doormat you saw in the latest catalog that randomly showed up at your house because the FAA publishes certificate holder information publicly, and YES, you do need to get in the big sky buses to go to various aviation related events in all parts of the globe. Flying isn’t a drug or an addiction, flying is something that was always there in your blood that you woke up and now that it is awake it is no longer something you can ignore.

I didn’t get to go to Oshkosh this year but I am making up for it by attending The AOPA Summit in Long Beach, CA in a couple weeks. In the mean time I am hoping the weather will cooperate and I will end up getting 2 or 3 more hours of flight time in before I leave. Some take-off and landing work would be excellent…

Ten Hours

Today marks ten hours in the log book and a little over a month of flying lessons. I have been to (and landed at) two airports other than my home base and am starting to get comfortable with the multitude of tasks that you have to complete while operating an aircraft. The ground operations are pretty solid (preflight, planning, taxi, run-up, getting around the airport) and on smooth days I’m pretty confident during climb, cruise and descent.

Take-off and landing is coming along, now that my instructor is more or less hands (and feet) off during flight I’ve had some confidence set-backs but I can feel it coming back the more practice I get in. Unfortunately weather and work have encroached on my lessons so I have not had as many as I would liked to have by now but I can feel a lot of the stuff coming together.

We started working on emergency procedures today and I can feel some of the aeronautical decision making stuff come together.

Having a couple-week break between lessons really showed, I felt some of the rusty-ness during take-off and climb-out of SDC (my home airport) but thankfully that started to clear up once I got en-route. I still have trouble with some of the landmarks, usually heading south for some reason but I’m usually pretty confident moving back towards the north.

Radio calls are proceeding pretty well, I’ve got a good feel for it even though now and then I forget bits (most often I forget the end) so I end up with stuff like “Penn Yan traffic, cherokee five five whisky is ten miles north of the field, penn yan traffic” and my instructor will ask “and what are we gonna do when we get there?” Oh yeah, land… heh.

All in all I’m still super-glad I decided to pursue this. The whole world of aviation is just amazing and being a part of it (as well as getting to use the big blue yonder that my tax dollars are paying for) is fantastic.

Almost two hours in…

I had my first honest-to-goodness-fly-the-plane flight lesson last week and I swear I haven’t been able to stop thinking about it since. I have no trouble understanding the science of flying, it all makes perfect sense to me. The best part of science though is always practical application. Sure, theorizing about the Higgs Boson is fun but when you get down to the part where you are slamming subatomic particles together at nearly the speed of light something totally different happens. Similarly when you are going down the runway at full throttle and the plane gets light and you get the nose up above the horizon it is just something you can’t quite get out of your skull.

I really do understand why flight has for most of our history captivated us. I feel supremely lucky to be alive in a time where aviation is an accessible hobby.

So far the hardest part has been the time between landing and takeoff.

1.8 hours down… next lesson in 4 days…

Site updates — unification the themes

I spent a little time in the last day or so updating WordPress and loggerhead with themes that look pretty close to the main ub3rgeek.net page. I apologize if something looks funky. Please feel free to let me know if you find anything broken.

WiFi Calling – The biggest reason I am staying with T-Mobile

It is pretty obvious that T-Mobile is a fourth-string carrier here in the United States. They were pretty late to the 3G game and while now with their HSPA+ network they have the fastest network currently in wide availability in the US they still have a long way to go to even catch up with Sprint.

That said, around 3 years ago I left Sprint (after 7 years) and went over to the magenta side. I started traveling internationally and wanted a GSM phone and so, firmly remembering my hatred for Cingulair wireless (now AT&T) I went to the only provider at the time that would enable me to roam almost anywhere. And boy am I lucky that I did.

No other US nation-wide carrier presently supports dual-mode calling (WiFi and Cellular) which is a huge shame as it has been the single best feature of being a T-Mobile customer. Being able to fill the gaps in their coverage with WiFi is wonderful by itself but the true surprise is being able to reduce or even eliminate roaming charges when traveling internationally. I spent two weeks in Australia earlier this summer and while I was there when I could I spent much of my time actually registered on the T-Mobile network via UMA as opposed to roaming on Vodafone AU. This saved me about $80 on my bill versus the previous year when I was in Australia without having a phone capable of WiFi.

I really don’t know why every carrier doesn’t support this, being able to fill the gaps in your network by using unlicensed access technologies sure beats the hell out of the expensive femtocell technology, and the customer benefits of reducing roaming charges is a pretty compelling business case (if you ask me).

As an aside, presently only the BlackBerry and a couple Nokia handsets on T-Mobile support UMA. This is one of the huge reasons I won’t even consider switching to Android at this point.

Advantages to working smarter…

Now I know that this may not even apply universally to everyone in the Information Technology field but I felt that I needed to share anyway.

One of the greatest things about my current job is the ability to work from just about anywhere using just about any computer as long as I have an Internet connection and a SSH client. I frequently find myself working from home on my personal laptop, or from friend’s houses using their own computers and I am as effective as I would be at the office. Today really cemented that fact with me when I made the Monday morning blunder of arriving to work without my laptop. Instead of driving all the way back home (and wasting even more of the morning) I just grabbed a spare laptop that is laying around the office and using that today.

I remember a number of nights earlier in the year when I was on-call and out somewhere either without my laptop or without an Internet connection. Though it was slightly slower than I would have been on a laptop I managed to resolve a number of production critical emergencies using nothing more than my BlackBerry and MidpSSH.

I leverage a few tools to make this happen but the most important one I think of them all is GNU screen. The usefulness of this to anyone who wants to be able to use any computer in the world to be productive cannot be over-stressed. I recall one evening while I was on vacation in Australia last month where I noticed (via our internal IRC channel) that people were trying to troubleshoot a problem with our network storage infrastructure and was able to connect in to my work computer 12,000 miles away and fix the issues with nothing more than a terminal application on my laptop.

For IM and IRC I have as much connected to Irssi as I can reasonably manage. Irssi is a really great compliment to screen and there are a number of sites advocating that particular use case. I never miss anything with this configuration as I am always online and Irssi is kind enough to log everything and provide a pretty lengthy scroll-back buffer. What I end up with is essentially one screen session that contains my e-mail (using mutt) for both home and work, IRC, IM, and many of our internal web-based tools (thanks to w3m). Thanks to work being almost exclusively a Linux shop all of this can sit running all the time on a system sitting in one of our data centres with what essentially amounts to a Gigabit Ethernet connection to the Internet. From there all I ever really need is ssh(1) to connect to any of several hundred servers scattered across the United States and I can manage to be just as productive as if I was sitting in the office.

I honestly can’t imagine considering a job where I’m tied down to an office, or at the very least one where I was tied down to an office and then expected to work non-normal business hours. With pretty simple tools it is extremely easy to allow IT people to work from anywhere happily and securely. The ability for employees to continue to do their job from anywhere on the planet should the need arise is not only great for morale and general quality of life (you mean I can work from the beach?!) but makes sense when business critical applications and services require specific expertise that may not be replicated anywhere else in the company (what do you mean Bob is on vacation in Fiji?!).

The heady dreams of youth revisited… flight!

Earlier this year I had the good fortune to visit some friends in California and take my first ride in a light airplane. In spite of the 29,000+ miles I have flown this year the smallest plane I had been on prior was a twin-turboprop Delta Connection flight out of IAD to ROC so this was a really unique experience for me.

We flew from AUN out over the San Francisco Bay, out to the Golden Gate Bridge and then down the Pacific coast to HAF where we went and had dinner. After dinner we flew over San Francisco back up to AUN after dark which was a truly beautiful view and if I am honest was almost magical.

I think it jarred something loose in my head. For as long as I can remember I have always been fascinated with aviation and have always wanted to learn to fly. For years though that dream sat idle in the pile of things that would be nice to do someday when I made ‘real money’ and could afford such extravagances (like air conditioning or food). I’m not really sure how much I was smiling after that first flight but I imagine you probably could have seen it from at least 10,000ft. I think I was hooked at that very moment. It really doesn’t help that my friend is a very persuasive individual when he is passionate about something, and flying is something he is exceptionally passionate about. It took me a few months, and a few more trips to convince me that it was truly something within my grasp but last week I finally got down to the local airport SDC, and took an intro flight with the resident CFII.

We flew out of SDC and then down towards my house in Fairport, circled around and came back. It was only a half-hour of flying time but it was easily the most memorable half hour of my life in the last few years. And to be clear in the last few years I have been to Edinburgh and London, Australia… twice as well as California three times. So that is a statement that I do not take lightly.

There are some logistics still up in the air but never before have I been quite as excited about anything as I am about the prospect of being able to take to the sky and explore the world of General Aviation.

While my wallet will never be quite the same again I can’t thank everyone who finally kicked me off this cliff enough. I can’t promise I will take malicious notes here chronicling my experience as I seem to have an aversion to updating this damnable thing but I am sure I will update more often than I have recently.

PXEboot VMWare ESXi 4.0 Installer

Introduction, or Why I care…

I have been meaning to install ESXi on a server I have sitting in the house for a while but a lack of CD-ROM drive has been stalling things for a while so I wanted to see if there was any chance of booting the ISO image from the network.

The server has internal storage, so I’m not RUNNING ESXi off the PXEBoot image, I’m installing it.

1. Yes, there sure is
2. All the guides currently around the Internet, SUCK.

Prerequisites, or what you should already know…

You will obviously require a working PXEboot Network infrastructure, which means a functioning TFTP server and a functioning DHCP server, which is setup to give out your TFTP server and PXELinux executable. There are numerous guides on this so I’m not going to go into that here. For reference, I’m running Ubuntu Server 9.10 on my TFTP host and OpenBSD 4.7-current on my DHCP server.

Process, or what the hell do I do…

Get the VMWare ESXi ISO from their website, this should be more straightforward than it is, but what can you do. Once you have the image, use something like Daemon Tools under Windows, or a loopback mount(8) under Linux to get access to the ISO. The root directory of the ISO should look something like this:

boot.cat
cimstg.tgz
cim.vgz
cisco_n1k
ienviron.tgz
image.tgz
install.tgz
isolinux.bin
isolinux.cfg
mboot.c32
menu.c32
README.txt
sys.vgz
vmkboot.gz
vmkernel.gz

If you look at the isolinux.cfg file, you will see what files are loaded for boot. I copied mboot.c32 (the chainloader), and *gz to a subdirectory in my tftpboot dir (the Ubuntu default of /var/lib/tftpboot in my case) called esxi. I then added the following line to my pxelinux.cfg/default file:


LABEL esxi
kernel mboot.c32
append esxi/vmkboot.gz --- esxi/vmkernel.gz --- esxi/sys.vgz --- esxi/cim.vgz --- esxi/ienviron.tgz --- esxi/install.tgz --- esxi/image.tgz


You should now be able to boot your system and select Network Boot, it will take a while to tftp over the image.tgz file but once it does it should fire up the ESXi installer and let you install as you would over the harddrive. A few minutes later you should have a screen telling you to reboot and then you should have a working ESXi server.

Bluetooth DUN with BlackBerry 9700, T-Mobile and Ubuntu 9.10

I just bought a new laptop (Thinkpad T500) but the only WWAN cards available built in were for either Verizon or AT&T, which is suboptimal as I use T-Mobile which uses the AWS UMTS bands (1,4,8) instead of the bands used in the US by AT&T (1,2,5,6). Since my Blackberry 9700 supports T-Mobile 3G as well as their implementation of UMA I decided to try to get the Bluetooth DUN working.

I was hoping that with 9.10, things would have progressed enough that I could pair the phone with the computer and let NetworkManager take care of all of the irritating tedious stuff. With my GPRS PCMCIA card this is basically the case, NetworkManager notices that it is a GPRS WWAN card, and asks me who my carrier is and connects just fine. Unfortunately this is not the case with Bluetooth. I can get the phone paired but for some reason it exclaims that this is not a GPRS modem and refuses to let me tell it otherwise, so I’m forced to go about this the hard way.

After much searching, this post got me the closest though recent changes either with the new OS 5.0 on the 9700 or T-Mobile’s network have made some tweaks necessary.

http://pegelinux.wordpress.com/2008/06/11/blackberry-curve-8310-as-bluetooth-modem-on-ubuntu-hardy/

I will not go into all the step-by-step details as it is covered above and elsewhere, but first you need to get your computer to bind to the Blackberry DUN service via Bluetooth, I chose to edit /etc/bluetooth/rfcomm.conf to make this happen automatically, but BlueManager also seems to be able to do the same thing with a much more friendly UI if you want to try that route, anyway you cut it, you need to have the DUN service show up as a /dev/rfcomm# device. Once you have completed that you will want to setup pppd(8) much like the above article suggests.

This resulted in a session that looked like the following for me:

mernisse@mernisse-laptop:/etc/chatscripts$ pppd call gprs
Press CTRL-C to close the connection at any stage!
defining PDP context...

OK

OK
waiting for connect...

CONNECT
Connected.
If the following ppp negotiations fail,
try restarting the phone.

Script /etc/chatscripts/gprs-connect-chat finished (pid 14835), status = 0x0
Serial connection established.
using channel 1
Using interface ppp0
Connect: ppp0 < --> /dev/rfcomm0
sent LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x5db98b32>
rcvd LCP ConfReq id=0x0 <asyncmap 0x0> <auth pap>
sent LCP ConfAck id=0x0 <asyncmap 0x0> <auth pap>
rcvd LCP ConfRej id=0x1 <magic 0x5db98b32>
sent LCP ConfReq id=0x2 <asyncmap 0x0>
rcvd LCP ConfAck id=0x2 </asyncmap><asyncmap 0x0>
sent PAP AuthReq id=0x1 user="mernisse-laptop" password=""
rcvd PAP AuthAck id=0x0
PAP authentication succeeded
sent CCP ConfReq id=0x1 <deflate 15> </deflate><deflate (old#) 15>
sent IPCP ConfReq id=0x1 <addr 0.0.0.0> <ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
rcvd LCP ProtRej id=0x1 80 fd 01 01 00 0c 1a 04 78 00 18 04 78 00
Protocol-Reject for 'Compression Control Protocol' (0x80fd) received
rcvd IPCP ConfReq id=0x2 <addr 169.254.1.1>
sent IPCP ConfAck id=0x2 </addr><addr 169.254.1.1>
rcvd IPCP ConfNak id=0x1 <ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
sent IPCP ConfReq id=0x2 <addr 0.0.0.0> <ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
rcvd IPCP ConfNak id=0x2 </ms><ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
sent IPCP ConfReq id=0x3 <addr 0.0.0.0> <ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
rcvd IPCP ConfNak id=0x3 </ms><ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
sent IPCP ConfReq id=0x4 <addr 0.0.0.0> <ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
rcvd IPCP ConfNak id=0x4 </ms><ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
sent IPCP ConfReq id=0x5 <addr 0.0.0.0> <ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
rcvd IPCP ConfNak id=0x5 </ms><ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
sent IPCP ConfReq id=0x6 <addr 0.0.0.0>
rcvd IPCP ConfNak id=0x6
sent IPCP ConfReq id=0x7 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x7
sent IPCP ConfReq id=0x8 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x8
sent IPCP ConfReq id=0x9 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x9
sent IPCP ConfReq id=0xa </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0xa </addr><addr 14.36.76.18>
sent IPCP ConfReq id=0xb
rcvd IPCP ConfAck id=0xb
Could not determine local IP address
Connect time 0.1 minutes.
Sent 186 bytes, received 120 bytes.
sent IPCP TermReq id=0xc "Could not determine local IP address"
rcvd IPCP TermAck id=0xc
sent LCP TermReq id=0x3 "No network protocols running"
rcvd LCP TermAck id=0x3
Connection terminated.

Sending break to the modem

PDP context detached
Script /etc/chatscripts/gprs-disconnect-chat finished (pid 14843), status = 0x0
Serial link disconnected.


As you can see, a number of Naks have been received for things that do not work and the connection fails. Tracking each error down step by step I removed noauth from the peers file and replaced it with user “” to force a null user-name in the PAP authentication. I also removed userpeerdns as it appears that the Blackberry was refusing to give it to me. The final piece was that I was hitting what I assumed to be a race condition, as the Blackberry did not have an IP address to give me yet, it kept sending ConfNak to me until it finished setting up the tunnel to T-Mobile on my behalf… (see inset)

To overcome this I had to up the ipcp-max-configure to accomidate, the default according to pppd(8) is 10 so I set it to 20. This, along with a few other tweaks allowed me to connect successfully.


mernisse@mernisse-laptop:/etc/ppp/peers$ pppd call gprs
Press CTRL-C to close the connection at any stage!
defining PDP context...

OK

OK
waiting for connect...

CONNECT
Connected.
If the following ppp negotiations fail,
try restarting the phone.

Script /etc/chatscripts/gprs-connect-chat finished (pid 15068), status = 0x0
Serial connection established.
using channel 12
Using interface ppp0
Connect: ppp0 < --> /dev/rfcomm0
sent LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x448e9c82>
rcvd LCP ConfReq id=0x0 <asyncmap 0x0> <auth pap>
sent LCP ConfAck id=0x0 <asyncmap 0x0> <auth pap>
rcvd LCP ConfRej id=0x1 <magic 0x448e9c82>
sent LCP ConfReq id=0x2 <asyncmap 0x0>
rcvd LCP ConfAck id=0x2 </asyncmap><asyncmap 0x0>
sent PAP AuthReq id=0x1 user="" password=<hidden>
rcvd PAP AuthAck id=0x1
PAP authentication succeeded
sent IPCP ConfReq id=0x1 <addr 0.0.0.0>
rcvd IPCP ConfReq id=0x1 </addr><addr 169.254.1.1>
sent IPCP ConfAck id=0x1 </addr><addr 169.254.1.1>
rcvd IPCP ConfNak id=0x1
sent IPCP ConfReq id=0x2 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x2
sent IPCP ConfReq id=0x3 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x3
sent IPCP ConfReq id=0x4 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x4
sent IPCP ConfReq id=0x5 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x5
sent IPCP ConfReq id=0x6 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x6
sent IPCP ConfReq id=0x7 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x7
sent IPCP ConfReq id=0x8 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x8 </addr><addr 14.34.55.230>
sent IPCP ConfReq id=0x9 </addr><addr 14.34.55.230>
rcvd IPCP ConfAck id=0x9 </addr><addr 14.34.55.230>
replacing old default route to wlan0 192.168.196.1
Cannot determine ethernet address for proxy ARP
local IP address 14.34.55.230
remote IP address 169.254.1.1
Script /etc/ppp/ip-up started (pid 15070)
Script /etc/ppp/ip-up finished (pid 15070), status = 0x0
^CTerminating on signal 2
Connect time 4.7 minutes.
Sent 300422 bytes, received 1437245 bytes.
restoring old default route to wlan0 192.168.196.1
Script /etc/ppp/ip-down started (pid 15112)
sent LCP TermReq id=0x3 "User request"
rcvd LCP TermAck id=0x3
Connection terminated.

Sending break to the modem

PDP context detached
Script /etc/chatscripts/gprs-disconnect-chat finished (pid 15117), status = 0x0
Serial link disconnected.
Script /etc/ppp/ip-down finished (pid 15112), status = 0x0
mernisse@mernisse-laptop:/etc/ppp/peers$


Looking at the phone it appears that it snuck into Modem Mode and I was able to ssh around and perform the speedtest that adorns the top of this post. I used the gprs-disconnect-chat file unmodified from the post I linked to, but modified the gprs and gprs-connect-chat files which you can get by clicking on their names or looking at http://www.ub3rgeek.net/gprs/

Hopefully this will help someone get past this needlessly difficult hurdle and also enjoy wireless data anywhere in T-Mobile’s coverage area. Cheers.

Backups!

With hard drive capacity growing by leaps and bounds it is not uncommon for most people to have computers with several hundred Gigabytes worth of internal storage. While drives have gotten much more reliable the importance of good backups have only increased with more and more of our data being stored on our computers. Digital photos, movies, music, financial documents, e-mails, all live on our computers and could easily be deleted, or corrupted by user error, application faults, or a physical failure of a part in your computer.

Backups prevent:

1. Data loss from computer error. (Operating System or Application Crash)
2. Data loss from drive failure.
3. Data loss from user error. (Accidental deletion, alteration, etc)
4. Data loss from disaster. (House burns down, computer lost or stolen)

It is important to note that RAID systems ONLY address drive failures and should NOT be thought of as a backup solution.

My Goals:
I really like Time Machine. Apple did an AMAZING job with that application and bundled with the Time Capsule you get a really robust, high-capacity versioned backup solution that solves 90% of the problems that backups are supposed to solve. If you periodically copy your Time Machine backup to an external hard drive and store that at a friend’s house or in a safe deposit box or at the office, then you get 100% of the backup goals in a neat little package. You can even restore a Time Machine backup from the OS X restore CD. Slick.

Since I don’t run OS X I have to engineer a solution myself. I want a system that allows me to produce versioned backups on the network without much interaction. It must be encrypted on disk as my laptop’s /home is encrypted and storing the backup unencrypted would be a super-easy attack vector. Ideally the storage would be swappable so I can have 2 drives and swap it with an off-site drive periodically and so I could fairly easily restore the data to a new drive when the drive in my laptop (or the backup drive) scorches itself.

My Solution:

• USB 2.0 hard drive with dm-crypt whole-drive encryption
• rdiff-backup over ssh
• cron(8) and some custom scripts

I have an old IBM ThinkPad R42 laptop that I’m not using for anything anymore so I tossed Ubuntu 9.10 on it and plugged it into the network. (I would use my server but I do not have any USB 2.0 ports on that machine currently.) I connected a 250GB USB hard drive to the ThinkPad and encrypted a partition using dm-crypt to store the backups and meta-data.

I then setup ssh on the ThinkPad to trust my ssh key from my laptop and I setup sudo(8) to allow my user to run rdiff-backup without entering my password. Combined with my backup.sh script, i can fire off a backup in the background from cron(8) that doesn’t really affect the performance of my laptop over the network.

I am a big fan of jwz and this is how he does backups.

13 years with Linux

I first installed Linux from floppy disks back in 1996 on an old computer I had laying around. A friend of mine whom I knew through a local BBS had told me about this thing “Linux”, it was like Unix but for cheap Intel computers. I was amazed and after some tinkering I was also completely hooked. From that point I have always had at least one Linux machine running in my home. I got my mother to buy our first Internet connection with the promise that everyone in the house could use it, all thanks to Linux and this new thing called IP Masquerading.

That first Linux distribution was Slackware, a distribution by Patrick J. Volkerding which has always focused on simplicity, stability and having a sane collection of packages and defaults right out of the box. It is kind of like the OpenBSD of Linux if I was to try to make a really unnecessary analogy. Over the years my home server has always been Slackware, it was what I was the most comfortable with and it has never let me down. I even ran Slackware on my workstations up until a few years ago when World of Warcraft came around and I needed something a little more bleeding edge to get an X server with all the fancy direct rendering and 3d acceleration bits to make wine happy. I have trusted literally TeraBytes of data to Slackware and have not once been let down.

Over the last few days I have been struggling with a number of problems with my Sirius transcoder scripts. Between Sipie being unloved and requiring some work, and vlc being a royal pain sometimes I decided to upgrade my home server to Slackware 13.0. I have to start by saying i’m impressed with Slackware and the progress it has made over the years, a very small team (usually of one) has been able to make a stable, secure, reliable distribution for a very long time and the new version is no exception. And Slackware 13.0 is no exception to this rule, now bringing in amd64 support into the core distribution.

After a good 2 days of fighting the realization came that I just had to embrace, I have changed. The sad truth is that as with hardware I have come to expect software to Just Work(tm) and leave me out of it. I spend 60 hours a week wrestling with hardware and software in my job as a Systems Administrator for an ISP and I just don’t enjoy fighting it when I get home like I used to. My server acts as a media server for my PS3 so I can enjoy media content on my TV, a VOIP server for my home (I do not have a land line any longer) and hosts a number of status monitoring applications and stream recorders that I turn into podcasts for my private consumption. I need it all to just quietly work and cause me as little grief as possible.

So sadly it became time for me to go from loyal Slackware user for over a decade to full-on Ubuntu convert. We use Debian (and now Ubuntu) at work and I have been using Ubuntu in varying places since a friend of mine gave me a link in a chat pointing to what he promised was going to be a new Linux distribution that would change how people looked a Linux. It just made sense to convert, it meant I didn’t have to maintain 2 local mirrors anymore for security updates, I am much more comfortable with apt(8) and dpkg(8) and utilities like slapt-get(8) and swaret(8) are just kind of (very good) 3rd party hacks on top of the Slackware package management system.

It is amazing to see how far this dream has come, more and more I find myself quietly relying on various Free Software products to make my life simpler, easier and better and Slackware and Ubuntu have both been huge parts of that software stack. Thanks Pat, for 13 years of reliability, stability, security and sanity. Thanks also to Mark and Matt and the entire Ubuntu/Canonical gang for literally changing how thousands of people think of Linux. I converted my Mother a few weeks ago and have had less calls about the computer than I ever did with Windows.