ub[3]rgeek.net

Am I growing up or is Moore’s law hiding my obsession?

I’ve been interested in technology and computing for a very long time. I built my first full time Linux server on Jun 13 1998 which I used to do what a $50 Linksys router does today, only for a 14,400kbps dial-up Internet link (which I assure you was impressive at the time).

Over the next several years I endeavored to have more and more computers running for various reasons. Back before laptops became accessable for the average person I solved the portability problem by having computers all over the house so that wherever I went I had a terminal on the network that could access whatever it was I wanted. I think back in the day before AIM allowed multiple sessions I had 5 or 6 AIM names which I left logged in at each workstation. It sure made finding me, even at home, a fun logistical problem.

Sometime around 2003 I received surplus from a company that was moving an enclosed network equipment rack. Part of the problem of having something like this is the desire to stuff more and more computers into it until it is full to bursting.

Part of this desire is augmented by the fact that I tend to do various complex things with computers and sometimes you just need a number of them to get these jobs done. Thankfully computers have been getting better and better and you can cram a lot more things into them. Also since I started working with technology for a living I have started to get more and more interested in being a user while I’m at home as opposed to being a Systems Engineer 24×7. So I guess the answer to the question in this post’s title is this: I am both growing up and at the same time the faster better stronger computers are making it easier and easier to hide from the rest of the world my raging computer problem.

To help illustrate that a quick comparason: in 1998 my very first Linux server was a 486 DX4-100 with about 320MB of hard drive space, and a whopping 48MB of RAM. In 2011 my sole home server is a dual-core Athlon II X250 with 4TB (6TB raw) of disk and 2GB of RAM.

So, for your concideration, the last few years in pictures of my home network, ‘internal.ub3rgeek.net’:

2011

2010

Late 2007

Early 2007

2005

WiFi Calling – The biggest reason I am staying with T-Mobile

It is pretty obvious that T-Mobile is a fourth-string carrier here in the United States. They were pretty late to the 3G game and while now with their HSPA+ network they have the fastest network currently in wide availability in the US they still have a long way to go to even catch up with Sprint.

That said, around 3 years ago I left Sprint (after 7 years) and went over to the magenta side. I started traveling internationally and wanted a GSM phone and so, firmly remembering my hatred for Cingulair wireless (now AT&T) I went to the only provider at the time that would enable me to roam almost anywhere. And boy am I lucky that I did.

No other US nation-wide carrier presently supports dual-mode calling (WiFi and Cellular) which is a huge shame as it has been the single best feature of being a T-Mobile customer. Being able to fill the gaps in their coverage with WiFi is wonderful by itself but the true surprise is being able to reduce or even eliminate roaming charges when traveling internationally. I spent two weeks in Australia earlier this summer and while I was there when I could I spent much of my time actually registered on the T-Mobile network via UMA as opposed to roaming on Vodafone AU. This saved me about $80 on my bill versus the previous year when I was in Australia without having a phone capable of WiFi.

I really don’t know why every carrier doesn’t support this, being able to fill the gaps in your network by using unlicensed access technologies sure beats the hell out of the expensive femtocell technology, and the customer benefits of reducing roaming charges is a pretty compelling business case (if you ask me).

As an aside, presently only the BlackBerry and a couple Nokia handsets on T-Mobile support UMA. This is one of the huge reasons I won’t even consider switching to Android at this point.

Advantages to working smarter…

Now I know that this may not even apply universally to everyone in the Information Technology field but I felt that I needed to share anyway.

One of the greatest things about my current job is the ability to work from just about anywhere using just about any computer as long as I have an Internet connection and a SSH client. I frequently find myself working from home on my personal laptop, or from friend’s houses using their own computers and I am as effective as I would be at the office. Today really cemented that fact with me when I made the Monday morning blunder of arriving to work without my laptop. Instead of driving all the way back home (and wasting even more of the morning) I just grabbed a spare laptop that is laying around the office and using that today.

I remember a number of nights earlier in the year when I was on-call and out somewhere either without my laptop or without an Internet connection. Though it was slightly slower than I would have been on a laptop I managed to resolve a number of production critical emergencies using nothing more than my BlackBerry and MidpSSH.

I leverage a few tools to make this happen but the most important one I think of them all is GNU screen. The usefulness of this to anyone who wants to be able to use any computer in the world to be productive cannot be over-stressed. I recall one evening while I was on vacation in Australia last month where I noticed (via our internal IRC channel) that people were trying to troubleshoot a problem with our network storage infrastructure and was able to connect in to my work computer 12,000 miles away and fix the issues with nothing more than a terminal application on my laptop.

For IM and IRC I have as much connected to Irssi as I can reasonably manage. Irssi is a really great compliment to screen and there are a number of sites advocating that particular use case. I never miss anything with this configuration as I am always online and Irssi is kind enough to log everything and provide a pretty lengthy scroll-back buffer. What I end up with is essentially one screen session that contains my e-mail (using mutt) for both home and work, IRC, IM, and many of our internal web-based tools (thanks to w3m). Thanks to work being almost exclusively a Linux shop all of this can sit running all the time on a system sitting in one of our data centres with what essentially amounts to a Gigabit Ethernet connection to the Internet. From there all I ever really need is ssh(1) to connect to any of several hundred servers scattered across the United States and I can manage to be just as productive as if I was sitting in the office.

I honestly can’t imagine considering a job where I’m tied down to an office, or at the very least one where I was tied down to an office and then expected to work non-normal business hours. With pretty simple tools it is extremely easy to allow IT people to work from anywhere happily and securely. The ability for employees to continue to do their job from anywhere on the planet should the need arise is not only great for morale and general quality of life (you mean I can work from the beach?!) but makes sense when business critical applications and services require specific expertise that may not be replicated anywhere else in the company (what do you mean Bob is on vacation in Fiji?!).

PXEboot VMWare ESXi 4.0 Installer

Introduction, or Why I care…

I have been meaning to install ESXi on a server I have sitting in the house for a while but a lack of CD-ROM drive has been stalling things for a while so I wanted to see if there was any chance of booting the ISO image from the network.

The server has internal storage, so I’m not RUNNING ESXi off the PXEBoot image, I’m installing it.

1. Yes, there sure is
2. All the guides currently around the Internet, SUCK.

Prerequisites, or what you should already know…

You will obviously require a working PXEboot Network infrastructure, which means a functioning TFTP server and a functioning DHCP server, which is setup to give out your TFTP server and PXELinux executable. There are numerous guides on this so I’m not going to go into that here. For reference, I’m running Ubuntu Server 9.10 on my TFTP host and OpenBSD 4.7-current on my DHCP server.

Process, or what the hell do I do…

Get the VMWare ESXi ISO from their website, this should be more straightforward than it is, but what can you do. Once you have the image, use something like Daemon Tools under Windows, or a loopback mount(8) under Linux to get access to the ISO. The root directory of the ISO should look something like this:

boot.cat
cimstg.tgz
cim.vgz
cisco_n1k
ienviron.tgz
image.tgz
install.tgz
isolinux.bin
isolinux.cfg
mboot.c32
menu.c32
README.txt
sys.vgz
vmkboot.gz
vmkernel.gz

If you look at the isolinux.cfg file, you will see what files are loaded for boot. I copied mboot.c32 (the chainloader), and *gz to a subdirectory in my tftpboot dir (the Ubuntu default of /var/lib/tftpboot in my case) called esxi. I then added the following line to my pxelinux.cfg/default file:


LABEL esxi
kernel mboot.c32
append esxi/vmkboot.gz --- esxi/vmkernel.gz --- esxi/sys.vgz --- esxi/cim.vgz --- esxi/ienviron.tgz --- esxi/install.tgz --- esxi/image.tgz


You should now be able to boot your system and select Network Boot, it will take a while to tftp over the image.tgz file but once it does it should fire up the ESXi installer and let you install as you would over the harddrive. A few minutes later you should have a screen telling you to reboot and then you should have a working ESXi server.

Bluetooth DUN with BlackBerry 9700, T-Mobile and Ubuntu 9.10

I just bought a new laptop (Thinkpad T500) but the only WWAN cards available built in were for either Verizon or AT&T, which is suboptimal as I use T-Mobile which uses the AWS UMTS bands (1,4,8) instead of the bands used in the US by AT&T (1,2,5,6). Since my Blackberry 9700 supports T-Mobile 3G as well as their implementation of UMA I decided to try to get the Bluetooth DUN working.

I was hoping that with 9.10, things would have progressed enough that I could pair the phone with the computer and let NetworkManager take care of all of the irritating tedious stuff. With my GPRS PCMCIA card this is basically the case, NetworkManager notices that it is a GPRS WWAN card, and asks me who my carrier is and connects just fine. Unfortunately this is not the case with Bluetooth. I can get the phone paired but for some reason it exclaims that this is not a GPRS modem and refuses to let me tell it otherwise, so I’m forced to go about this the hard way.

After much searching, this post got me the closest though recent changes either with the new OS 5.0 on the 9700 or T-Mobile’s network have made some tweaks necessary.

http://pegelinux.wordpress.com/2008/06/11/blackberry-curve-8310-as-bluetooth-modem-on-ubuntu-hardy/

I will not go into all the step-by-step details as it is covered above and elsewhere, but first you need to get your computer to bind to the Blackberry DUN service via Bluetooth, I chose to edit /etc/bluetooth/rfcomm.conf to make this happen automatically, but BlueManager also seems to be able to do the same thing with a much more friendly UI if you want to try that route, anyway you cut it, you need to have the DUN service show up as a /dev/rfcomm# device. Once you have completed that you will want to setup pppd(8) much like the above article suggests.

This resulted in a session that looked like the following for me:

mernisse@mernisse-laptop:/etc/chatscripts$ pppd call gprs
Press CTRL-C to close the connection at any stage!
defining PDP context...

OK

OK
waiting for connect...

CONNECT
Connected.
If the following ppp negotiations fail,
try restarting the phone.

Script /etc/chatscripts/gprs-connect-chat finished (pid 14835), status = 0x0
Serial connection established.
using channel 1
Using interface ppp0
Connect: ppp0 < --> /dev/rfcomm0
sent LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x5db98b32>
rcvd LCP ConfReq id=0x0 <asyncmap 0x0> <auth pap>
sent LCP ConfAck id=0x0 <asyncmap 0x0> <auth pap>
rcvd LCP ConfRej id=0x1 <magic 0x5db98b32>
sent LCP ConfReq id=0x2 <asyncmap 0x0>
rcvd LCP ConfAck id=0x2 </asyncmap><asyncmap 0x0>
sent PAP AuthReq id=0x1 user="mernisse-laptop" password=""
rcvd PAP AuthAck id=0x0
PAP authentication succeeded
sent CCP ConfReq id=0x1 <deflate 15> </deflate><deflate (old#) 15>
sent IPCP ConfReq id=0x1 <addr 0.0.0.0> <ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
rcvd LCP ProtRej id=0x1 80 fd 01 01 00 0c 1a 04 78 00 18 04 78 00
Protocol-Reject for 'Compression Control Protocol' (0x80fd) received
rcvd IPCP ConfReq id=0x2 <addr 169.254.1.1>
sent IPCP ConfAck id=0x2 </addr><addr 169.254.1.1>
rcvd IPCP ConfNak id=0x1 <ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
sent IPCP ConfReq id=0x2 <addr 0.0.0.0> <ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
rcvd IPCP ConfNak id=0x2 </ms><ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
sent IPCP ConfReq id=0x3 <addr 0.0.0.0> <ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
rcvd IPCP ConfNak id=0x3 </ms><ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
sent IPCP ConfReq id=0x4 <addr 0.0.0.0> <ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
rcvd IPCP ConfNak id=0x4 </ms><ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
sent IPCP ConfReq id=0x5 <addr 0.0.0.0> <ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
rcvd IPCP ConfNak id=0x5 </ms><ms -dns1 0.0.0.0> </ms><ms -dns2 0.0.0.0>
sent IPCP ConfReq id=0x6 <addr 0.0.0.0>
rcvd IPCP ConfNak id=0x6
sent IPCP ConfReq id=0x7 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x7
sent IPCP ConfReq id=0x8 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x8
sent IPCP ConfReq id=0x9 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x9
sent IPCP ConfReq id=0xa </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0xa </addr><addr 14.36.76.18>
sent IPCP ConfReq id=0xb
rcvd IPCP ConfAck id=0xb
Could not determine local IP address
Connect time 0.1 minutes.
Sent 186 bytes, received 120 bytes.
sent IPCP TermReq id=0xc "Could not determine local IP address"
rcvd IPCP TermAck id=0xc
sent LCP TermReq id=0x3 "No network protocols running"
rcvd LCP TermAck id=0x3
Connection terminated.

Sending break to the modem

PDP context detached
Script /etc/chatscripts/gprs-disconnect-chat finished (pid 14843), status = 0x0
Serial link disconnected.


As you can see, a number of Naks have been received for things that do not work and the connection fails. Tracking each error down step by step I removed noauth from the peers file and replaced it with user “” to force a null user-name in the PAP authentication. I also removed userpeerdns as it appears that the Blackberry was refusing to give it to me. The final piece was that I was hitting what I assumed to be a race condition, as the Blackberry did not have an IP address to give me yet, it kept sending ConfNak to me until it finished setting up the tunnel to T-Mobile on my behalf… (see inset)

To overcome this I had to up the ipcp-max-configure to accomidate, the default according to pppd(8) is 10 so I set it to 20. This, along with a few other tweaks allowed me to connect successfully.


mernisse@mernisse-laptop:/etc/ppp/peers$ pppd call gprs
Press CTRL-C to close the connection at any stage!
defining PDP context...

OK

OK
waiting for connect...

CONNECT
Connected.
If the following ppp negotiations fail,
try restarting the phone.

Script /etc/chatscripts/gprs-connect-chat finished (pid 15068), status = 0x0
Serial connection established.
using channel 12
Using interface ppp0
Connect: ppp0 < --> /dev/rfcomm0
sent LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x448e9c82>
rcvd LCP ConfReq id=0x0 <asyncmap 0x0> <auth pap>
sent LCP ConfAck id=0x0 <asyncmap 0x0> <auth pap>
rcvd LCP ConfRej id=0x1 <magic 0x448e9c82>
sent LCP ConfReq id=0x2 <asyncmap 0x0>
rcvd LCP ConfAck id=0x2 </asyncmap><asyncmap 0x0>
sent PAP AuthReq id=0x1 user="" password=<hidden>
rcvd PAP AuthAck id=0x1
PAP authentication succeeded
sent IPCP ConfReq id=0x1 <addr 0.0.0.0>
rcvd IPCP ConfReq id=0x1 </addr><addr 169.254.1.1>
sent IPCP ConfAck id=0x1 </addr><addr 169.254.1.1>
rcvd IPCP ConfNak id=0x1
sent IPCP ConfReq id=0x2 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x2
sent IPCP ConfReq id=0x3 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x3
sent IPCP ConfReq id=0x4 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x4
sent IPCP ConfReq id=0x5 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x5
sent IPCP ConfReq id=0x6 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x6
sent IPCP ConfReq id=0x7 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x7
sent IPCP ConfReq id=0x8 </addr><addr 0.0.0.0>
rcvd IPCP ConfNak id=0x8 </addr><addr 14.34.55.230>
sent IPCP ConfReq id=0x9 </addr><addr 14.34.55.230>
rcvd IPCP ConfAck id=0x9 </addr><addr 14.34.55.230>
replacing old default route to wlan0 192.168.196.1
Cannot determine ethernet address for proxy ARP
local IP address 14.34.55.230
remote IP address 169.254.1.1
Script /etc/ppp/ip-up started (pid 15070)
Script /etc/ppp/ip-up finished (pid 15070), status = 0x0
^CTerminating on signal 2
Connect time 4.7 minutes.
Sent 300422 bytes, received 1437245 bytes.
restoring old default route to wlan0 192.168.196.1
Script /etc/ppp/ip-down started (pid 15112)
sent LCP TermReq id=0x3 "User request"
rcvd LCP TermAck id=0x3
Connection terminated.

Sending break to the modem

PDP context detached
Script /etc/chatscripts/gprs-disconnect-chat finished (pid 15117), status = 0x0
Serial link disconnected.
Script /etc/ppp/ip-down finished (pid 15112), status = 0x0
mernisse@mernisse-laptop:/etc/ppp/peers$


Looking at the phone it appears that it snuck into Modem Mode and I was able to ssh around and perform the speedtest that adorns the top of this post. I used the gprs-disconnect-chat file unmodified from the post I linked to, but modified the gprs and gprs-connect-chat files which you can get by clicking on their names or looking at http://www.ub3rgeek.net/gprs/

Hopefully this will help someone get past this needlessly difficult hurdle and also enjoy wireless data anywhere in T-Mobile’s coverage area. Cheers.

Backups!

With hard drive capacity growing by leaps and bounds it is not uncommon for most people to have computers with several hundred Gigabytes worth of internal storage. While drives have gotten much more reliable the importance of good backups have only increased with more and more of our data being stored on our computers. Digital photos, movies, music, financial documents, e-mails, all live on our computers and could easily be deleted, or corrupted by user error, application faults, or a physical failure of a part in your computer.

Backups prevent:

1. Data loss from computer error. (Operating System or Application Crash)
2. Data loss from drive failure.
3. Data loss from user error. (Accidental deletion, alteration, etc)
4. Data loss from disaster. (House burns down, computer lost or stolen)

It is important to note that RAID systems ONLY address drive failures and should NOT be thought of as a backup solution.

My Goals:
I really like Time Machine. Apple did an AMAZING job with that application and bundled with the Time Capsule you get a really robust, high-capacity versioned backup solution that solves 90% of the problems that backups are supposed to solve. If you periodically copy your Time Machine backup to an external hard drive and store that at a friend’s house or in a safe deposit box or at the office, then you get 100% of the backup goals in a neat little package. You can even restore a Time Machine backup from the OS X restore CD. Slick.

Since I don’t run OS X I have to engineer a solution myself. I want a system that allows me to produce versioned backups on the network without much interaction. It must be encrypted on disk as my laptop’s /home is encrypted and storing the backup unencrypted would be a super-easy attack vector. Ideally the storage would be swappable so I can have 2 drives and swap it with an off-site drive periodically and so I could fairly easily restore the data to a new drive when the drive in my laptop (or the backup drive) scorches itself.

My Solution:

• USB 2.0 hard drive with dm-crypt whole-drive encryption
• rdiff-backup over ssh
• cron(8) and some custom scripts

I have an old IBM ThinkPad R42 laptop that I’m not using for anything anymore so I tossed Ubuntu 9.10 on it and plugged it into the network. (I would use my server but I do not have any USB 2.0 ports on that machine currently.) I connected a 250GB USB hard drive to the ThinkPad and encrypted a partition using dm-crypt to store the backups and meta-data.

I then setup ssh on the ThinkPad to trust my ssh key from my laptop and I setup sudo(8) to allow my user to run rdiff-backup without entering my password. Combined with my backup.sh script, i can fire off a backup in the background from cron(8) that doesn’t really affect the performance of my laptop over the network.

I am a big fan of jwz and this is how he does backups.

13 years with Linux

I first installed Linux from floppy disks back in 1996 on an old computer I had laying around. A friend of mine whom I knew through a local BBS had told me about this thing “Linux”, it was like Unix but for cheap Intel computers. I was amazed and after some tinkering I was also completely hooked. From that point I have always had at least one Linux machine running in my home. I got my mother to buy our first Internet connection with the promise that everyone in the house could use it, all thanks to Linux and this new thing called IP Masquerading.

That first Linux distribution was Slackware, a distribution by Patrick J. Volkerding which has always focused on simplicity, stability and having a sane collection of packages and defaults right out of the box. It is kind of like the OpenBSD of Linux if I was to try to make a really unnecessary analogy. Over the years my home server has always been Slackware, it was what I was the most comfortable with and it has never let me down. I even ran Slackware on my workstations up until a few years ago when World of Warcraft came around and I needed something a little more bleeding edge to get an X server with all the fancy direct rendering and 3d acceleration bits to make wine happy. I have trusted literally TeraBytes of data to Slackware and have not once been let down.

Over the last few days I have been struggling with a number of problems with my Sirius transcoder scripts. Between Sipie being unloved and requiring some work, and vlc being a royal pain sometimes I decided to upgrade my home server to Slackware 13.0. I have to start by saying i’m impressed with Slackware and the progress it has made over the years, a very small team (usually of one) has been able to make a stable, secure, reliable distribution for a very long time and the new version is no exception. And Slackware 13.0 is no exception to this rule, now bringing in amd64 support into the core distribution.

After a good 2 days of fighting the realization came that I just had to embrace, I have changed. The sad truth is that as with hardware I have come to expect software to Just Work(tm) and leave me out of it. I spend 60 hours a week wrestling with hardware and software in my job as a Systems Administrator for an ISP and I just don’t enjoy fighting it when I get home like I used to. My server acts as a media server for my PS3 so I can enjoy media content on my TV, a VOIP server for my home (I do not have a land line any longer) and hosts a number of status monitoring applications and stream recorders that I turn into podcasts for my private consumption. I need it all to just quietly work and cause me as little grief as possible.

So sadly it became time for me to go from loyal Slackware user for over a decade to full-on Ubuntu convert. We use Debian (and now Ubuntu) at work and I have been using Ubuntu in varying places since a friend of mine gave me a link in a chat pointing to what he promised was going to be a new Linux distribution that would change how people looked a Linux. It just made sense to convert, it meant I didn’t have to maintain 2 local mirrors anymore for security updates, I am much more comfortable with apt(8) and dpkg(8) and utilities like slapt-get(8) and swaret(8) are just kind of (very good) 3rd party hacks on top of the Slackware package management system.

It is amazing to see how far this dream has come, more and more I find myself quietly relying on various Free Software products to make my life simpler, easier and better and Slackware and Ubuntu have both been huge parts of that software stack. Thanks Pat, for 13 years of reliability, stability, security and sanity. Thanks also to Mark and Matt and the entire Ubuntu/Canonical gang for literally changing how thousands of people think of Linux. I converted my Mother a few weeks ago and have had less calls about the computer than I ever did with Windows.

BlackBerry SMS logger and site changes

Firstly let me apologize for the mess, right now I have moved the WordPress installation to a standalone domain for back-end architecture reasons. There are redirects in place for most things but some things may be broken. Accept my apologies in advance for the inconvenience.

Secondly I have removed my old Gallery instance. I’ve been using Gallery 1 for ages because I don’t see a good reason to upgrade to Gallery 2 which I consider HIGHLY inferior due to the reliance on a RDBMS. As a replacement I have redirected all /gallery/ URLs to my Flickr page.

Thirdly, as a result of moving to Bazaar from CVS, much of the code that has been linked in previous entries is now over at repo.ub3rgeek.net.

Finally, for anyone with a BlackBerry that would like to get the SMS messages off the device so they can save the memory without losing the messages, or just to be able to use Unix tools to manipulate the messages I have written a small Python script which can be used, alongside the Barry Project‘s btool program to extract the SMS messages to a set of files sorted by Phone Number of the person you are corresponding with.
You can get the script at http://repo.ub3rgeek.net/branches//top/bbtools/files. Feel free to let me know if you find it useful.

Revision Control Revision

I’ve been using CVS for revision control for quite some time for a number of different things here, not the least of which being a public repository of scripts and miscellany that I have tossed into the Internet for public consumption. Recently I have started to feel the desire to change to one of the newer version control systems out there. Git and Hg smell a lot of ‘ooh shiny’ syndrome and we use Subversion at work so I was sort of naturally drawn towards Bazaar. It doesn’t help that I’m a huge fan of Canonical and Ubuntu, and as that seems to be the VCS du jour over there these days I figured that this would give me a good excuse to learn it.

The moral of this story as I’m sure no one cares why I chose which VCS is to say that any of the code hosted on my CVSweb and linked to in one of my previous posts is likely to have moved to the new loggerhead interface over at repo.ub3rgeek.net. If you happen to follow me on FriendFeed or watch the home page here, you will see updates from my bazaar commits as they happen.

Fancy, eh?

Processing my digital photos

I’ve finally gotten to the point where I HAD to do something about my digital photos. The ~/Photos/lumix folder has over 1100 images in it and it’s just painful loading it up anymore. Nautilus takes like 45 seconds just loading the thumbnails for the folder during which time the scroll bar is jumping all over the place making navigation impossible.

So here is how I finally made this work out in a really manageable way.
1) I wrote a Python script image-process.py that moves the image file to a place in the format of CCYY/MM_Month/CCYY-MM-DD_HH-MM-SS_Exif Make-Exif_Model.jpg, so a photo from my Panasonic Lumix called P1020411.JPG gets moved to 2008/12_December/2008-12-21_19-08-17_Panasonic-DMC-FZ7.jpg

2) Setup Gnome to run my script upon mounting of a device:
Under Ubuntu, navigate to System -> Preferences -> Removable Drives and Media then under the Digital Camera section I enabled “Import digital photographs when connected” and pointed the Command to my script using the %m macro and my script’s -d flag. This passes the location that Gnome mounted my camera at to my script as a directory and lets it do the rest.

3) My laptop is backed up periodically to an external hard drive, so all my photos are more or less safe, barring a catastrophic failure of both my laptop’s hard drive and the external drive.

I am hoping to get bbtrack working on my BlackBerry so I can save GPS tracks, then using the Exif Capture Date and the GPS track my image-process.py can geolocate the photos automatically. I’m sure I will post an update if I ever get that working.

I’d be interested to hear how other people solve this particular problem. Also any suggestions of a decent photo viewing application for Gnome that I can point at my directory hierarchy, that won’t go through and try to move the photos into it’s own crap like F-Spot does?

Why social networks are lying bastards.

You’ve seen it I’m sure. Myspace purports “a place for friends”, Bebo says it “is a social media network where friends share their lives and explore great entertainment” and Facebook says it “helps you connect and share with the people in your life.”

They all are lying. Granted it may be a lie of omission, but that’s a lie just the same. What they are forgetting to tell you is that they are little islands of popularity that are only as useful as the percentage of your friends that they have already tempted onto their bosom. Which means that most of us are constantly taunted to join up to one or more of them to connect to our friends, and we can’t shake the nagging feeling that we’re missing out on re-connecting with what’s-her-name, the cutie you snogged back in grade 12 and have been wondering about lately when the lights are low. And pretty soon my friends we have 2 or 3 or 12 different sites maintaining lists of our ‘friends’ with varying (infuriating in some cases) levels of overlap and omission.

Everyone claims the web 2.0 is all about communities and openness and sharing and blah blah blah lies. What we really have is a bunch of people who see what they think is a problem. And they have this grandiose idea of how to organize us and the stream of information we generate daily into useful and manageable pieces. Granted some of these apps work great, but the problem is that they all either seek to lock your entire online world into their framework, or do one thing really well… but only that one thing. The sheer hubris of FaceBook and MySpace trying to lock your friends, and pictures, and videos all up inside their walls is staggering and yet the omission of anything other than the one thing that Flickr of Vimeo does leaves them somewhat lacking. Someday, probably soon; someone will come invent something better than whatever the social network site du-jour is and users will go running abandoning their old profiles to become husks as a snake shedding its used skin. That is life in the big bad void. MySpace becomes GeoCities becomes Prodigy and the cycle repeats.

What I want to see some pervasive, accepted way of having one giant friend list that is bound to me that I take from site to site so I don’t EVER have to rebuild it (or worse: harass my friends to join the new hotness so I can ‘friend’ them.)

I don’t care what it is under the covers be it FOAF, or XFN, or some other thing that someone else invents, or hell all of them. I just want one goddamn friend list that I can take everywhere.
Granted this means something like XFN is going to have to get into bed with something like OpenID, but damnit the technology is there and wouldn’t you rather keep all these little husks of Internet identities all tied together and relevant?

I started thinking about this a little while ago and was set off by an update by jwz on his livejournal. My irritation at this phenomenon became even worse when I started to organize all the profile pages I had out there. It is getting so cluttered that most of the information out there about me is old, unloved and abandoned.

Something really needs to be done.

PS, you can follow me on:

• FriendFeed
• LaunchPad
• Twitter
• Flickr
• LinkedIn
• FaceBook
• Vimeo
• last.fm
• Slashdot
• Google
• Google Reader

to name a few…

twitterpy and lastfriends

I keep forgetting to write something about these guys. I wrote a pair of Python scripts to watch my Last.FM and Twitter friends and pop up a little bubble when something new happens.

There are Ubuntu packages available in my PPA over on Launchpad for both of these guys that should work just fine on 8.04, and hopefully will be updated to 8.10.

For those of you not using Ubuntu, my CVS repository has the scripts themselves that you are welcome to try out.

~mernisse’s Launchpad PPA
my CVS web interface

Mutt and Google Calendar

I’ve been meaning to throw this out there for a bit. I wrote a little python shim to connect mutt to Google Calendar. My particular use-case is as follows. I use fetchmail to connect to a Microsoft Exchange Server at work which delivers mail into an IMAP account that I check with mutt. Appointments and conference calls show up as .ics files attached to messages. I use Google Calendar connected to my BlackBerry to keep track of all the stuff I do so I need a quick and easy way to get those events out of mutt into my Google Calendar.

Enter ics-gcal.py.

I associate this script to the ics / vcs files and simply exec the attachment from within mutt. This adds it to my Google Calendar.

You can find the script in my CVS web repository.

I’d be super interested if anyone else finds this useful.

Sirius Radio on my PS3? Yes Please!

So it looks like the svn version of mediatomb (which will become 0.12.0) now supports streaming PCM audio to the PlayStation(R) 3! This is super good news. So I went and updated my Python shim between Sipie and Mediatomb to output PCM.

You can get mediatomb-sirius.py from my CVS repository by visiting this url:

http://bagend.ub3rgeek.net/cgi-bin/cvsweb.cgi/misc-scripts/mediatomb-sirius.py

Once I have the config sorted out, I will post more.

Edit:
I got FLAC streaming working, and shot a quick video of it in action.

Showing off Mediatomb transcoding FLAC to the PlayStation 3 from Matthew Ernisse on Vimeo.

Edit 2:
I got the sirius stuff working finally, had to overcome a few really annoying behaviours of all the bits involved, but it seems like it is working pretty well. You can head on over to the http://www.ub3rgeek.net/wp/sirius-on-the-playstation-3/ page for more details and another video.

Things I demand…

Dear Roku / Sony
Hulu.com support now please. If the PS3 gets it first I may still buy a Roku box once NetFlix has a better selection of stream-able movies. Though if the Roku box gets Hulu first I’d buy the thing RIGHT NOW. And while I’m ranting, Sony, you still have to address all the other things I bitched about…

BBC iPlayer Guys:
I would subscribe to get access to iPlayer right now. Please figure out a way to take my money!! I would also like PS3 streaming akin to the Wii one, it shouldn’t be hard, the dude at PS3 iPlayer made it work.

Also: a pony.

BlackBerry reboot loop :(

So last night I went and cleaned a bunch of stuff off the crackberry, like midpssh, which the 7100t is too slow to really use, and a few advanced themes (like the Dimension Zen theme) which were also slow on the 7100t. I rebooted to finish the removal and the crackberry went into a reboot loop. Terrified that I had hosed the system up big-time I tried to recover it using the awesome howto on crackberry.com (http://crackberry.com/blackberry-101-lecture-12-how-reload-operating-system-nuked-blackberry). Unfortunately vmware-player on linux takes too long to detect the USB device that I couldn’t interrupt the boot process and re-flash the device. Thankfully vmware on the Mac was quick enough to blow the OS back on.

Now to restore all my settings and themes…

Roadrunner and IPv6?

So I was at my buddy’s house and he has Roadrunner hooked up to an Apple Airport Extreme and I just noticed that my last login to my colocated server at the Frontier CyberCenter was from a IPv6 IP. The EUI-64 part matches my laptop’s MAC address. The prefix isn’t familiar to me. It was: 2002:4a4a:ca8b:0:213:2ff:feba:83cc

Is Roadrunner quietly rolling out IPv6? It would explain why my ssh(1) connections were flaky.

Edit:

Screenshot ftw.

Transclusioning Images without permission is bad, mmmkay?

Why do people think that it is ok to steal my bandwidth to host images for them? I guess this internet thing must be free.

Another dork thought it was a good idea to use an image off my website in their forum profile. http://forum.dzmusique.com/discussion-generale/se-separer-a-cuase-du-voile-t1298.75.html

Not only that, but it happened to be a 1280x1360px image that was being sized down to 100×100 by the forum, so each page hit was 350kb of data off my server. Thanks dude, how about instead you get a 680byte .gif called “i-suck-balls.gif”

For those interested, the following code block is responsible for this. Thanks apache and mod_rewrite:


RewriteCond %{HTTP_REFERER} habboxforum\.com [NC,OR]
RewriteCond %{HTTP_REFERER} forum\.dzmusique\.com [NC]
RewriteCond %{REQUEST_URI} !^/images/i-suck-balls.gif [NC]
ReWriteRule .*\.(gif|jpg|png|jpeg)$ http://www.ub3rgeek.net/images/i-suc
k-balls.gif [L]


Why the computer should think for you.

I am tired of thinking for the computer. So instead of having to learn the gritty irritating useless internal details about every single piece of software I need to use I think it makes more sense to have a set of sensible defaults that work for as much of the user base as possible and then if need be allow customization away from the defaults to encompass the rest of your potential users.

A few (4?) years ago I wrote a RSS feed reader because I wanted to think for the computer and didn’t like any of the readers out there at the time. Similarly I have found myself writing things instead of using software that is out there for trivial tasks because the software that was out there was too infuriating, or complex, or just broken by design.

Working, as I do now, in an environment that needs to scale beyond that of the typical open source geek I find the short-sighted-ness of much of the community depressing. So much otherwise good software is written to ‘scratch an itch’ and is only engineered to work in the author’s basement. Scaling to a million users and many tens to hundreds of megabits/sec of traffic in a lot of cases is a huge headache where we have to work around shortcoming after shortcoming.

As such I’ve started to look for services that do most if not all of the work for me that I can just tap into and think less. The front page of www.ub3rgeek.net is a perfect example. That is 30 or 40 lines of python using the FriendFeed API to aggregate all the various stuff I do on the internet down into one easily manageable stream of data. I don’t have to think about sorting the RSS feeds, or tracking down the links, I just parse some standardized markup from FriendFeed and I am done.

More software needs to be written with the understanding that the user doesn’t want to have to think, that’s why we use the computer.

PlayStation 3 with MediaTomb on Linux and OpenBSD

I have posted a few journal entries on this subject, but this page is here to be sort of a more general overview. There is some detailed information in these posts if you are interested.

1. How to make the Playstation 3 co-exist with Linux and OpenBSD
2. More Playstation 3 Media Fun

Nat Type 3
I run an OpenBSD router, and some port forwarding needs to be done to make sure that the PlayStation 3 can properly communicate with the internet. I have verified the following works for WarHawk, including voice chat, as well as the PlayStation Store, and the voice chat built into the XMB.

Allow Outbound (only necessary in some cases with extremely restrictive firewall rulesets)
TCP: 80, 443, 5223 and UDP: 3478, 3479.

Port Forward inbound
TCP: 9293 and UDP: 3658
TCP/9293 is the port for Remote Play, and 3658 is for other users to connect to you. If you do not forward 3658 you will get the dreaded NAT Type 3.

Media sharing with MediaTomb
Since MediaTomb is the only Linux UPnP media server that I have found that can do transcoding I have been using it for some time now. It is highly configurable and fairly stable. I tend to track the SVN version fairly closely, but now that 0.11.0 is out the transcoding features are available to anyone. Most of the new transcoding features are really well documented on the MediaTomb site, so I won’t reproduce my entire config here, but my transcoding stuff looks like:

  <transcoding enabled="yes">
     <mimetype-profile-mappings>
        <transcode mimetype="video/quicktime" using="vlc-sh"/>
        <transcode mimetype="video/x-matroska" using="vlc-sh"/>
        <transcode mimetype="video/mp2p" using="ffmpeg-sh"/>
        <transcode mimetype="video/ogg" using="ffmpeg-sh"/>
        <transcode mimetype="video/mp4" using="ffmpeg-sh"/>
        <transcode mimetype="video/avi" using="ffmpeg-avi"/>
    </mimetype-profile-mappings>
    <profiles>
        <profile name="ffmpeg-sh" enabled="yes" type="external">
            <mimetype>video/mpeg</mimetype>
            <accept-url>no</accept>
            <first-resource>yes</first>
            <agent command="/staff/mernisse/bin/ffmpeg-tr" arguments="%in %out"/>
            <buffer size="6144000" chunk-size="131072" fill-size="2048000"/>
        </agent></profile>
        <profile name="ffmpeg-avi" enabled="yes" type="external">
            <mimetype>video/mpeg</mimetype>
            <accept -url>no</accept>
            <first-resource>yes</first>
            <agent command="/staff/mernisse/bin/ffmpeg-tr" arguments="%in %out"/>
            <buffer size="6144000" chunk-size="131072" fill-size="2048000"/>
            <avi-fourcc-list mode="ignore">
                <fourcc>XVID</fourcc>
                <fourcc>DIVX</fourcc>
                <fourcc>DX50</fourcc>
                <fourcc>WVC1</fourcc>
            </avi>
        </agent></profile>
        <profile name="vlc-sh" enabled="yes" type="external">
            <mimetype>video/mpeg</mimetype>
            <accept-url>yes</accept>
            <first-resource>yes</first>
            <agent command="/staff/mernisse/bin/vlc-tr" arguments="%in %out"/>
            <buffer size="6144000" chunk-size="131072" fill-size="2048000"/>
        </profile>
    </profiles>
  </transcoding>

Some notes about that config:

• I transcode MP4 – You may not have to if your MP4 files are all playable on the PS3, I would try it out without first and then transcode if necessary. I have more files that don’t work than files that do.
• Audio transcoding doesn’t work right now on the PS3 – A number of weird things that the PS3 does is keeping this from working, hopefully the MediaTomb guys will nail it and make it work, it sounds like the on the horizon ‘built in transcoding’ might fix this issue, but that is still a bit off.