Recovering from Disaster

Sat, Jan 12, 2008 @ 17:28

Some people may not call it a “disaster”, but in today’s digital world the loss of digital data, is in most people’s minds, a disaster. Loss of a carefully crafted music library is one thing, but what about family photos that can never be re-created, or the years of work invested in a dissertation? Obviously there is a need to safe guard this data.

The following is an account of my experience with my 15” Powerbook G4 failing 31 December, 2007. I awoke that morning not suspecting anything was wrong, looking forward to ushering in the new year, although not looking forward to ending my vacation. However, when I tapped my trackball to wake up the screen, I was greeted with… nothing. No response. Rebooting brought my computer back on, but things were not right. In addition to constant waiting for file system operations, some of my desktop scripts were yielding unusual results. Then finally it hung completely again and I was force to do a hard power-down. All subsequent attempts at booting resulted in the prohibitory sign (grey circle with a slash through it) appearing. It wouldn’t boot at all. My computer was dead.

Triage

My first step was to determine what was causing the boot issues. A cursory look online told me that sign I was getting indicated the computer couldn’t find the boot volume. OK, I tried to boot from my Leopard install DVD. It would give me the same result. Nothing. I tried all the standard tricks: I reset my PRAM and my NVRAM and reset all that I could in Open Firmware. Holding Option (⌥) during the boot process provides a list of available boot sources. Mine showed both the hard disk and the Install DVD, both of which it identified correctly by name. One, both, nothing.

Without DVD booting, I feared that the logic board might have suffered. The glimmer of hope was that I was able to run the hardware tester on my original system discs. Quick test. Extended test. Both said everything was fine. Both also completely skipped my hard drive in their scans. An interesting development. Unfortunately at that point, I had to catch a plane back to Bermuda. I packed up the carcass of my computer and all my accessories, it was a sad moment.

Returning to Bermuda, I was getting depressed about the computer. I didn’t have the tools required to really do anything about it, and that was frustrating. So after a couple of days I brought my dead laptop into work. I figured, at least there I would have the tools to take it apart if it came to that. On Friday 4 January, I even tried to boot off of a colleague’s Macbook Pro. Again, no luck. Saturday had me returning to the office since that was where my computer was. I brought a couple of blank CDs this time in hopes of downloading and burning a Linux LiveCD or a recovery CD for PowerPC. I had good experience with these types of discs for problems with other people’s computers in the past.

Even here I had very little success. I could get into a boot loader, but the systems would complain of an error reading from the filesystem on the CD. My colleague Joseph, who is a serious Linux tinkerer helped me a lot that day. In addition to examining what little information we could glean from the LiveCD, he even went out to the store on his scooter when we realized that we didn’t have a Torx screwdriver. I remember clearly when he returned, success written on his face, and said, “It’s surgery time.” Yes, it was certainly looking that way.

Surgery

This is the part that separates the boys from the men. Or perhaps I should say those with warranties and those without. Any power user can try holding a few different key strokes when booting or even burn an ISO of a recovery CD. Some might even brave taking the side of their case off a desktop machine, but it takes a special breed of dedication to pry open a notebook computer. Fortunately I have experience with laptop autopsies. I’ve assisted on one involving a Sony Viao, and then spent several hours resurrecting an iBook G4 with a dead hard drive–similar to my problem. Unfortunately, the ardour of the iBook opening was actually dissuading me from taking a peak inside my own computer. I had heard that Powerbook were easier to open, but I was still reluctant.

Reluctant that is, until Joseph returned with the set of allen keys which was the missing tool in my kit at that point. I’m not sure who at Apple decided that it would be better to use some unusually obscure screw head for only two out of twenty-three screws that I needed to remove–but they were wrong. The remaining Phillips size 0 screws were small, but still not unusual in the IT circuit. Fortunately, as with my iBook experience, I found the Internet to be full of wonderful resources, including step by step guides to open Powerbooks. I used one from ifixit which was targeted for those users looking to replace a hard drive… which was pretty much what I was going after, except I didn’t have a replacement drive.

Deftly removing tiny screw after tiny screw, I removed enough to be able to lift the top cover. There is a bit of prodding required around the DVD drive, which was the worst part of the iBook as well. This was far easier because at that point you have the rest of the top pried up. A single narrow ribbon cable is all that connects the top case to the logic board. Keep an eye on this, because the connector just snaps in place lightly on the logic board, so if you lift the top off too far, it will disconnect. It doesn’t seem to harm it, but if you don’t reconnect you won’t be able to turn the computer back on. Another tapered ribbon cable connects the IDE connector on the drive to the tiny receptor on the logic board. You will also notice a fair bit of yellow tape holding various connectors onto the logic board. A low-tech solution, but it seems to work. I needed to peel back some of the tape before I could disconnect the hard drive from the computer. Once I had the hard drive out, I was ready to try again. Obviously I wouldn’t be able to access my files, but would I be able to boot from a CD?

In short, yes! Instead of completely reassembling the computer without the hard drive, I just gently set the top case on the bottom part of the computer. I turned it on and it started the boot process. Only this time it was able to read from the CD and boot fully. Oh, this was definitely a start towards something positive. I shut down, unplugged and once again peered under the hood. I decided to try plugging the drive back in, just in case the connection to the logic board was loose. That wasn’t the case, as soon as the drive was reattached, the problem returned.

Joseph pointed out that it still might be the ribbon cable or the hard drive. However, thinking back, the hard drive is really your best bet because they are so much more likely to fail. At this point it was late Saturday evening, and I wasn’t going to be able to buy a replacement or find any way to test the problem further until Monday. So I left my countless screws and dissembled laptop on the side of desk and headed home. I was back into work on Sunday for my fix of the Internet and I was feeling bolstered by the previous day’s success. Although my computer was in pieces beside me, I was feeling that the end was in sight.

Monday morning, I ducked out from work to purchase a USB enclosure for 2.5 inch drives. Since no one had one for me to borrow, I figured it would be a small price to pay to test the drive further. I hooked my drive to the IDE connector of the enclosure and plugged it into another co-workers Macbook Pro. It mounted but was only showing up as 5GB total size, far short of the actual 100GB. No, something was wrong with the disk. When I tried to access it from Disk Utility on my Leopard Install Disc, the enclosure started making a loud beeping noise. Not good. Fine, I would buy a replacement. It was end of the work day at this point and I headed back to the store where I got the enclosure. They had one 2.5 inch EIDE drive left! A 120GB 5400rpm Western Digital. It was 20 GB more than my dead drive and it was at a decent price. I bought it.

Returning and plugging it in, I tried booting from the install DVD again. It booted. This was it, the atmosphere at my desk was palpable. I checked in Disk Utility that the drive was visible; yes it was. I clicked Install, but it didn’t show up as a target. Damn. So close. Then I went back into the Disk Utility and formatted the drive with a HFS+ Journalled partition. This time the partition showed up as an installation target. I clicked go, and I was about one hour away from a functioning system. A small miracle was being enacted at my desk. Leopard installed while my computer was still in pieces and booted without problem. I shut the computer down and sewed her up. Ready to take back home and nurse back to full health.

Recovery

As I mentioned in my introduction, digital data is the key asset nowadays. I had my computer running, but my former hard drive was toast. No data.

Enter Time Machine. Time Machine is the new backup software system included with Mac OS X 10.5 Leopard. And it is great based on one simple reason. People want to use it. If you plug in an external drive to a Leopard system for the first time, it will ask you if you want to use it for Time Machine. You click Yes and you are done. It just does the right thing. It creates a full incremental backup of your drive without getting in your way. Granted, if your drive is full, the first backup does take some time, but after that, backing up is quite transparent. Fortunately I had started using it when I installed Leopard in early November. I’m glad I did, it literally pulled me from the fire on this one.

When I got my computer home, I booted it up and it came on as expected. Perfect, the reassembly hadn’t caused any problems. The moment of truth, I plugged in my external drive and it mounted without a problem. And there it was, a perfect replica of my old hard drive sitting there… several times over. I should mention at this point, that when I was finishing up the installation of Leopard, the installer asked me whether I would like to migrate old data from, among other options, a Time Machine backup. This means that you could basically startup again where you left off. I’m not sure how the Migration Assistant actually assists you in choosing what you want brought over, I’ve always done it manually. I’ve heard mixed reviews in the past about that process breaking a few things, but if you want to start up on a new installation where you left off before, it is likely the quickest and easiest way. But as I mentioned in a previous post, I was interested in moving over slowly, only taking things that I actually need.

Of course, moving over this way isn’t for the feint of heart either. Documents and files are easy to bring over, as are most applications. Some applications however can have extensive configuration that you have built up over time. All of this trickiness boils down to one primary folder: ~/Library/. This is the folder which holds all of your preferences, configuration files and in some case even application data. A wholesale copy is defeating the purpose of a clean new system, but what is important?

If you take a look in a clean install, there are a number of folders that are in there by default. The list can be daunting but there are some important ones to consider. There should be a sub folder of the library called Application Support. This is where applications can store miscellaneous bit of information and data to support their operation. Things that are specific to a user, and that shouldn’t be clobbered when an upgrade is installed. Some software will have a folder directly inside the Library, like Safari, but in general, they should be in Application Support. The other folder inside the library to make note of is Preferences. These will be revisited often.

I have been keeping track of what software I have reinstalled and what procedures I have gone through to recover any data I need from my backup. I’m not going to talk about all of it here, but some of the initial parts might apply to most people and could be helpful.

Things I’m Restoring to Philyra

At this point I have my recovery command centre up and running. Ready to proceed with some of the more heavy lifting and tweaking. Obviously the default applications that come with Leopard are already on my system at this point: iTunes, Mail, etc. It is Mail which is my next goal.

At this point I am happily running along with my development environment pretty much all set up the way I like it. A computer is a dynamic entity with new data constantly arriving and old data being lost in a collection of magnetized bits. As my last post about Ruby alluded, there might be a new exploration beginning into the world of Rails, but time will tell.

Summary

It was a rough week without my computer. Being without Internet is nothing compared to being without a computer. But if anything can be learned from this experience, it is that regardless of how careful you are, disaster can strike, and you are always going to handle it better if you are prepared. I had backups, Time Machine saved me on that one, and you better believe that I will continue to use Time Machine and advocate it to anyone who will listen.

I only hope that this article is useful to someone. I suspect I’ll never know, but it was cathartic to write and might be somewhat enlightening or entertaining for my usual readers. Cheers.