By Lyle Bateman
Contributing Writer, [GAS]
Its been too long since I wrote something for Geeks Are Sexy. I’ve been in a fairly non-geek period of my life recently, focusing on my stand-up comedy side rather than my techie side. In that spirit, I thought I might combine the two sides a bit with some (hopefully humourous) recollections of my life as a system administrator in Lagos, Nigeria from 1993-1998.
I was hired by a seismic processing company based in Calgary, initially to get their computer operations department up and running 24/7. For those who’ve never worked in a data intensive area, seismic processing involves truly mind-boggling amounts of information, far more than even the best disks of the time could hold. Data is stored in huge libraries of magnetic tapes, and for processing to proceed smoothly, a staff of people who’s only job is to find and load the correct tapes at the correct time was an integral part of the operation. The volume of data is so massive that even with the fastest computers available, processing of 3d seismic volumes often took weeks or months of 24/7 operations. Large marine surveys could literally span years of processing time, and while a certain amount of automation was available, it simply wasn’t cost effective for most processing houses to fully automate the tape mounting procedures.
My first look at the computer room I was set to manage was a bit surreal, to say the least. I’d just traveled through Lagos in a car, from the airport to our office/accommodation, for the first time, an absurdly surreal experience in its own right, and arriving at the “office” (which was really just converted from a large house), I dropped my bags in the manager’s office and poked my head into the “computer room.” Converted from the grand dining room of the house, the room itself was fairly decent, though it was apparent immediately that air conditioning was going to be an issue. More surreal, however, was the fellow who had shipped the equipment and set it up, hunched over an old RM05 VAX disk drive unit, working on it with a cigarette dangling out of his mouth and his Irish setter laying beside him. Needless to say, I was picking long red dog hairs out the equipment for years.
My job when I arrived in Lagos was to get the basic operations running, set up the procedures required for the full-scale operations, and hire and train staff to man the operations centre 24/7. Tied in with that, of course, was maintenance of the computer systems so that the operators had functional systems to work with. One of the biggest problems I had at the beginning turned out to be something that I could hardly have been prepared for based on my previous experience in North America … stable power.
Initially, the company decided to run the computer room from local power, with a UPS and conditioner to allow us to turn on a generator when the power went out, which was a pretty common occurrence in Lagos. Over the course of the first three months, we spent almost 3 weeks of it trying to fix the UPS, as batteries and circuits continually had problems. Finally, we decided that it probably wasn’t wise to even bother trying to condition a power source that was only available about 12 hours a day, and varied from its expected 220V output in the range of about 290V down to 160V. Is it any wonder we were blowing circuits in the UPS?
Once we decided to run the computer room from a generator 24/7, my life as a maintenance guy got a whole lot easier, not only was there no longer a panicked switch over to generator power when local power went down, but the life span of my disks and servers increased dramatically with no sudden power cuts from UPS problems. Running a diesel generator 24/7 introduces its own set of problems to deal with, of course, not least the need for maintenance. Turned out, whatever stress I removed from my life from dumping local power, came right back with the bi-weekly generator maintenance sessions that required a complete shutdown of all systems. It was a nice example of “be careful what you wish for …”
Even with a generator powering us 24/7, we still had our power issues to deal with, sometimes from very unexpected sources. It took me about 6 months to get the operations running smoothly, with a staff trained well enough to work unsupervised, during which time I was often working 18 hours a day in some form or other. When I finally was able to relax a little bit, I tried to make use of some of that free-time by catching up on lost sleep. I remember being woken up one night by my operator to be informed that there was “no power.” I could hear the generator running outside (its pretty hard to miss the sound of a 120KVa diesel generator under your window), but sure enough, I couldn’t get any of the lights, air conditioners, or computers in the computer room to work
The obvious culprit, of course, was a blown fuse, so I made my way to the fusebox with my flashlight. The electrical systems over there were pretty primitive, and the fuse box in the house we converted had exposed positive and negative terminals. As I approached the fusebox, I could smell an acrid, burning smell from it, but when I opened it, I found something I didn’t quite expect … a small lizard, quite heavily BBQ’d, stretched between the wrong two live poles, shorting out the whole system. Now, up to that point in my career, I’d had a lot of experience dealing with bugs in my systems … that was the first (and hopefully last) time I had to deal with lizards in the system.
It was an interesting place to spend a few years, but as a computer guy, I ran across problems I couldn’t really have imagined before I went there. In hindsight, I’m shocked it all ran as smoothly as it did, in a tropical country that bordered the Sahara Desert (for the record, sand and disk drives are a VERY bad mix, and the humidity and heat certainly didn’t help matters). I wouldn’t want to give up my time in Lagos for anything, but I can also say I’m not rushing out to re-experience lizard-related power problems any time soon.
Tags: lagos, nigeria, seismic processing