Melting Servers
In the peak years of SETI@home all our servers were racked up in a nearby closet which we retrofitted with barely adequate air conditioning. Due to leaks the cooling system would slowly weaken over time and required occasional tune ups. So we set up various sensors and had a beeper that went off if the temperatures rose too high. The systems typically ran at 50 degrees Celsius. We'd get warnings if they shot above 60.
We'd trade off who had the beeper and thus who was responsible in case of crisis. I had possession of it one weekend afternoon when it started alerting with the message that temperatures were at 60. A minute later it beeped again. They were now at 62. Uh oh. This wasn't a slow leak, but something more like complete cooling system failure.
I shut down as many servers as I could remotely, but it was difficult to do them all or determine their current states (given slow network speeds and machines hanging on a nest of cross dependencies, plus we didn't have IPMI or web-enabled power strips back then). Anyway it would be much faster to deal with this in person and it usually took me under 15 minutes to get to the lab.
So I jumped in my car and sped off. Temperatures now at 64. Fearing typical weekend traffic on highway 24 I took surface streets, driving into Berkeley and up around campus. The path to the lab was a steep road that rises above the stadium.
As I approached I discovered a college football game was happening at the moment. The traffic quickly worsened as football fans were struggling to park. I zig zagged in an angry panic around people and cars until I hit a road block.
I asked the campus policeman preventing my passage, "hey I work at the space lab - Can I get through?"
"Sorry," he said, "the road is closed during the game."
I pulled out my campus ID and explained this was an emergency, and again he simply apologized. The beeper warnings continued, now showing temperatures well above 70.
I yelled, "I have a million dollars worth of computer hardware melting right now! I need to get through!"
No dice. He wouldn't budge. He suggested I approach by way of the Claremont Hotel. Completely frustrated I swung the car around as swiftly as noisily as possible and screeched away. The detour added about 25 minutes to my journey - felt like an hour.
When I arrived at the lab I ran upstairs and threw the closet doors open. I was greeted with a chorus of beeps (the cries of servers in pain) and the smell of melting plastic. I quickly got the remaining computers powered off and left it that way for the rest of the weekend.
On Monday we got the air conditioning serviced and turned everything back on. We only lost a few hard drives (all safely in RAIDs), but otherwise no other obvious damage. Still.. thanks, college football.