Techie Went Home Rather Than Fix Mistake That Caused A Massive Meltdown

Who, Me? Welcome to another working week and therefore another installment of Who, Me? – The Register‘s reader-contributed column where you confess to making a mess and somehow find safe egress.

This week, meet a reader we’ll Regomize as “Stuart” who hoped his career would see him pursue pure research but found himself working as IT manager for what he described as “a very young startup spun out of a local research center to commercialize a novel gene analysis technique.”

The IT team was small – Stuart plus a couple of contractors – but that meant our hero had “very nearly free rein in designing the systems, so the job was anything but boring.”

He duly made security keys mandatory for authentication, implemented “paranoia-grade network partitioning” – native IPv6-only, naturally – and created an encrypted, append-only off-site data archive.

“You name it, if I could show it made sense I got to implement it,” Stuart told Who, Me?

Stuart spotted one project that made a lot of sense. The startup kept a lot of samples in freezers and was contractually obliged to keep them safe for years. Creating a system to monitor that the freezers were in working order was an obvious win.

Lab-grade freezers are built to make this sort of project doable because they include sensors galore, employ the Modbus serial communication protocol, and can even pack an RS485 serial port to help the machines share sensor info with the world.

Stuart therefore pulled some extra wiring through the cabling ducts, configured a Raspberry Pi 1B to ingest data, and wrote Python scripts to feed info into the startup’s monitoring infrastructure.

The resulting system was very simple. If a freezer door was left open too long, alerts went off.

“It proved hugely successful and popular among the lab techies because it was all too easy to leave those doors ajar, so we set out to extend the scope,” Stuart wrote.

“It was only then that I realized that the specs provided by the freezer manufacturer were not in total agreement with reality on the freezers we had,” he added.

Stuart tried to make sense of it all. One Friday, he accidentally tweaked a setting so the freezers monitored temperatures measured in Fahrenheit instead of Celsius – but didn’t change alarm thresholds accordingly.

That has obvious potential for mayhem given that 32° Fahrenheit is 0° Celsius.

Within seconds of unwittingly making this mistake, alerts flashed across Stuart’s freezer-monitoring dashboards.

He had no idea of the cause, but as the freezer doors were closed, he didn’t think it was important enough to warrant an immediate fix.

Then another emergency erupted and consumed the rest of the Friday. Stuart figured he could fix the freezer monitor on Monday, so he turned off his alarms and went home for the weekend.

The next day, an electrical contractor made a worse mistake.

“He managed to somehow trip every single circuit breaker in the building and left without saying a word to anyone,” Stuart told Who, Me?

The problem went unnoticed until Monday morning, when Stuart arrived at work and was relieved to find nobody blamed him for the mess.

“Our lease specified redundant power for the freezers and required that weekend electrical work be announced well in advance,” he told Who, Me?

Recriminations therefore focused on the landlord, not Stuart’s tech.

This story has a happy ending because, despite the meltdown, the samples in the freezers survived intact.

“Even so, my very first thought after coming to work on that day and learning what had happened was ‘Oh, BUGGER,'” Stuart told Who, Me?

Has someone else’s mistake saved you from being blamed for your own errors? If so, click here to send email to Who, Me? We always appreciate fresh stories to feed you each Monday. ®

Original Source