Admin Brought His Drill To Work, Destroyed Disks And Crashed A Datacenter

Who, Me? Welcome once again to “Who, Me?”, the reader contributed column in which we invite Reg reader to tell tales of the times when they got things very wrong.

This week, meet a reader we’ll Regomize as “Slim” who told us of his time as head of IT in Asia for a global trading company – the sort of outfit that plays the markets and needs its tech to run 24 x7 without even the briefest glitch that could slow down a client’s trades.

I still have nightmares about those disks

As Slim built a new team, head office sent him a server admin he described as “a guy who half-assed everything and cleverly managed to always figure out the laziest way to do something.”

Slim needed to expand his team because his employer was growing and had just leased a set of racks in colocation facility that had previously housed telecoms kit. “It was old, but physically close to the Futures exchange – and distances mattered when trades were measured in micro-seconds,” he told Who, Me?

The colo had another problem: The racks were built for comms kit, and the screw holes were too small for the sliding mounts that held servers.

Slim gave the lazy server admin the job of sorting this out, and thought he was doing a decent job as purchase orders for new hardware soon arrived. Once the hardware arrived, the lazy admin’s reports about configuration and testing indicated things were going well, if inexplicably slowly.

The project reached the racking and stacking stage, and the lazy admin told Slim he was about a week away from having all the hardware in place.

A few days before that deadline, Slim received a red alert: Trading systems had gone down – during business hours – and the company was dead in the water.

Slim immediately suspected the lazy admin, so called and demanded to know if the racking and stacking effort had gone awry.

“I promise I’m not touching anything,” came the reply.

He was lying.

Slim learned that the lazy admin knew about the too-small screw holes on the rented racks but left it too late to hire someone with the skills and equipment to rightsize them.

The lazy admin decided to do the job himself, but didn’t fancy doing it overnight as is sensible for this sort of dangerous job.

“He brought his own drill from home, threw a bedsheet over the top server so metal flecks from drilling won’t go into it, and started drilling these rack holes during trading hours,” Slim told Who, Me?

To make matters worse – if that’s possible – the lazy admin set his drill into Concrete Mode, which in this column’s experience usually means the machine both spins the bit at furious speed and pumps it back and forth in a percussive hammering motion.

The result?

“EVERY. SINGLE. HARD DRIVE. DEAD,” Slim all-CAPSed in his mail to Who, Me?

All the file servers in the datacenter died, too. So did the machines that ran a futures trading gateway.

Slim learned about the drilling because the datacenter manager caught the lazy admin in the act – and even had photos of the incident.

“They made him stop as soon as they heard the drill, but alas it was too late for those poor disks,” Slim told Who, Me?

“I still have nightmares about those disks,” he added, and told us he left the job shortly afterwards.

Slim never learned how the lazy admin’s career progressed, but told us the dolt with the drill at least had the good sense never to ask for a reference.

“I told you he was clever,” Slim wrote.

When Slim wrote he told us “I don’t know if this is a ‘Who, Me?’ story, but it seems like it would belong in The Register’s memory banks. Just get it out of mine.”

If you need to get a similar story off your chest, finding relief starts with a click on this link as it opens an email to Who, Me? Once we receive your yarn, we will consider sharing it on a future Monday. ®

Original Source