An investigation concluded that a power line next to a vertical ventilation overheated, melted the wires and created a short circuit electrical arc. The power properly went off but a flammable outer lining in the ducts caught fire. The insulation correctly slowed down the fire, but the plastic on the wires caught fire. The wires went up into the data center and carried the flames inside through the ducts. All in all, the fire preventions worked to some extent but still several million dollars in servers got ruined and rendered inoperative. A whole rack burned until the halon triggered and mostly snuffed the fire, but the furniture in an office below, adjacent to the duct lines burned into a crisp further damaging the data center with smoke and heat.
It was the most surreal week on that job, taking all the equipment to the dining hall building next door, tearing everything apart, cleaning and testing every component to separate what was damaged beyond repair and what could still work at least as contingency. The funny thing is that the backup servers where in an office next to the one That burned down so the service was off for good. They got lucky and the tape archive was properly sealed from the fire, it was inside the data center. But we still had to verify each of the tens of thousands of tapes for data corruption.
I guess the moral of the story is, off site backup means more than 20 meters away.
Add: oh I forgot to mention. The reason you will certainly die in a data center fire is not just the oxygen displacing gas. The doors are designed to create an airtight magnetical seal and when the fire suppression triggers it locks the doors permanently. It needs power to break the seal and open. There’s a big red emergency button next to the door, but in case of a fire it just buys you an extra 30 seconds to scram out of there before the system rearms the gas release. Miss the window and you’re trapped inside in an oxygen depleted environment. Firefighters had to tore the entire door frame off the concrete to access the data center.
The fuck happened?
An investigation concluded that a power line next to a vertical ventilation overheated, melted the wires and created a short circuit electrical arc. The power properly went off but a flammable outer lining in the ducts caught fire. The insulation correctly slowed down the fire, but the plastic on the wires caught fire. The wires went up into the data center and carried the flames inside through the ducts. All in all, the fire preventions worked to some extent but still several million dollars in servers got ruined and rendered inoperative. A whole rack burned until the halon triggered and mostly snuffed the fire, but the furniture in an office below, adjacent to the duct lines burned into a crisp further damaging the data center with smoke and heat.
It was the most surreal week on that job, taking all the equipment to the dining hall building next door, tearing everything apart, cleaning and testing every component to separate what was damaged beyond repair and what could still work at least as contingency. The funny thing is that the backup servers where in an office next to the one That burned down so the service was off for good. They got lucky and the tape archive was properly sealed from the fire, it was inside the data center. But we still had to verify each of the tens of thousands of tapes for data corruption.
I guess the moral of the story is, off site backup means more than 20 meters away.
Add: oh I forgot to mention. The reason you will certainly die in a data center fire is not just the oxygen displacing gas. The doors are designed to create an airtight magnetical seal and when the fire suppression triggers it locks the doors permanently. It needs power to break the seal and open. There’s a big red emergency button next to the door, but in case of a fire it just buys you an extra 30 seconds to scram out of there before the system rearms the gas release. Miss the window and you’re trapped inside in an oxygen depleted environment. Firefighters had to tore the entire door frame off the concrete to access the data center.