The flood has gotten everywhere. Both the ground floor and the basement with the generators’ fuel were under water. It was a disaster. “We really didn’t expect the storm to have a significant effect on the data center,” Peer 1’s vice president Ryan Murphey recounted a year later. His company had not prepared for the hurricane Sandy that was about to hit New York. Only two employees were left in the data center at the time. He added: “At the worst, we thought the facility might lose utility power, and Jeff and Mike would need to switch to the building’s backup generators.”
Their servers, located in the building’s second floor, survived the storm intact.
However, the water flooded the underground, where fuel reserves were located. As well as pumps that were to carry that fuel up to the generators on the seventeenth floor. The storm disrupted not only the power coming from the grid, but also the data center’s backup system.
The generators were okay, but they were running out of fuel fast. Their fuel consumption was too high to be covered just by a small backup fuel tank on the generators’ floor.
The disaster created one of the most engaging data center stories of the recent past.
‘Bucket Brigade’ took on hundreds of stairs
The two engineers thought of contacting other employees. All those people then created a chain taking buckets of fuel up the stairs to the seventeenth floor. Their activity sparked so much interest that the data center’s customers with their employees and even passers-by started joining in. “People were tired of being helpless against the storm, they were thrilled they could do something with a purpose,” said one of the volunteers. After the storm cleared out, their actions piqued the interest of media and ‘the bucket brigade’ as they were soon dubbed, even got a documentary made about them.
Thanks to their cooperation, the generators survived the storm and its aftermath without a failure. After a few days, the tired but proud and happy team of volunteers was replaced with backup pumps and fuel hoses. The data center has survived the storm without any of its customers losing access to their services.
The hurricane did take its toll on the facility, though. “Looking back, it was a very rough recovery and one that I hope to never experience again,” Murphey admitted.
Their company and its clients were incredibly lucky. The data center had no complex system of coping with such a huge failure. At least four other data centers in the same part of New York were not so lucky and failed.
Fires, floods and rockets. The things data centers need to cope with
No one knows about natural disasters beforehand.
They just strike, out of the blue. And it’s not just floods. Even fires can do serious damage to data centers. Recently, there were news of a fire in Belfast Telephone data center in the UK, another fire in Milan, Italy and one more in Apple’s data centers in Arizona.
There have even been explosions in data centers, like the one in Los Angeles. It’s usually the generators or fuel storage that are the culprits in these cases. But some data centers have to cope with even more serious threats. Like the ones in Israel that need to be ready for a rocket attack.
A recent study by Zenium Technology Partners has revealed, that half of the data centers would not be able to withstand a natural disaster without failures. Their customers would be cut off from their services for hours or even days.