First of all, please accept our apologies for Saturday's power outage in MasterDC Prague data center.
During Saturday afternoon we have been dealing with a power problem in the tower server room. When switching the technology for backup power line, there was a human error which caused the outage of primary power branch. Customers with dual power service as well as the cloud hosting customers did not experience any outage. The entire power backup system including two separate power branches, UPS units and diesel generators are designed to standard power outage. On Saturday, unfortunately due to human intervention an abnormal outage occured and that is why even diesel generators could not be started. As we informed our customers, power has been successfully restored in about 30 minutes. Promptly after that, our technicians started to help customer with restoring their servers. Monitoring system evaluated an abnormal behaviour of our network elements, namely switches, with few malfunctioning ports. Technicians in Prague and Brno administrators took care of the transfer to the working ports and the whole night and during Sunday actively dealt with customer requirements and assisted in restarting their technology.
We have information about the units where even customers with the Dual Power from two branches experienced power outages. As one of the possible causes, we evaluated a wrong wiring of customer technologies in rack. If you are interested in help with this matter, please contact us at firstname.lastname@example.org.
Based on the outage, we have decided to improve the process of crisis intervention through the following measures:
We apologize again for any inconvenience this outage have caused. If you have any further questions, please contact the Prague Sales Director Petr Štěpánek on email@example.com. If you want to say something directly to me, I am - even on vacation - available on firstname.lastname@example.org.
Thank you for the support that we have from many of you received, and thank you for staying with us.
On behalf of Master Internet
Filip Špaček, Chief Operating Officer