It’s not very often that you hear of equipment explosions in our colocation Vietnam industry, but if it occurs, the consequence is so bad.
Few months ago, in Los Angeles, a generator located in the basement of a shared office building exploded, injuring 4 and causing a ripple effect that impacted multiple DCs within the immediate area. This was largely in part due to a power station located nearby that was damaged by the explosion.
We want to look at 2 of the DCs that were impacted the most from this event and what lessons should be learned from it.
Data center #1: Cooling system power loss
One of the impacted DCs was located close to where the explosion happened, and while they did not lose full power to their facility, they lost power to their entire cooling system. This resulted in highly elevated data center temperatures throughout their facility. While there were no reported damages or customer outages as a result of this, it did place their staff and customers on high alert.
Data center #2: Connectivity outage
A connectivity provider operates a facility near to where the explosion taken place, and as a result of the power station being damaged by the blast, their facility lost utility power. While it occurs that they had a generator for backup power, they cannot switch successfully over to generator power. As a result, their connectivity services were out for a noticeable amount of time.
A lesson to take from this
I think that the lesson wen can learn from this incident is the importance of testing for any DC and testing your requirements as well as your redundancy procedures.
Redundancy: It is important to test your redundancy measures themselves on a regular basis, to make sure that your equipment failover processes will work when they want to, should a piece of equipment fail. In the case of the connectivity provider’s facility, it cannot switch over to generator power when they lost utility power. This situation is bad, and is something that can definitely be mitigated through ongoing testing and optimization.
Equipment: You never want to be in a position where a generator explodes on you. While the specific reason for the explosion has not been determined, the risk of failure for a generator or any piece of equipment can be decreased when that equipment is properly maintained and tested on a regular basis. When you have many types of equipment that your DC relies on for redundancy, this kind of testing is crucial
There are many lessons to be learned from any incident, and this one revealed all too clearly that having ongoing equipment and redundancy testing procedures in place is crucial for DCs.