Emergencies in data centres

Data centres are designed to keep running through failures, and their operators plan in great detail for equipment emergencies: utility outages, UPS faults, chiller trips, generator failures. People emergencies need the same care. A fire, a clean agent discharge, an electrical injury, a cardiac arrest in a remote corner of a data hall, a water leak near energized equipment or a lithium-ion battery going into thermal runaway each call for fast, correct action by people who may have only a few seconds to decide. The security, layout and noise of a data centre can make those decisions harder: locked doors and mantraps slow paramedics, loud halls mask alarms, and large windowless buildings are easy to get lost in.

This lesson covers the emergency plan, the alarms you will hear and what they mean, how to respond to the most likely emergencies, and how drills keep the plan alive. Fire wardens and evacuation leaders are covered in HSE-222, first aid and CPR in HSE-110 and HSE-111, the suppression systems themselves in HSE-323, battery emergencies in HSE-322 and electrical incident response in HSE-224 and HSE-321.

Life safety comes first

The first principle of emergency response in a data centre is the same as anywhere else: people come first. A facility that has spent millions on redundancy can tolerate the loss of a hall; it cannot undo the loss of a life. In an emergency, nobody should stay behind to save equipment, finish a step in a MOP, shut down servers gracefully or collect their laptop. Operations staff with specific emergency roles may carry out actions in the emergency operating procedures, such as isolating a fuel supply or confirming that a suppression system has operated, but only where the procedure says so and it is safe.

The emergency plan

Every data centre should have a written emergency response plan, required in one form or another by OHS law in both countries and by fire codes. In the United States, OSHA's emergency action plan requirements in 29 CFR 1910.38 set out minimum contents; Canadian provinces have similar requirements in their OHS regulations and fire codes. A good plan for a data centre covers the following:

• The types of emergency that could occur at the site and the response to each.
• Alarm signals and what each means, including fire alarms, suppression pre-discharge alarms, refrigerant and gas alarms and any site-specific signals.
• Roles: who takes charge, who calls emergency services, who meets them, who leads evacuation and who accounts for people.
• Evacuation routes, refuge areas and assembly points, with plans for people with disabilities.
• How headcount is done, including visitors, contractors and customer technicians.
• How emergency services get through security quickly, including keys, codes, escorts and site plans for firefighters.
• Locations of first aid kits, automated external defibrillators (AEDs), eyewash stations, spill kits and EPO buttons.
• Communication with customers, management and authorities after the immediate emergency.

Security is a special consideration. Mantraps, biometric readers and locked doors that keep intruders out also keep paramedics and firefighters out unless the plan provides for them. Security staff must know that their job in a medical emergency or fire is to get responders to the scene as fast as possible, with escorts and override access ready, and to keep the headcount. Many sites agree pre-incident plans with the local fire department, including building layout, hazards such as battery rooms and fuel storage, suppression systems and the location of EPOs and shutoffs.

Alarms and what they mean

Data centres can have more kinds of alarm than most buildings, and in a loud hall they can be hard to hear. Learn the alarms at your site during orientation, and look for strobes as well as listening for horns.

Clean agent systems deserve particular care. When detection confirms a fire condition in a room protected by a clean agent or inert gas system, a pre-discharge alarm sounds and a short countdown begins, often a few tens of seconds, before the agent is released. The countdown exists so that people can get out. Some systems have a hold switch, sometimes called an abort switch, that pauses the countdown while it is held; it is for use only by trained people who are certain there is no fire, for example during an accidental activation by a contractor's dust, and must never be used as a way to keep working. Halocarbon agents are designed for occupied spaces at their design concentrations, but their decomposition products from a fire can be harmful, and inert gas systems reduce oxygen. The discharge itself is loud, can be forceful near nozzles and can drop visibility. Nobody should be in a protected room during a discharge, and nobody should re-enter until the room has been ventilated and authorized persons say it is safe. HSE-323 explains the agents and their hazards.

import random

# Can everyone leave a protected room during the pre-discharge countdown? (illustrative)
random.seed(11)
countdown_s = 30
walking_speed = 1.0          # m/s, slowed by aisles, doors and reaction time
reaction_s = 5               # time to recognize the alarm and start moving
workers = {f"worker {i}": random.uniform(5, 40) for i in range(1, 7)}   # path length to exit, m
for name, path_m in sorted(workers.items(), key=lambda kv: kv[1]):
    t = reaction_s + path_m / walking_speed
    margin = countdown_s - t
    flag = "OK" if margin >= 5 else "AT RISK"
    print(f"{name}: {path_m:4.1f} m ({path_m * 3.281:5.1f} ft) to exit, out in {t:4.1f} s,"
          f" margin {margin:5.1f} s  {flag}")

worker 1: 20.8 m ( 68.4 ft) to exit, out in 25.8 s, margin   4.2 s  AT RISK
worker 4: 21.3 m ( 69.9 ft) to exit, out in 26.3 s, margin   3.7 s  AT RISK
worker 5: 22.8 m ( 74.7 ft) to exit, out in 27.8 s, margin   2.2 s  AT RISK
worker 2: 24.6 m ( 80.7 ft) to exit, out in 29.6 s, margin   0.4 s  AT RISK
worker 6: 25.6 m ( 83.9 ft) to exit, out in 30.6 s, margin  -0.6 s  AT RISK
worker 3: 37.3 m (122.5 ft) to exit, out in 42.3 s, margin -12.3 s  AT RISK

The numbers are illustrative, but they show why people working deep in a large protected room must react to the pre-discharge alarm at once and why they should know their nearest exit before work starts. Hesitating for even ten seconds to see whether the alarm is real can turn a comfortable margin into none. Planning work in protected rooms includes checking the exit routes, keeping them clear and, for long tasks in very large rooms, discussing with operations whether the system's operating mode should be changed under a formal impairment.

Responding to common emergencies

Fire or smoke. If you see smoke or fire, raise the alarm using the nearest manual pull station and by telling operations, and evacuate. Use a portable extinguisher only if you are trained, the fire is small and in its early stage, you have a clear exit behind you and the site procedure allows it; fires in electrical equipment and batteries can reignite and produce toxic smoke. Do not open cabinets or rooms where you suspect fire. Close doors behind you to contain smoke and protect any suppression system's agent concentration.

Electrical injury. If someone is in contact with an energized conductor, do not touch them. Have the power removed by the fastest safe means, such as opening the supply breaker or, if the plan says so, operating the EPO, then call emergency services and give first aid. Start CPR and use an AED if the person is not breathing normally. Cool arc flash burns with cool running water and treat for shock. Everyone who has received an electric shock should be medically assessed, even if they feel fine, because effects on the heart can be delayed. Secure the scene for investigation; HSE-224 and HSE-321 cover electrical incident response in detail.

Medical emergencies. Data centres can be large, isolated and sparsely staffed, so a cardiac arrest in a data hall at 3 a.m. may not be noticed quickly. Call emergency services and site security, start CPR, send someone for the nearest AED and have security meet the paramedics at the entrance with access and an escort. Lone work check-in systems, described in Lesson 3, exist partly for this reason.

Water leaks. Water near energized equipment is both an electrical hazard and a slip hazard. Do not step into pooled water near electrical equipment or under raised floors with power cabling. Report the leak to operations, who will follow the emergency operating procedure to isolate the source and, if necessary, the affected electrical equipment. Leak detection cables under floors and around cooling equipment help find leaks early; HSE-323 covers leaks from liquid cooling systems.

Battery and refrigerant emergencies. Unusual smells, hissing, smoke or swelling from battery cabinets, or a battery room gas alarm, may signal thermal runaway or off-gassing. Leave the area, raise the alarm and do not try to deal with it yourself unless you are trained and the procedure says so; HSE-322 covers battery emergencies. Refrigerant alarms in machinery rooms mean leave and stay out, as Lesson 6 explained.

Power outages. A utility outage should be invisible to the IT load, but people in the building may notice lights going out briefly, generators starting and alarms. Emergency lighting should light exit routes. Stop work, stay where you are if it is safe, avoid moving around open floor tiles in poor light and follow instructions from operations. Electrical work in progress should be stopped and made safe, because system configurations may change during the outage.

Drills and exercises

Plans that are never practised fail when they are needed. Data centres should run evacuation drills, including at night and with contractors on site, at the frequency required by the fire code and the site's plan. Operations teams also run tabletop exercises and walk-throughs of emergency operating procedures, for example a utility outage combined with a generator failure, and some practise live responses during planned tests. Drills should test the things most likely to go wrong in a data centre: whether alarms can be heard and seen in every part of the hall, whether security can get responders through the mantraps quickly, whether the headcount includes visitors and contractors, and whether people working in protected rooms react to pre-discharge alarms at once. After each drill and each real event, record lessons and fix what did not work.