Leaks, spills and emergency response

Sooner or later, every liquid-cooled hall has a leak and every protected room has an alarm. What happens next decides whether the event becomes an injury, an outage or a non-event. This lesson covers how leaks are detected, how to respond to a coolant leak near energized equipment, how to clean up and report spills, and how fire protection and cooling emergencies fit into the site's emergency response, including working with the fire service and re-entering a room after a gaseous discharge.

HSE-320 Lesson 11 introduced emergencies in data centers, including emergency power off and evacuation, and HSE-322 Lesson 11 covered battery incidents. HSE-251 covers spill response in general and HSE-222 the role of fire wardens. This lesson focuses on what is specific to suppression systems and liquid cooling.

How leaks are detected

Liquid cooling systems use several layers of leak detection, because a small leak found early is a nuisance and a large leak found late can take down a row.

• Leak detection cable, sometimes called rope sensor, runs along the floor under racks, manifolds, CDUs and piping, and raises an alarm when liquid bridges its conductors. Some systems report the distance along the cable to the leak.
• Spot sensors sit in drip trays, sumps and the bottoms of racks and CDUs.
• CDUs monitor flow, pressure and reservoir level. A falling level or a pressure drop can indicate a leak before any liquid reaches a sensor.
• Some servers and rack manifolds have their own leak sensors that report through the server management system and can shut down the server or close valves.
• People, through regular rounds, cameras and the simple habit of looking under racks, find drips that sensors miss.

Alarms usually go to the building management system or data center infrastructure management system and to the operations desk. Some designs isolate automatically: a leak sensor or a pressure drop closes valves for the affected rack or row, or stops the CDU pumps. Automatic isolation limits the volume released, but it also stops cooling to the equipment downstream, which will then heat up within seconds to minutes. Servers protect themselves by throttling and shutting down, but an unplanned shutdown of an AI training cluster is costly, so the isolation logic is a business decision as well as a safety one, and people on site need to know what their system will do.

How much can leak?

The volume released by a leak depends on the size of the opening, the pressure behind it, and how long it continues before it is stopped. Pressure at the leak is the loop pressure less the friction losses in the piping between the pump and the fault, which can be estimated with the Darcy–Weisbach equation. Flow through the opening can be estimated with the orifice equation used in Lesson 6 for vents. The program below combines these for a failed quick disconnect on a rack manifold at Cedar Ridge, with invented values, to show how detection and response time govern the volume released.

import math
# Teaching estimate of coolant released by a failed quick disconnect.
rho, mu = 1020.0, 0.0025          # PG25 density kg/m3 and viscosity Pa.s (approx.)
# Darcy-Weisbach friction loss in 25 m of 50 mm supply pipe at 190 L/min
L, D, flow_lpm = 25.0, 0.050, 190.0
v = flow_lpm / 60000 / (math.pi * D**2 / 4)
Re = rho * v * D / mu
f = 0.316 / Re**0.25               # Blasius approximation for smooth pipe
dp_friction = f * (L / D) * rho * v**2 / 2
p_pump = 3.5e5                     # Pa gauge at CDU outlet (illustrative)
p_leak = p_pump - dp_friction
print(f"Pipe velocity {v:.2f} m/s, Re {Re:,.0f}, friction loss {dp_friction/1e5:.2f} bar")
print(f"Pressure at the manifold about {p_leak/1e5:.2f} bar")
# Orifice flow through a partly failed coupling seal
Cd = 0.6
for gap_mm2 in (2, 10, 40):
    A = gap_mm2 * 1e-6
    q = Cd * A * math.sqrt(2 * p_leak / rho) * 60000     # L/min
    for minutes in (1, 5, 20):
        print(f"opening {gap_mm2:2} mm2: {q:5.1f} L/min; after {minutes:2} min "
              f"{q*minutes:6.1f} L ({q*minutes/3.785:5.1f} US gal)")

Pipe velocity 1.61 m/s, Re 32,901, friction loss 0.16 bar
Pressure at the manifold about 3.34 bar
opening  2 mm2:   1.8 L/min; after  1 min    1.8 L (  0.5 US gal)
opening  2 mm2:   1.8 L/min; after  5 min    9.2 L (  2.4 US gal)
opening  2 mm2:   1.8 L/min; after 20 min   36.9 L (  9.7 US gal)
opening 10 mm2:   9.2 L/min; after  1 min    9.2 L (  2.4 US gal)
opening 10 mm2:   9.2 L/min; after  5 min   46.1 L ( 12.2 US gal)
opening 10 mm2:   9.2 L/min; after 20 min  184.4 L ( 48.7 US gal)
opening 40 mm2:  36.9 L/min; after  1 min   36.9 L (  9.7 US gal)
opening 40 mm2:  36.9 L/min; after  5 min  184.4 L ( 48.7 US gal)
opening 40 mm2:  36.9 L/min; after 20 min  737.5 L (194.8 US gal)

A small seal failure leaks a liter or two per minute; a coupling that has partly come apart can release tens of liters per minute. If detection and isolation take 20 minutes, the larger leak releases several hundred liters, enough to flood the floor of a row, run under cabinets and reach power cables, busway taps and floor boxes. If detection triggers automatic isolation within a minute, the release is a few tens of liters. The numbers are illustrative, but they show why fast detection, automatic isolation and a practiced response matter more than cleaning up afterwards.

Liquid and electricity

The most serious hazard of a coolant leak is not the coolant itself but its contact with energized equipment. Treated water and glycol mixtures conduct electricity, and even deionized water picks up ions quickly once it leaves the loop and meets dirt and metal. Liquid reaching power distribution units, busway plug-in units, power cords, floor boxes or the bottom of racks can cause short circuits, arcing and ground faults, and a person standing in a pool of liquid touching an energized enclosure with a fault is at risk of shock.

Ground-fault protection for personnel, such as ground fault circuit interrupters, is required by the electrical codes on some receptacles in wet or damp locations, but most IT power circuits in data halls are not ground-fault protected for personnel, because nuisance tripping would interrupt critical loads. Do not assume that a circuit will trip before a person is hurt. The electrical safety rules of HSE-321 apply: liquid near energized equipment is an electrical hazard to be assessed by qualified people, and decisions about de-energizing circuits are made by the electrical team in consultation with operations and the customer.

Dielectric immersion fluids do not conduct electricity, which is the point of using them, but a spill of oil near equipment is still a slip hazard, and if the fluid is contaminated with water it may conduct. Fluorinated fluids are also non-conductive.

Responding to a leak

A good leak response is practiced and calm. The steps below are typical; each site's procedure, shaped by its equipment and contracts, governs.

• Confirm the alarm. Check the monitoring system for the sensor location, CDU pressures and levels, and any automatic actions taken. Look at cameras if available.
• Approach with care. Go with a second person if possible, carrying absorbent materials. Look before walking into a row: is liquid pooling, spraying or dripping, and is it near power equipment?
• Keep out of liquid near energized equipment. Do not step into pooled liquid that is in contact with power distribution, cords or enclosures. Set up barriers and slip signs.
• Stop the source if trained and authorized. Close the rack or row isolation valves, or stop the CDU pumps, following the procedure, and confirm on the monitoring system that pressure and flow have fallen.
• Contain the spread. Place absorbent socks or booms around the spill, use pads and trays to catch drips, and protect floor boxes and cable openings. Use absorbents compatible with the fluid.
• Notify. Tell the operations lead, the customer (whose equipment may need to be shut down or will throttle) and, if liquid has reached electrical equipment, the electrical team.
• Clean up and dispose. Clean the area thoroughly, because glycol residues remain slippery and oil films spread. Collect used absorbents and recovered fluid in labeled containers for proper disposal; do not pour coolant into floor drains or sinks unless the site's environmental procedure allows it.
• Find the cause and restore. Do not simply reconnect and refill. Find the failed component, repair or replace it, pressure test, refill with the correct coolant, purge air, and check leak detection before returning the equipment to service.

Spill reporting

A coolant spill that stays on the data hall floor and is cleaned up is usually an internal matter, recorded in the site's incident system. A spill that reaches a storm drain, the ground or a watercourse, or a large release of glycol, oil or treatment chemicals outdoors, may have to be reported to the state environmental agency and, for some substances above reportable quantities, to the National Response Center. Ethylene glycol, for example, is listed as a hazardous substance with a federal reportable quantity, while propylene glycol is not. Oil spills that reach water fall under the Clean Water Act. Refrigerant releases are governed by EPA's Section 608 rules, and some states have their own rules for halocarbon agents. The site's environmental procedure should list the thresholds and contacts.

Fire protection and cooling in the emergency plan

Fire and cooling emergencies are part of the site emergency response plan described in HSE-320 Lesson 11 and HSE-105. The parts specific to this course are these.

• Alarm response roles. Who responds to a clean agent pre-alarm, a sprinkler water flow alarm, a leak alarm or an off-gas alarm, and what each person does. Staff who are not trained responders evacuate on fire alarms.
• Fire service information. The pre-incident plan given to the fire service should show the locations of clean agent and any carbon dioxide protected rooms with their agents, battery rooms, immersion tanks with fluid type and quantity, liquid cooling isolation valves, CDU locations, sprinkler control valves, emergency power off stations and electrical rooms, together with safety data sheets and keys. HSE-322 Lesson 11 recommended inviting the fire service to tour the site; this course adds the liquid cooling systems to that tour.
• A site liaison. A knowledgeable person meets the fire service on arrival with drawings, the alarm panel display and the latest information about who is in the building and which systems have operated.
• Isolation decisions. The plan should say who can decide to shut down liquid cooling loops, close sprinkler valves after a fire is out, or de-energize equipment, and that the fire service has the final say during an incident.

Re-entry after a gaseous discharge

After a clean agent or carbon dioxide discharge, the protected room contains the agent, possibly decomposition products and the products of any fire. Re-entry is controlled.

• Keep the room closed for the hold time unless the fire service decides otherwise, so that the agent can finish extinguishing and cooling any fire.
• Do not let anyone enter until the fire service, or trained staff with self-contained breathing apparatus if the fire service is not involved, has checked the room.
• Ventilate the room to the outside using the designated post-discharge exhaust, if there is one, or by opening doors to the outside, not into occupied areas. Remember that carbon dioxide and halocarbon mixtures collect in low areas, including under raised floors.
• Test the atmosphere before general re-entry: oxygen at least 19.5 percent, and, where a fire occurred, carbon monoxide and acid gases such as hydrogen fluoride below the site's limits, using suitable instruments.
• Treat the system as impaired until it is recharged and restored: provide alternative protection such as a fire watch and extinguishers, follow the impairment process, and arrange cylinder replacement.
• Record the discharge, investigate its cause, and make any environmental reports required for the agent released.