Working near UPS systems and batteries

Batteries are the part of a data center's power system that cannot be switched off. They are why the servers keep running for the seconds between a utility failure and the generators taking over, and they are also one of the most serious hazards in the building. A UPS battery string holds hundreds of volts of direct current and can deliver enormous fault current into a short circuit. Some batteries produce flammable gas; others can overheat and burn in ways that are very hard to stop. This lesson covers what a technician needs to know to work near UPS systems and battery rooms, to recognize trouble and to stay out of situations that belong to specialists.

UPS systems and their stored energy

A double-conversion UPS converts incoming AC to DC, keeps its batteries charged from that DC, and converts DC back to clean AC for the load. Inside each UPS module are rectifiers, inverters, a DC bus and large capacitors that smooth it. The DC bus of a large UPS typically operates at several hundred volts. Even when a module's breakers are open, its capacitors can hold a lethal charge for minutes, which is why every UPS manual gives a discharge time to wait, and why the lockout procedure includes measuring the DC bus before work.

At Stonebridge, UPS modules sit in UPS rooms beside each hall's electrical room, with their battery cabinets or battery rooms beside them. Hall A has lead-acid batteries in a dedicated battery room; Halls B, C and D use lithium-ion battery cabinets in the UPS rooms. The edge site uses lithium-ion batteries inside its prefabricated modules. Technicians enter UPS rooms and battery rooms only with the access level their training record allows and, at first, only with an escort from the critical facilities team.

Battery types in the data center

Lead-acid batteries are the long-established choice. They are heavy, need more floor space and need replacing every four to six years or so, but their hazards are well understood. Lithium-ion batteries are lighter and smaller, last longer and tolerate higher temperatures, which is why most new UPS systems use them. Their main hazard is different: thermal runaway.

Shock and short-circuit hazards

A UPS battery string is made of many cells or modules connected in series to reach a DC voltage of several hundred volts. Every intercell connection, terminal and busbar in the cabinet is energized at all times. A metal tool, a watch strap, a ring or a dropped nut that bridges two terminals can cause a short circuit that welds metal, throws molten droplets and starts an arc. DC arcs are harder to extinguish than AC arcs because the current never passes through zero. For these reasons:

• Only trained battery technicians and qualified electrical workers open battery cabinets or work on battery terminals.
• Jewelry, watches and metal-framed glasses are removed before anyone works near open batteries, and tools are insulated.
• Nothing is placed on top of batteries or battery cabinets.
• The DC disconnects and battery breakers are part of the lockout procedure, but locking them out does not de-energize the batteries themselves; the cells stay charged.

Hydrogen from lead-acid batteries

Lead-acid batteries produce hydrogen and oxygen when they are charged, especially when they are overcharged at the end of charge or when a cell is failing. Vented batteries release gas continuously; VRLA batteries recombine most of it inside but vent some through their valves when pressure builds. Hydrogen is lighter than air, odorless and colorless, and burns or explodes in air at concentrations between about 4 % and 75 % by volume. Battery rooms are therefore ventilated to keep hydrogen far below the lower flammable limit, commonly to below 1 % (a quarter of the limit) under the battery standards and fire codes, and many rooms have hydrogen detectors that alarm and increase ventilation. Sources of ignition, such as open flames, grinding and smoking, are prohibited in battery rooms, and hot work there requires a permit.

The code below shows the kind of estimate engineers use to check ventilation. It uses the textbook figure that each ampere-hour of overcharge current produces about 0.42 liters of hydrogen per cell, and a conservative overcharge current. Real calculations follow the battery maker's data and the applicable code.

# Hydrogen from a lead-acid battery room and the ventilation needed (illustrative)
cells = 480                  # cells in the room (all strings)
i_overcharge = 2.0                 # amperes of overcharge current per string, worst case
h2_l_per_ah_cell = 0.42          # liters of hydrogen per ampere-hour per cell (textbook value)
room_volume = 6350               # cubic feet
h2_lph = cells * i_overcharge * h2_l_per_ah_cell          # liters per hour
target = 0.01                    # keep hydrogen below 1 % by volume
air_needed_m3h = h2_lph / 1000 / target
cfm = air_needed_m3h * 35.315 / 60
print(f"Hydrogen produced: {h2_lph * 0.035315:.1f} ft3/h")
print(f"Fresh air needed to stay below 1 %: {cfm:.0f} cfm")
print(f"That is {cfm * 60 / room_volume:.2f} room air changes per hour")

Hydrogen produced: 14.2 ft3/h
Fresh air needed to stay below 1 %: 24 cfm
That is 0.22 room air changes per hour

The answer is a modest airflow, which is why a working exhaust fan is usually enough. The danger comes when ventilation fails, a charger fault overcharges a string or a failing cell produces gas at a much higher rate. A hydrogen alarm in the Hall A battery room is treated as an emergency: nobody enters to investigate except the critical facilities team, and they do it following the battery room procedure.

Acid and other chemical hazards

Lead-acid electrolyte is dilute sulfuric acid. In VRLA batteries it is held in glass mats or gel and does not normally leak, but a cracked case or a battery in thermal runaway can release it. It burns skin and eyes. Battery rooms have an eyewash station or emergency shower within easy reach, spill kits with neutralizing agent, and acid-resistant gloves and face shields for the battery technicians. Lithium-ion batteries contain flammable organic electrolytes and, in a failure, can release toxic and flammable gases including hydrogen fluoride and carbon monoxide. Their safety data sheets describe the hazards, and technicians are trained to recognize the signs of a failing battery and leave.

Thermal runaway

Thermal runaway is a self-feeding temperature rise inside a battery cell. Damage, a manufacturing defect, overcharging or high temperature starts a reaction inside the cell that produces heat; the heat speeds up the reaction, which produces more heat. In a lithium-ion cell the temperature can climb to several hundred degrees in seconds, the cell vents flammable gas, and the gas may ignite. The heat can then drive neighboring cells into thermal runaway, so a single failing cell can spread through a module and a cabinet. Lead-acid batteries can also suffer thermal runaway, more slowly, when a hot battery is charged at too high a voltage, causing swelling, venting and melting.

Lithium-ion UPS batteries are managed by a battery management system that monitors every cell's voltage and temperature, balances the cells and disconnects the battery if anything goes outside its limits. Fire codes and standards such as NFPA 855 for stationary energy storage systems and the fire code requirements adopted locally set rules for how much lithium-ion energy may be in one fire area, how it must be separated, detected and protected, and how it is tested by the manufacturer under UL 9540A. Stonebridge's lithium-ion cabinets have their own off-gas detection, which can detect the gas a cell vents before it ignites. An off-gas alarm is one of the earliest warnings of thermal runaway.

# Read battery management data: which modules need attention? (illustrative limits)
modules = [  # (module, max cell temp C, cell voltage spread mV, off-gas alarm)
    ("B2-R1-M03", 27.5, 18, False), ("B2-R1-M07", 29.0, 22, False),
    ("B2-R2-M01", 41.5, 35, False), ("B2-R2-M05", 28.0, 140, False),
    ("B2-R3-M04", 30.5, 25, True),
]
for mod, t, spread, gas in modules:
    t_show = f"{t * 9 / 5 + 32:.0f} F"
    if gas:
        action = "OFF-GAS ALARM: leave the room, follow the emergency procedure"
    elif t > 40:
        action = "high temperature: alert critical facilities now"
    elif spread > 100:
        action = "cell imbalance: open a ticket for the battery vendor"
    else:
        action = "normal"
    print(f"{mod}  {t_show:>7s}  {spread:4d} mV  {action}")

B2-R1-M03     82 F    18 mV  normal
B2-R1-M07     84 F    22 mV  normal
B2-R2-M01    107 F    35 mV  high temperature: alert critical facilities now
B2-R2-M05     82 F   140 mV  cell imbalance: open a ticket for the battery vendor
B2-R3-M04     87 F    25 mV  OFF-GAS ALARM: leave the room, follow the emergency procedure

Signs of battery trouble

Technicians walk past UPS rooms and battery cabinets every shift. The signs they are trained to notice and report immediately are:

• Any alarm on a UPS, battery management system, hydrogen detector or off-gas detector.
• A smell: rotten eggs (hydrogen sulfide from a failing lead-acid battery), a sweet or solvent smell (electrolyte), or burning plastic.
• Swollen, cracked or leaking battery cases, white or green deposits on terminals.
• Hissing or popping sounds, visible vapor or smoke from a cabinet.
• Unusual heat felt from a cabinet door, or a battery room ventilation fan that is not running.

The response to a sign of thermal runaway or fire in a battery room is to leave, close the door, alert the shift lead and the fire alarm system, and keep everyone away. Lithium-ion battery fires produce toxic gases and can re-ignite hours after they appear to be out; they are left to the fire service and the site's emergency procedure. A technician never opens a battery cabinet that is alarming to look inside.

Rack-level batteries and small UPS units

Batteries are not only in the UPS rooms. Some customer racks contain small rack-mounted UPS units, and some new AI and storage systems use battery backup units inside the rack itself. Many network switches and servers have small lithium batteries for their clocks, and storage arrays may have cache-protection batteries. These are much smaller, but the same signs of trouble apply: swelling, heat, smell. Swollen rack-mounted UPS batteries are a common finding during walks. They are reported, the customer is told, and the unit is replaced by the customer or a qualified technician. Fire codes increasingly count the total lithium-ion energy in a room, including rack-level batteries, so customers must declare them before they are installed.

Moving and storing batteries

Replacement batteries arrive on pallets and are very heavy: a single large VRLA block can weigh more than 65 lb. They are moved with pallet jacks and lifting aids, never carried in bulk by hand. Lithium-ion modules are lighter but are dangerous goods for transport and storage; damaged modules are isolated in a designated area and handled following the manufacturer's procedure. Used batteries are collected for recycling under the provincial or state rules for hazardous waste. A technician receiving a battery delivery checks the packaging for damage and reports any dented, swollen or leaking unit before it enters the building.