Utility, IT and supply failures

A hospital is a machine that runs on electricity, water, medical gases, heating and cooling, data and a steady stream of supplies. Most of the time no one thinks about them. When one fails, the effects spread fast: a power failure stops pumps and monitors; a water main break stops hand washing, dialysis and the sterilizers; a cyberattack locks staff out of medication records; a shortage of IV bags after a factory is flooded forces changes in how drugs are given. These failures are among the most likely emergencies a hospital will face, which is why they ranked near the top of Northbridge's hazard analysis. This lesson covers the main utility, IT and supply failures, the backup systems that protect against them, and what staff do when the backups are not enough.

At Northbridge, a failure of an essential service is called as a Code Grey, with a plain description such as Code Grey, power failure, 4 West. Code Grey is also used for air exclusion, when outside air must be shut out because of smoke or a chemical release nearby.

Electrical power

Hospitals are built so that a loss of power from the utility does not stop critical care. Backup generators start automatically when utility power fails and take over selected circuits, known as the essential electrical system, within seconds. Battery-powered uninterruptible power supplies bridge the gap for equipment that cannot tolerate even a few seconds without power, such as some computer systems and imaging equipment.

In Canada, CSA Z32, Electrical safety and essential electrical systems in health care facilities, sets out how the essential system is divided and how it must perform, and CSA C282 covers emergency generators, including regular testing under load. The essential system typically supplies life safety loads, such as exit lights, corridor lighting and the fire alarm, critical patient care areas and outlets, and key equipment such as medical vacuum and air pumps.

Outlets on the essential system are usually red, or marked with a red dot or label. Equipment that must keep running in a power failure, such as ventilators, infusion pumps, monitors and dialysis machines, must be plugged into red outlets. Equipment that can wait, such as patient televisions and kettles, goes into normal outlets. Battery backup in portable equipment, such as infusion pumps and transport monitors, covers the transfer and short moves, but only if the batteries are charged.

When the power fails, staff on each unit:

• Check every patient who depends on equipment: ventilators, oxygen concentrators, infusion pumps, suction, monitors, beds with air mattresses, dialysis.
• Confirm that critical equipment is on red outlets and running, and move plugs from normal to red outlets if needed.
• Use manual methods if equipment stops: bag-valve-mask ventilation for a ventilated patient, manual suction, gravity infusions where safe.
• Keep flashlights at hand; corridor lights on the essential system stay on, but many room lights do not.
• Report to maintenance and the charge nurse, and follow updates from the switchboard.

Elevators may stop between floors when power is lost, so maintenance checks every elevator after a power failure. Patients should not be moved by elevator during a power failure unless the elevator is on emergency power and confirmed safe.

Water

A hospital uses a great deal of water: for drinking and cooking, hand hygiene, toilets, bathing, cleaning, the laundry, dialysis, the sterilizers that reprocess surgical instruments, the laboratory, heating and cooling equipment, and fire sprinklers. Losing water, from a main break, contamination or a boil-water advisory, shuts down many services at once.

Water planning starts with knowing how much each use takes and which uses are essential. Tom Bergstrom and Grant Ellison measured Northbridge's water use, by meter where possible and by estimate elsewhere, as part of the water outage annex that came out of the hazard analysis.

# Northbridge water use in L per day, and what can be cut in an outage
uses = {
    # use: (normal use, essential minimum)
    "drinking and food":      (9000, 6000),
    "hand hygiene":          (30000, 6000),   # switch to hand rub
    "toilets":               (60000, 15000),  # waste bags where possible
    "bathing":               (40000, 4000),   # wipes and basins
    "laundry":               (35000, 0),      # send out
    "sterile processing":    (25000, 20000),
    "dialysis":              (15000, 15000),
    "cooling towers":        (90000, 45000),
    "cleaning and other":    (30000, 8000),
}
normal = sum(n for n, e in uses.values())
essential = sum(e for n, e in uses.values())
tank = 150000   # emergency water storage on site, L

print(f"normal use:     {normal:7d} L per day")
print(f"essential only: {essential:7d} L per day")
print(f"stored water lasts about {tank / essential:.1f} days at essential use")

normal use:      334000 L per day
essential only:  119000 L per day
stored water lasts about 1.3 days at essential use

Cutting to essential uses brings daily water use down to well under half, but the stored water would still last little more than a day, and dialysis, sterile processing and cooling cannot be cut much further. The annex therefore includes agreements for water to be delivered by tanker trucks to a connection installed on the building, portable toilets and hand-washing stations, bottled water for drinking, and an early decision point for transferring dialysis patients and postponing surgery if the outage will last more than a day.

The figures are in litres because the hospital's meters and its tanker contract use litres, which avoids mistakes when ordering water in an emergency.

On the units, staff switch to alcohol-based hand rub for hands that are not visibly soiled, use bottled water for drinking and medication, use no-rinse bathing products, and follow instructions for toilets, such as using waste bags in commodes and limiting flushing. During a boil-water advisory, the hospital follows public health instructions on which water uses must stop, including drinking fountains, ice machines and some equipment.

Medical gases, heating, cooling and ventilation

Medical oxygen usually comes from a bulk liquid oxygen tank outside the building, piped to the wall outlets, with alarms that sound when pressure falls. If the supply fails, staff switch patients to portable cylinders, and respiratory therapy manages the supply and decides which patients get priority. Northbridge keeps a reserve of cylinders and a contract for urgent deliveries. Medical air and vacuum come from pumps in the building, on the essential electrical system.

Heating, ventilation and air conditioning matter more in hospitals than in other buildings. Operating rooms need controlled temperature, humidity and air changes. Isolation rooms for airborne infections need negative pressure to keep infection inside. Some drugs, vaccines and blood products must be kept within narrow temperature ranges. A failure of cooling in summer can make units dangerously hot for frail patients. Plans cover moving vulnerable patients, using portable cooling, monitoring drug refrigerators, and pausing services that cannot run without controlled air.

Gas leaks and elevator entrapment

A smell of natural gas, from a kitchen, a laboratory burner, a boiler room or a damaged line outside, is a utility emergency of its own. Staff who smell gas do not switch lights or equipment on or off, do not use phones or radios in the area, leave the area and keep others out, and call the emergency number from a safe place. Maintenance shuts off the supply, and the fire department and the gas utility check the area before anyone returns. If the leak is large or inside an occupied area, the incident commander may order an evacuation of the nearby rooms.

People trapped in a stopped elevator are usually safe, as long as they stay inside. Staff should talk to them through the doors, keep them calm and wait for maintenance or the elevator company. Forcing elevator doors open can cause falls into the shaft. If a patient in the elevator needs urgent care, the switchboard calls the fire department, which is trained to release people from elevators, and the response team waits at the nearest landing with equipment.

Communications

Phones, overhead paging, nurse call systems, pagers and wireless networks can fail separately or together. When they do, the hospital falls back on simpler methods: designated emergency phones on separate lines, two-way radios kept charged at key locations, and runners who carry messages between units and the command centre. Every unit at Northbridge has a list of these backups on the Code Grey page of its flip chart, and the switchboard tests the radios each month.

IT downtime and cyberattacks

Almost every part of care now runs on computers: the electronic health record, medication orders and administration, lab and imaging results, scheduling, pumps, monitors and even door locks. Systems go down for planned maintenance, for faults, and increasingly because of cyberattacks. In a ransomware attack, criminals encrypt the hospital's systems and demand payment; hospitals in Canada and the United States have lost their systems for days or weeks, diverted ambulances, postponed treatment and gone back to paper for everything.

Northbridge's downtime procedures assume that systems could be out for minutes or for weeks:

• Downtime kits on every unit with paper forms for orders, medication administration, nursing notes, lab requests and results, and patient lists.
• A read-only backup of key information, such as the medication administration record and recent results, printed or available on stand-alone computers, refreshed regularly during normal operations.
• Runners for lab results and imaging reports, and phone or radio for critical results.
• Manual processes for admissions, transfers and discharges, and for tracking patients.
• Clear rules for going back to the electronic system: who enters the paper records afterwards, and how to check that nothing was lost.

Staff practise downtime at least once a year. During Northbridge's first full downtime drill for the clinical systems, 4 West found that its downtime kit had forms from three years earlier and no blank medication records for the evening shift, and that most newer nurses had never written a paper medication record. The kits are now checked each month and the drill is repeated each year.

Staff also help prevent cyberattacks: most attacks begin with a phishing email or a stolen password. Reporting suspicious emails, never sharing passwords, locking screens and avoiding unapproved software are emergency preparedness too.

Supply chain failures

Hospitals rely on suppliers that may be far away, and a single disruption can affect hospitals across the continent. Drug shortages are common. In 2024, flooding from Hurricane Helene damaged a large IV fluid plant in North Carolina that supplied a big share of the IV solutions used in US hospitals, and hospitals had to conserve IV fluids for months. Early in the COVID-19 pandemic, PPE ran short everywhere at once.

Supply plans include knowing which items are critical, keeping a buffer stock of them, having more than one supplier where possible, tracking daily use during a shortage, using substitutes approved by pharmacy and the clinical leaders, and sharing with other hospitals in the region. During a shortage, staff are told clearly what has changed, such as switching from IV to oral medications where safe, and why.