Capstone: a chilled-water leak in Hall A from first alarm to root cause

This capstone follows one incident at Stonebridge's YUL-2 campus in Beauharnois from the first alarm to the last corrective action. It is the leak that appeared in Lesson 8 as an example of a call to leadership, and it touches almost every part of the course: a preventive maintenance job, a structured hand-off, an alarm response procedure, an emergency operating procedure, incident roles and communications, a root cause analysis, corrective actions, an extent of condition review and the records that tie them together. As you read, notice which defences worked, which did not, and which were there only because of something someone did weeks earlier.

Background: the week before

On Thursday 8 April 2027, the day shift cleaned the chilled-water strainer of CRAH-A11, one of the computer room air handlers serving Hall A. It was the corrective work order Jamal had raised after the rising differential pressure on his night round in Lesson 1. Roz took the unit out of service at 07:12, isolated it, drained the strainer body, removed the cover, cleaned the basket and refitted the cover with a new gasket from stores. She returned the unit to service at 11:45 with the strainer differential pressure back down to 21 kPa (3.0 psi), and watched it for ten minutes without seeing a drip. The work order was closed the same afternoon.

That evening, as Lesson 11 described, Dmitri's structured hand-off caught that leak detection zone A-4, under rows 3 and 4 of Hall A, had been left inhibited after floor work. Jamal checked the area and the zone was restored. Nobody connected the two events at the time, and there was no reason to.

The night of the leak

In the early hours of Sunday 11 April, Hall A is fully loaded, the weather is mild and the night shift is Dmitri as shift lead with Jamal and Ike as technicians. Nothing in the hand-off at 19:00 was unusual.

Detect and contain

The first defence was detection. The leak cable under rows 3 and 4 sensed the water within minutes of it reaching the slab. If zone A-4 had still been inhibited, as it had been until the hand-off three nights earlier caught it, the first sign of the leak might have been a PDU fault or a tenant's equipment alarm, perhaps an hour later and with water under live electrical equipment. Dmitri says this is the part of the incident he thinks about most.

The second was the move from alarm response to emergency procedure. The leak detection ARP from Lesson 6 tells the shift lead to send a technician to confirm, and Jamal's report, fresh water, flowing, no visible source, met the ARP's condition for entering the chilled-water leak EOP. Dmitri did not wait to find the source before acting. The EOP's immediate actions are to keep people clear of water near electrical equipment, isolate the smallest section of piping that contains the leak and confirm that the remaining cooling can carry the load. Jamal and Ike went to valve station A-VS2, read the placard, the one corrected after the drill in Lesson 6, and closed CHW-A-16 and CHW-A-17. The hall's other loop carried the load, as it is designed to, with the three CRAH units on loop A-3 off.

The third was declaring the incident early. At 02:52, with the leak contained but not stopped and the source unknown, Dmitri declared severity 1 under the criteria of Lesson 7: water under the floor near live electrical equipment, a redundant cooling path lost and the hall running without its usual margin. He took command and gave Ike the operations role, and called in Tomás as communications lead, so that he himself would not be writing tenant notices while directing the response.

# Response times for the Hall A leak, measured from the first alarm
from datetime import datetime
fmt = "%H:%M"
alarm = "02:38"
events = {
    "acknowledged": "02:39",
    "confirmed on site": "02:44",
    "EOP entered": "02:47",
    "severity 1 declared": "02:52",
    "loop A-3 isolated": "02:58",
    "first tenant notice": "03:07",
    "water stopped": "03:15",
    "cooling restored": "06:30",
    "resolved": "06:52",
}
t0 = datetime.strptime(alarm, fmt)
for name, t in events.items():
    mins = (datetime.strptime(t, fmt) - t0).seconds // 60
    print(f"{name:21} {t}  +{mins:3d} min")
targets = {"acknowledged": 5, "severity 1 declared": 15, "first tenant notice": 30}
print()
for name, limit in targets.items():
    mins = (datetime.strptime(events[name], fmt) - t0).seconds // 60
    print(f"{name:21} {mins:3d} min against a target of {limit:2d}: {'met' if mins <= limit else 'missed'}")

acknowledged          02:39  +  1 min
confirmed on site     02:44  +  6 min
EOP entered           02:47  +  9 min
severity 1 declared   02:52  + 14 min
loop A-3 isolated     02:58  + 20 min
first tenant notice   03:07  + 29 min
water stopped         03:15  + 37 min
cooling restored      06:30  +232 min
resolved              06:52  +254 min

acknowledged            1 min against a target of  5: met
severity 1 declared    14 min against a target of 15: met
first tenant notice    29 min against a target of 30: met

All three targets were met: acknowledgement in one minute, declaration in fourteen and the first tenant notice in twenty-nine, inside the thirty-minute commitment in Stonebridge's customer contracts. The loop was isolated twenty minutes after the alarm. Restoring full cooling took almost four hours, most of it waiting for the contractor and refilling the loop, but the hall was never at risk of overheating because the other loop carried the load throughout.

Communicate

Tomás sent the first notice at 03:07 to every tenant in Hall A, using the template from Lesson 8: what happened, what it means for them, what is being done and when the next update will come. It said that a chilled-water leak had been found under the floor of Hall A, that the leak had been isolated and the hall's cooling was being carried by its second loop, that no tenant equipment was affected and that the next update would follow by 03:37. Updates followed every thirty minutes, and Anjali was kept informed by phone at each one. Two tenants called the operations centre; Tomás took both calls, so that Dmitri and the technicians were not interrupted. The resolution notice at 06:52 said that the leak had been repaired, the loop was back in service, the hall had full redundancy again and that an incident report would follow within five working days.

In the incident review, Corvid AI's facilities manager, Carmen Ortiz, said that the regular updates, sent on time even when there was little new to report, were the main reason her team did not escalate. Silence, she pointed out, is what makes customers nervous.

Recover and hand off

Recovery was careful and deliberate. The contractor replaced the gasket and refitted the cover with a calibrated torque wrench, tightening in a cross pattern to the torque in the manufacturer's manual. Ike refilled and vented loop A-3 and held it at operating pressure for twenty minutes with eyes on the cover before returning the CRAH units to service one at a time. Jamal dried the floor and checked the leak cable along its length, because a cable that has been wet can give false alarms or, worse, none. The return-to-service checklist from Lesson 4 guided each step, and the leak detection zone was confirmed live before the incident was closed.

At 07:00, the day shift took over a facility that had just come out of a severity 1 incident. Dmitri's hand-off followed the SAFE headings, with one addition: a briefing on the incident record, the open questions and the evidence kept for the investigation, which included the old gasket, the cover bolts in a bag with a tag, photographs of the strainer cover taken before the repair, and the torque readings found when the bolts were removed.

Root cause analysis

The analysis was led by Nadia Okoro, the site's reliability engineer, with Roz, Jamal, Dmitri and the contractor's supervisor. Nadia started, as Lesson 9 recommends, from the evidence rather than from opinions about what had happened. The photographs showed the gasket pinched at one side, where it had been squeezed out of its groove. The bolt torques measured when the cover was removed ranged from well above the manufacturer's value to less than half of it. The gasket was the right part number. Nobody had done anything they believed to be wrong.

The team used the 5 whys to follow the main thread.

• Why did water leak under the floor? Because the strainer cover of CRAH-A11 leaked.
• Why did the cover leak? Because the gasket was pinched and had begun to extrude on one side.
• Why was it pinched? Because the cover bolts were tightened unevenly, some far beyond the specified torque and some well below it.
• Why were they tightened unevenly? Because the PM procedure for strainer cleaning says to refit the cover, with no torque value, no pattern and no requirement for a torque wrench.
• Why did the procedure not include them? Because it was written from an older strainer model that used a different cover, and was not reviewed when the CRAH units in Hall A were replaced.

A fishbone, built in the same session, found contributing causes in other branches: the post-maintenance leak check was a ten-minute visual check, which can miss a slow leak that appears only after several heat and pressure cycles; the strainer sits above the raised floor's tiles at the end of the row, so a slow drip runs under the floor out of sight; and the work order had no field for torque readings, so nobody would have known what had been applied.

The analysis named two root causes, both about the system rather than the person: the procedure was missing the information needed to do the job right, and the change control for the equipment replacement did not include reviewing the procedures for the new equipment. Roz followed the procedure she was given. As Lesson 10 put it, the aim is to change the system so that the next person, following the procedure, gets it right.

Corrective actions and extent of condition

The extent of condition question came first: where else could the same thing happen? The same strainer model is fitted to every CRAH unit in Halls A and B, and the same procedure had been used to clean eleven of them in the past year. Nadia's team inspected all eleven within a week. Two had unevenly tightened covers with no leak yet, and their bolts were re-torqued under a planned job. The procedure library was then searched for other procedures written for equipment that had since been replaced, and four more were found.

# Corrective action tracker for the Hall A leak incident record
from datetime import date
today = date(2027, 5, 21)
actions = [
    ("Add torque value, cross pattern and calibrated wrench to strainer PM procedure", "Roz",   date(2027, 4, 16), date(2027, 4, 15)),
    ("Add torque readings field to strainer work orders",                         "Elif",  date(2027, 4, 30), date(2027, 4, 27)),
    ("Inspect and re-torque all CRAH strainer covers cleaned in the past year",      "Jamal", date(2027, 4, 18), date(2027, 4, 17)),
    ("Extend post-maintenance leak check to a 24 h follow-up inspection",            "Dmitri", date(2027, 4, 30), date(2027, 4, 29)),
    ("Add procedure review to change control for equipment replacement projects",   "Nadia", date(2027, 5, 14), None),
    ("Review four other procedures written for replaced equipment",                 "Roz",   date(2027, 6, 11), None),
    ("Add drip trays with leak sensors under CRAH strainers in Halls A and B",        "Nadia", date(2027, 8, 27), None),
]
closed = [a for a in actions if a[3] is not None]
print(f"{len(closed)} of {len(actions)} actions closed")
for what, owner, due, done in actions:
    if done:
        state = "closed" + (" late" if done > due else "")
    elif today > due:
        state = f"OVERDUE by {(today - due).days} days"
    else:
        state = f"open, due {due.isoformat()}"
    print(f"  {owner:7} {state:22} {what}")

4 of 7 actions closed
  Roz     closed                 Add torque value, cross pattern and calibrated wrench to strainer PM procedure
  Elif    closed                 Add torque readings field to strainer work orders
  Jamal   closed                 Inspect and re-torque all CRAH strainer covers cleaned in the past year
  Dmitri  closed                 Extend post-maintenance leak check to a 24 h follow-up inspection
  Nadia   OVERDUE by 7 days      Add procedure review to change control for equipment replacement projects
  Roz     open, due 2027-06-11   Review four other procedures written for replaced equipment
  Nadia   open, due 2027-08-27   Add drip trays with leak sensors under CRAH strainers in Halls A and B

Six weeks after the incident, four of the seven actions are closed, all on time. The change control action is a week overdue, because it needs agreement from the projects group, and Nadia has raised it at the monthly reliability review so it does not drift. The drip trays are an engineering change with a longer lead time, and will go through the change management process of DCT-330 like any other change. Each action has an owner and a date, and each will be checked for effect when it closes: the procedure change, for instance, will be checked by observing the next three strainer cleanings.

What the incident teaches

Looking back over the night, Dmitri lists what worked: a leak cable that was live because a hand-off caught an inhibit; a placard that was correct because a drill found it wrong; a hall that had a second loop to carry the load; early declaration and clear roles; tenant updates on time; and a careful return to service. Each of these was the result of ordinary work done well, often long before the leak. What did not work was a procedure that had not kept up with the plant, and a leak check that was too short to catch a slow leak.

That is the message of the whole course. Incidents are rarely caused by a single dramatic failure; they come from small gaps in maintenance, procedures and documentation that line up on the wrong night. They are also rarely contained by heroics; they are contained by alarms that work, procedures that are right, people who know their roles and records that let the next person pick up where the last one left off. A technician who does the routine work carefully, reports what does not look right and writes it down is doing incident response every shift, long before any alarm sounds.