Capstone: a utility outage and a generator run

This capstone follows one real event at Stonebridge from the first alarm to the final report: a full utility outage, the transfer to generators, nearly seven hours of generator operation, the return to utility power, and the review afterwards. It brings together every lesson in the course. Jamal is on the night shift with Dmitri as shift lead; Roz is the critical facilities technician on call. As you read, notice what the technician does and does not do, which numbers matter, and how each lesson's knowledge turns into a decision or a record.

The setting

It is a night in late January. Freezing rain has been falling across the Montérégie since the evening, and Environment Canada has issued a freezing rain advisory. Hydro-Québec's lines are carrying ice, and outages have been reported across the south shore. Both of YUL-2's 120 kV lines run through the same wooded corridor for the last few kilometres before the campus.

Before the shift, Dmitri checked the redundancy whiteboard. Everything is green except one item: GEN-D2 in the Hall D generator block had a starter motor replaced two days ago and passed its test, so it is green but marked for a follow-up check. The Hall A batteries show normal float. Corvid AI is running a large training job in Hall D at about 7.4 MW of IT load.

Step 1: the utility fails

At 02:41:07, the operations room screens fill with alarms. Line 1 lost; 0.2 seconds later, Line 2 lost. Every UPS system on campus reports on battery. The lights in the corridors and offices, which are not on UPS, go out and the emergency lights come on. In the data halls, the IT load does not notice: the UPS inverters keep feeding it exactly as before.

Jamal's first job, which he has rehearsed in drills, is not to do anything to the equipment. It is to watch and record. He starts the event log on his tablet, notes the time, and reads aloud from the screens to Dmitri: both utility lines lost, all UPS systems on battery, generator start signals sent. Dmitri calls Roz and starts the outage communication procedure.

Step 2: the transfer

Over the next ten seconds, the screens follow the sequence from lesson 8. The transfer controllers wait out their two-second start delay and send the start signal at 02:41:09. The Hall A sets fire within two seconds; GEN-A1 and GEN-A2 are at voltage and frequency at 02:41:16, GEN-A3 a fraction of a second later, and the Hall A block transfers at 02:41:17.6, about ten and a half seconds after the utility failed. The Halls B and C block and the Hall D block follow within a second.

One set does not start. GEN-D2's controller shows three cranking attempts and then failure to start. The Hall D block has six sets, N+1, so the paralleling switchgear closes the five that started onto the bus and transfers Hall D's load to them. No load is shed. On the whiteboard, Dmitri turns the Hall D generator block to amber: it is now at N, with no spare.

# Event log analysis: transfer times and start results (illustrative extract)
log = '''
2027-01-21 02:41:07.3 UTILITY Line 1 lost
2027-01-21 02:41:07.5 UTILITY Line 2 lost
2027-01-21 02:41:09.3 GENBLOCK-A start signal
2027-01-21 02:41:16.0 GEN-A1 ready
2027-01-21 02:41:16.4 GEN-A2 ready
2027-01-21 02:41:16.9 GEN-A3 ready
2027-01-21 02:41:17.6 GENBLOCK-A transferred
2027-01-21 02:41:16.8 GEN-D1 ready
2027-01-21 02:41:17.2 GEN-D3 ready
2027-01-21 02:41:17.5 GEN-D4 ready
2027-01-21 02:41:17.9 GEN-D5 ready
2027-01-21 02:41:18.0 GEN-D6 ready
2027-01-21 02:41:18.6 GENBLOCK-D transferred
2027-01-21 02:41:31.4 GEN-D2 failed to start
'''
from datetime import datetime
events = []
for line in log.strip().splitlines():
    d, t, who, *what = line.split()
    events.append((datetime.strptime(d + " " + t, "%Y-%m-%d %H:%M:%S.%f"), who, " ".join(what)))
t0 = min(e[0] for e in events if e[1] == "UTILITY")
for when, who, what in sorted(events):
    dt = (when - t0).total_seconds()
    flag = ""
    if what == "transferred" and dt > 12: flag = "  <- slower than 12 s"
    if "failed" in what: flag = "  <- FAILED START: block now at N"
    print(f"+{dt:5.1f} s  {who:12s}{what}{flag}")

+  0.0 s  UTILITY     Line 1 lost
+  0.2 s  UTILITY     Line 2 lost
+  2.0 s  GENBLOCK-A  start signal
+  8.7 s  GEN-A1      ready
+  9.1 s  GEN-A2      ready
+  9.5 s  GEN-D1      ready
+  9.6 s  GEN-A3      ready
+  9.9 s  GEN-D3      ready
+ 10.2 s  GEN-D4      ready
+ 10.3 s  GENBLOCK-A  transferred
+ 10.6 s  GEN-D5      ready
+ 10.7 s  GEN-D6      ready
+ 11.3 s  GENBLOCK-D  transferred
+ 24.1 s  GEN-D2      failed to start  <- FAILED START: block now at N

The log shows that every block transferred within about eleven seconds, using well under one per cent of the UPS battery reserve. GEN-D2 failed after its third attempt, 22 seconds after the start signal, by which time the other five sets had already taken the load.

Step 3: the plant settles

Over the next minute, the UPS rectifiers ramp up on generator power, the UPS systems return from battery, and the batteries begin to recharge. The building management system restarts the chillers and CRAH units in their planned sequence. In Hall D, the coolant distribution units never stopped, because they are on UPS. Hall A's cold aisle rises by 1.1 °C (2.0 °F) before the CRAHs catch up, and Halls B and C by a similar amount. Nothing reaches an alarm limit.

Roz arrives at 02:58. She and Jamal walk the generator yard together, with hearing protection. Every running set is checked against its controller: load, voltage, frequency, coolant temperature and oil pressure, and a look for leaks or unusual noise. GEN-D2's controller shows the starter motor did not engage; the failure is mechanical, not a fuel or control problem. Roz leaves it in its failed state for the vendor rather than trying to start it again, because a repeated failed start could damage the new starter or the engine's ring gear further, and the block is carrying its load on five sets.

Step 4: AI load on generators

At 03:05, the operations screens begin to show short alarms from the Hall D UPS systems: bypass out of tolerance, clearing a few seconds later, then returning. Roz recognizes the pattern from lesson 6. Corvid AI's training job makes the Hall D load swing up and down by several hundred kilowatts every few seconds as the processors compute and synchronize. On the utility, these swings are absorbed by the grid. On five generators, each swing pulls the frequency down slightly until the governors respond. The UPS inverters keep the IT load steady, but the bypass supply drifts in and out of the UPS's tolerance, so the UPS reports it would not be able to transfer to bypass cleanly at those moments.

# Hall D generator bus frequency during the AI training load (sampled every 0.5 s, illustrative)
samples = [60.02, 59.91, 59.55, 59.32, 59.70, 59.98, 60.05, 59.88, 59.47, 59.28, 59.66, 60.01,
           60.03, 59.90, 59.52, 59.30, 59.71, 59.97, 60.04, 59.89, 59.50, 59.27, 59.68, 60.00]
tolerance = 0.5          # bypass tolerance setting, Hz, illustrative
out = [f for f in samples if abs(f - 60.0) > tolerance]
dips = sum(1 for a, b in zip(samples, samples[1:]) if abs(a - 60) <= tolerance < abs(b - 60))
print(f"Lowest frequency: {min(samples):.2f} Hz; samples outside +/-{tolerance} Hz: {len(out)} of {len(samples)}")
print(f"Excursions in {len(samples) * 0.5:.0f} s: {dips} -> about {dips / (len(samples) * 0.5) * 60:.0f} per minute")

Lowest frequency: 59.27 Hz; samples outside +/-0.5 Hz: 5 of 24
Excursions in 12 s: 4 -> about 20 per minute

This is not dangerous to the IT load at the moment, because the inverters isolate it, but it means that if a UPS needed its static bypass, for example to clear a fault, the transfer might not be clean. Dmitri calls Carmen Ortiz at Corvid AI under the agreed emergency contact procedure. Corvid's agreement with Stonebridge includes a power-capping option for exactly this situation: Carmen's team sets a cap on the training job that smooths its power swings. At 03:22 the cap takes effect, the frequency excursions shrink to a few tenths of a hertz, and the bypass alarms stop.

Step 5: fuel, weather and customers

By 04:00 the campus has run on generators for over an hour. Jamal reads every belly tank and the bulk tanks, and works out the burn rate and the remaining time with the method from lesson 7. With the bulk storage full at the start, the campus has well over two days of fuel. Even so, Dmitri calls the fuel supplier at 04:30 to place a precautionary order, because freezing rain will make the roads slow if the outage lasts.

# Fuel status after the first hours on generator (illustrative)
blocks = {  # block: (load kW, litres per kWh at this load, usable litres at start)
    "Hall A": (2580, 0.287, 72000),
    "Halls B and C": (10900, 0.262, 265000),
    "Hall D": (9700, 0.258, 245000),
}
hours_run = 1.5
for name, (kw, rate, usable) in blocks.items():
    burn = kw * rate                     # litres per hour
    left = usable - burn * hours_run
    print(f"{name:14s}: burning {burn:6,.0f} L/h, {left:8,.0f} L left, {left / burn:5.1f} h remaining")

Hall A        : burning    740 L/h,   70,889 L left,  95.7 h remaining
Halls B and C : burning  2,856 L/h,  260,716 L left,  91.3 h remaining
Hall D        : burning  2,503 L/h,  241,246 L left,  96.4 h remaining

Meanwhile, Jamal handles the customer side. The outage notice, sent automatically when the campus went to generator, tells customers that the site is on generator power, the IT load is unaffected, and updates will follow. He answers three calls from customers' night staff, all asking the same question: is my equipment at risk? His answer is factual: the load is on generator, the UPS batteries are recharging, one generator in the Hall D block is out but the block has enough capacity, and the next update will come in an hour or sooner if anything changes. He does not speculate about when the utility will return.

Step 6: the utility returns

At 08:12, Hydro-Québec restores Line 2, and at 08:40, Line 1. The transfer controllers see stable voltage on both lines, but Stonebridge's storm procedure sets the retransfer to manual after a storm outage: the critical facilities team decides when to return. Roz calls the utility's large-customer line, which reports that crews are still repairing parts of the network nearby and that more switching is expected during the morning.

The decision is a trade-off. Staying on generator protects against the utility failing again, but burns fuel, adds run hours to the environmental permit totals, and keeps the campus on a plant that has no spare in the Hall D block. Returning to utility power restores full redundancy upstream but risks a second interruption, which would put the UPS batteries, not yet fully recharged, through another discharge. Roz and Dmitri wait until the utility confirms its switching is finished, at 09:20, then carry out the closed-transition retransfer at 09:30, block by block. The generators cool down for five minutes and stop at 09:36, returning to automatic. GEN-D2 stays tagged out for the vendor.

Step 7: after the event

When the generators stop, the event is not over. The UPS batteries are still recharging; the VRLA strings in Hall A take several more hours to return to float. Until then, a second outage would have less reserve, so Jamal records each system's state of charge and runtime every hour.

# Battery recharge after the outage: what if the utility failed again? (illustrative)
full_runtime_min = 23.0            # UPS-A2 at today's 600 kW per side, full charge
soc = {"09:40": 0.71, "10:40": 0.82, "11:40": 0.90, "12:40": 0.95, "13:40": 0.99}
for t, s in soc.items():
    runtime = full_runtime_min * s
    note = "enough for a normal transfer and a second attempt" if runtime >= 5 else "SHORT"
    print(f"{t}: state of charge {s * 100:3.0f} %, runtime about {runtime:4.1f} min -> {note}")

09:40: state of charge  71 %, runtime about 16.3 min -> enough for a normal transfer and a second attempt
10:40: state of charge  82 %, runtime about 18.9 min -> enough for a normal transfer and a second attempt
11:40: state of charge  90 %, runtime about 20.7 min -> enough for a normal transfer and a second attempt
12:40: state of charge  95 %, runtime about 21.8 min -> enough for a normal transfer and a second attempt
13:40: state of charge  99 %, runtime about 22.8 min -> enough for a normal transfer and a second attempt

Even at 71 % charge just after retransfer, the Hall A UPS systems could still ride through a normal generator transfer several times over. The figures are logged for the incident report.

Other follow-up items are opened as tickets: GEN-D2's starter motor failure goes to the generator vendor, with a request to examine why a new starter failed after one test; the fuel used is calculated and a delivery scheduled to refill the bulk tanks; the generator run hours, about six hours fifty minutes for each running set, are recorded for the environmental permit; the Hall D power-capping episode goes to the Corvid AI account team and the electrical engineers, to review the generator plant's response to large AI load swings.

Step 8: the incident report

Within two days, Dmitri and Jamal write the incident report, following the format covered in DCT-340. It states facts in order, with times from the event log, and avoids blame. Its main findings:

• The design worked: the utility outage caused no interruption to any customer's IT load. All three generator blocks transferred within about eleven seconds; the UPS batteries used under one per cent of their reserve.
• One generator, GEN-D2, failed to start because of a starter motor fault two days after the starter was replaced. The Hall D block ran at N for nearly seven hours. Action: vendor root cause analysis; a second test start is now required within 48 hours of any starter or starting system repair.
• Large AI load swings caused generator frequency excursions and UPS bypass tolerance alarms until Corvid AI's power cap took effect 41 minutes after the transfer. Action: add the power cap to the automatic generator transfer sequence, with Corvid's agreement, so it applies within a minute of any transfer.
• Cooling restarted as designed; hall temperatures stayed within limits.
• Customer communications worked: the automatic notice went out within two minutes, with hourly updates.
• The manual retransfer after a storm, waiting for the utility to finish switching, avoided a possible second interruption: the utility performed two further switching operations on nearby circuits between 08:40 and 09:20.

The report ends with a timeline, the actions, their owners and due dates. The redundancy whiteboard returns to green four days later, when GEN-D2's new starter has been fitted and the set has passed two test starts and a load test.

What the technician did, and did not do

Looking back, Roz goes through the night with Jamal. He operated no breakers, reset nothing, and did not try to start GEN-D2. What he did was watch, record and report: the event log that the report's timeline came from, the generator yard round with Roz, the fuel readings and calculations, the hourly battery records, the customer calls, and the tickets. Each of those depended on knowing the power chain: knowing that the UPS was supposed to carry the load for seconds, not minutes; that an N+1 block with one failed set is amber, not red; what a bypass tolerance alarm means on generator; and why the retransfer waits. That knowledge is what makes a technician useful on the night everything is tested.