Capstone: planning and running a change under a MOP

This capstone follows one change at Stonebridge from the first sign of a problem to the closed ticket, using every part of the course: monitoring and alarm design, DCIM and capacity, tickets and customer communication, change management, the MOP, and the documents. The change itself is small: replacing a drifting temperature sensor on a coolant distribution unit in Hall D and correcting its configuration in the BMS. Small changes on critical systems are where most procedure errors happen, precisely because they seem routine. Jamal plans much of this one himself, with Roz as the technical owner and Dmitri running the desk on the night.

Step 1: noticing the drift

Hall D's CDUs each have two supply temperature sensors: a primary that the CDU controller uses for control, and a secondary that the BMS reads separately as a check (DCT-230 lesson 9). For CDU-D2B, which serves part of row 2 with CDU-D2A and CDU-D2C as an N+1 group, the two used to agree within about 0.2 °F. During his hourly trend checks over a week, Jamal notices the secondary reading creeping upward relative to the primary.

# CDU-D2B: primary vs secondary supply temperature, daily averages over two weeks
primary   = [86.0, 86.2, 86.0, 85.8, 86.0, 86.2, 86.0, 86.0, 85.8, 86.0, 86.2, 86.0, 86.0, 86.2]
secondary = [86.2, 86.4, 86.4, 86.4, 86.5, 86.9, 86.9, 87.1, 87.1, 87.4, 87.6, 87.6, 87.8, 88.0]
limit = 0.9          # agreed limit for sensor disagreement
for day, (p, s) in enumerate(zip(primary, secondary), 1):
    diff = s - p
    flag = "  <- over limit" if abs(diff) > limit else ""
    print(f"day {day:2d}: primary {p:5.1f}  secondary {s:5.1f}  difference {diff:+.2f}{flag}")
growth = (secondary[-1] - primary[-1] - (secondary[0] - primary[0])) / (len(primary) - 1)
print(f"Drift: about {growth:+.3f} per day")

day  1: primary  86.0  secondary  86.2  difference +0.20
day  2: primary  86.2  secondary  86.4  difference +0.20
day  3: primary  86.0  secondary  86.4  difference +0.40
day  4: primary  85.8  secondary  86.4  difference +0.60
day  5: primary  86.0  secondary  86.5  difference +0.50
day  6: primary  86.2  secondary  86.9  difference +0.70
day  7: primary  86.0  secondary  86.9  difference +0.90  <- over limit
day  8: primary  86.0  secondary  87.1  difference +1.10  <- over limit
day  9: primary  85.8  secondary  87.1  difference +1.30  <- over limit
day 10: primary  86.0  secondary  87.4  difference +1.40  <- over limit
day 11: primary  86.2  secondary  87.6  difference +1.40  <- over limit
day 12: primary  86.0  secondary  87.6  difference +1.60  <- over limit
day 13: primary  86.0  secondary  87.8  difference +1.80  <- over limit
day 14: primary  86.2  secondary  88.0  difference +1.80  <- over limit
Drift: about +0.123 per day

Which sensor is wrong? The primary agrees with CDU-D2A and D2C, which share the same facility water and serve similar loads, and with the rack inlet coolant temperatures reported by Corvid's own monitoring. The secondary is the odd one out: a drifting sensor, probably a failing element or a degrading connection. It is not dangerous yet, because the CDU controls on the primary, but the secondary is the independent check that would catch a primary failure, and its alarm will start raising false minor alarms within a few days. Jamal records the evidence and opens an incident ticket, P3: degraded, redundant, no load at risk.

Step 2: from incident to change

Roz confirms the analysis and decides the sensor should be replaced. The CDU maker's manual shows the sensor sits in a thermowell, a closed pocket in the pipe, so it can be removed without opening the coolant loop. Replacing it still needs the CDU out of service for about 40 minutes: the controller must be switched to a safe state while the sensor input is disconnected, and the maker recommends taking the unit to standby. That makes it a change.

Jamal drafts the change request with Roz:

• What and why: replace CDU-D2B secondary supply temperature sensor (drifting, evidence attached) and update its BMS point configuration.
• Systems and customers: CDU group 2 in Hall D, serving Corvid AI's row 2 racks; no other customers.
• Redundancy during the work: the group is N+1 with three CDUs; with D2B in standby, D2A and D2C carry the load at N. Corvid's row 2 load is about 1.3 MW, within the two remaining CDUs' rating (DCT-230 lesson 9).
• Risk: impact 3 (a second CDU failure during the 40 minutes would force a load reduction on row 2), likelihood 2 (simple work, but the CDU must be taken out and returned correctly). Score 6: approval by the critical facilities manager.
• Timing: Tuesday night 23:00 to 01:00, inside the change window, and outside Corvid's first-week freeze.
• People: Roz (technical lead), the controls contractor's technician Lee Brandt (BMS configuration), Jamal (reader and desk liaison in the gallery), Dmitri on the desk.
• Back-out: if the new sensor does not read correctly, refit the old one and return D2B to service; D2B can also return to service without a secondary sensor, with its secondary alarm shelved, as a last resort.

The capacity check comes from DCIM and the BMS. Jamal confirms that row 2's measured coolant heat load over the past month peaked at 1.34 MW and that D2A and D2C together are rated well above that, with the facility water temperature normal for the season. He attaches the figures.

Step 3: the customer notice

Because Corvid's cooling redundancy drops during the work, a notice goes to Corvid through the portal ten business days ahead. It says what will be done, when, that cooling redundancy for row 2 will be at N for about 40 minutes, that no change in coolant temperature or flow is expected, and that Stonebridge will call Corvid's operations team at the start and end of the work. Carmen Ortiz acknowledges and asks one question: will their power cap from the DCT-220 storm night be needed? The answer, recorded in the ticket, is no: power is unaffected, and a cooling-related cap would only be requested if a second CDU failed.

Step 4: the MOP and the walk-down

Roz writes the MOP from the standard CDU sensor replacement MOP, adding the steps specific to D2B and the BMS work. Jamal reviews it against the CDU flow diagram, the BMS point list and the DCIM records. The walk-down, two days before the change, finds two things.

• The replacement sensor in stores is a newer model with a different element type from the original. The controller accepts both, but the input must be reconfigured. The MOP gains a step for Lee to change the input type, with a verification that the reading matches the primary within the agreed limit.
• In the BMS, the three CDUs in group 2 appear next to each other in a list, with names that differ by one letter. Jamal suggests that the maintenance mode step say not only 'set CDU-D2B maintenance mode' but also how the desk will verify which unit is in maintenance mode. Roz adds: 'Desk confirms on the BMS that CDU-D2B, and only D2B, shows maintenance mode, and reads back the unit name to the gallery.'

The engineer reviews the revised MOP, the critical facilities manager approves the change, and the CAB notes it. The MOP's final revision is fixed.

Step 5: the night of the change

At 22:45, Roz, Lee and Jamal meet in the Hall D gallery; Dmitri is on the desk. The prerequisites are checked aloud: change approved, notice acknowledged, parts and tools checked, redundancy board green apart from nothing else in Hall D, CDU-D2A and D2C running normally, facility water normal, no other work in Hall D tonight. At 23:02 Jamal calls Corvid's night operations team: work starting. Dmitri logs it.

Steps 5 and 6 go as written: Roz confirms on the CDU panels that D2A and D2C are healthy, and the desk records their flows and supply temperatures as the baseline. Step 7 is the maintenance mode. Lee sets maintenance mode in the BMS from his laptop, and the desk carries out the verification exactly as the MOP says. Dmitri reads the BMS: 'Maintenance mode is showing on CDU-D2A. D2B is normal.'

Everyone stops. Lee checks his screen: in the list, D2A and D2B are next to each other, and he selected the line above. Nothing has been done to either CDU yet; D2A is running normally, but its alarms are now suppressed, which means a real failure of D2A would not alarm. Under the MOP's rule for any unexpected result, Roz calls a stop, and the team uses the step's rollback: Lee clears maintenance mode on D2A, Dmitri confirms D2A's alarms are live and reads back 'D2A normal, alarms active', and then Lee sets maintenance mode on D2B. Dmitri confirms: 'D2B, and only D2B, in maintenance mode.' Roz records the stop on the MOP with the time, 23:14, and the work continues. The whole interruption took four minutes.

The remaining steps go to plan. At 23:20 Roz takes D2B to standby at its panel; the desk sees D2A and D2C pick up the flow, with row 2 coolant supply steady at 86.0 °F. At 23:31, after the hold point where Dmitri confirms the group is stable at N, Roz withdraws the old sensor from its thermowell: the point of no return. At 23:38 the new sensor is in, and Lee changes the input type. With D2B still in standby, both its sensors read the coolant in the pipe; the new secondary reads within 0.2 °F of the primary. Roz returns D2B to service at 23:52; the three CDUs share the load again. At 23:58 Lee clears maintenance mode on D2B, and the desk verifies that D2B's alarms are active and that no CDU in group 2 has suppressed alarms. Jamal calls Corvid at 00:03: work complete, no effect on their service.

Step 6: close-out

The close-out follows the MOP's last section and the change close-out checklist:

• Facility state: redundancy board green; CDU group 2 at N+1; no suppressed or shelved alarms in Hall D.
• DCIM: the old sensor's asset record set to removed, the new sensor added with its model and serial against CDU-D2B.
• BMS point list and alarm database: the point's input type and sensor model updated; the drift alarm limit unchanged.
• Documents: the CDU flow diagram needs no change; the point list revision goes to the controls document owner.
• Tickets: the incident linked to the change and resolved; the change closed as 'completed with a deviation', with the stop described.
• Customer: Corvid notified by portal that the work is complete.

# Change closure summary for CHG-5640
planned_steps, done_steps = 18, 18
stops = [("23:10", "maintenance mode set on CDU-D2A instead of D2B; found by desk verification; rolled back")]
window_min, used_min = 120, 61          # change window 23:00-01:00; work 23:02-00:03
at_n_min = 32                           # 23:20-23:52
print(f"Steps completed: {done_steps}/{planned_steps}")
print(f"Time used: {used_min} of {window_min} min ({used_min / window_min * 100:.0f} % of the window)")
print(f"Time with group 2 at N: {at_n_min} min (planned 40)")
print(f"Stops: {len(stops)}")
for t, what in stops:
    print(f"  {t}: {what}")
result = "completed with a deviation" if stops else "completed as planned"
print("Closure:", result)

Steps completed: 18/18
Time used: 61 of 120 min (51 % of the window)
Time with group 2 at N: 32 min (planned 40)
Stops: 1
  23:10: maintenance mode set on CDU-D2A instead of D2B; found by desk verification; rolled back
Closure: completed with a deviation

Step 7: learning from the stop

Nothing went wrong for the load. But for four minutes, CDU-D2A's alarms were suppressed while D2A was running, and if the team had not verified the step, D2A could have stayed suppressed while D2B was out of service, leaving a CDU group at N with one of its two working units unable to alarm. That is exactly the hidden state that change management and MOPs exist to prevent, and it is worth more than a line in the ticket.

Dmitri writes a short post-change review, factual and blame-free, in the format DCT-340 describes for incidents. Its findings: the MOP's verification step worked as designed and caught the error immediately; the cause was two near-identical equipment names next to each other in the BMS list, with selection by mouse click; the rollback was quick because the MOP had one. Its actions: the controls contractor configures the BMS to ask the user to type the equipment tag before setting maintenance mode on any CDU, CRAH or UPS-related point; the standard CDU MOP gets Jamal's wording for the verification step; and the near miss is added to the desk's training examples.

The review is presented at the next CAB. Lee, who made the selection error, presents it with Roz, and the meeting spends more time on the BMS design than on the mistake. That is how near misses become improvements rather than secrets.

What the course looked like on one night

Looking back with Roz, Jamal maps the night to the course. Trend watching on the desk (lessons 1 and 2) found a drifting sensor that had not yet alarmed. Knowing the sensors and the alarm design (lesson 3) told him which reading to trust. The incident ticket and its priority (lesson 7) recorded the problem; the capacity check (lesson 6) used DCIM (lesson 5) to show the group could run at N; the customer notice and the call (lesson 8) kept Corvid informed. Change management (lesson 9) set the risk and approval; the MOP and walk-down (lesson 10) found two issues before the night and contained the third on the night; the documents and records (lesson 11) were updated before the change was closed. None of it required Jamal to operate a CDU. All of it required him to know what the CDU was doing.