Capstone: tracking down a hot spot and a CDU alarm in Hall D

It is the last Saturday of Jamal's cooling term, and he is on the night shift with Dmitri as shift lead and Roz on call. The outside air at Beauharnois (Ashburn) is unusually warm for the season, 29 °C (84 °F) at ten in the evening. Corvid AI is running a large training job across all of Hall D, and the hall is drawing close to its full 7.9 MW. Two problems will arrive in the next six hours: a hot spot in one rack and a CDU alarm affecting a whole row. Jamal will work both, using everything in this course: heat and flow, the guidelines, the cooling units and plant, airflow, liquid cooling, CDUs and monitoring. This lesson follows the night step by step, then reviews what he did.

22:40: Corvid reports hot accelerators

Carmen Ortiz calls the operations desk. Corvid's own monitoring shows accelerator temperatures in rack D3-07 running about 8 K (14 °F) hotter than in the rest of row 3, on four of its eighteen server trays. The chips are not yet throttling, but the job is sensitive to any slowdown, and she wants to know whether the facility is the cause. Dmitri opens an incident ticket and sends Jamal to look.

Before walking to the hall, Jamal checks DCIM for D3-07 and its neighbours. He writes down what he sees:

• Air inlet at the front of D3-07: 24.8 °C (76.6 °F) top, normal for the row.
• Rack coolant supply temperature: 30.1 °C (86.2 °F), the same as the other racks in row 3.
• Rack coolant return temperature: 43.4 °C (110.1 °F), about 3 K (5.4 °F) higher than the neighbours.
• Rack coolant flow: 128 L/min (34 gpm), against about 167 L/min (44 gpm) on the neighbours.

The air side looks normal, so this is not an airflow problem in the aisle. The coolant arrives at the right temperature but leaves hotter, with less flow. That points to a flow restriction somewhere in the rack's own path, not to the CDU, which feeds the neighbours normally.

# Heat balance for rack D3-07 and a neighbour: Q = flow x density x cp x (return - supply)
rho, cp = 1020, 3900                 # PG25
racks = {  # rack: (flow L/min, supply C, return C)
    "D3-06": (167, 30.1, 40.3),
    "D3-07": (128, 30.1, 43.4),
    "D3-08": (166, 30.1, 40.4),
}
for name, (lpm, sup, ret) in racks.items():
    kw = lpm / 60000 * rho * cp * (ret - sup) / 1000
    print(f"{name}: flow {lpm} L/min, rise {ret - sup:4.1f} K, heat to liquid {kw:5.1f} kW")

D3-06: flow 167 L/min, rise 10.2 K, heat to liquid 112.9 kW
D3-07: flow 128 L/min, rise 13.3 K, heat to liquid 112.9 kW
D3-08: flow 166 L/min, rise 10.3 K, heat to liquid 113.4 kW

The script confirms the reasoning in numbers. With the reduced flow, the coolant warms more to carry the same heat, and the trays furthest from the manifold inlet, where pressure is lowest, get the least flow. The heat balance also tells Jamal that the rack's heat to liquid has not changed: the servers are not drawing more power, so the problem is purely flow.

22:55: at the rack

In the hall, Jamal checks the front of D3-07: doors closed, blanking in place, inlet air normal by handheld. At the back, the rear manifolds look normal, and every tray's quick-disconnects are fully engaged. He opens the change log on his tablet: at 14:10 that afternoon, a technician from the server maker replaced a failed tray in D3-07, position 14. Changing a tray on a liquid-cooled rack means isolating the rack's row connection while the tray's couplings are disconnected and reconnected.

Under the floor tile behind D3-07, with a ticket and cones in place, he looks at the row connection: the supply isolation valve's handle stands at about forty degrees, not fully in line with the pipe. It was reopened after the tray change, but not all the way. Jamal does not touch it. He photographs it, reports to Dmitri by radio, and calls Roz, who is authorized on Hall D's liquid cooling procedures.

23:20: restoring flow

Roz arrives, confirms the finding, and checks the procedure: opening an isolation valve fully on a running rack is permitted, slowly, while watching the rack's flow and pressures. She opens the valve over about a minute. Rack flow rises to 166 L/min (44 gpm), the return temperature falls to 40.3 °C (104.5 °F) within three minutes, and Carmen reports the four trays' accelerators back in line with the row.

Jamal records the readings before and after, the photo and the cause in the incident ticket. Dmitri adds a follow-up: the tray-replacement procedure will gain a step requiring the person who reopens the row valves to record the valve positions and the rack flow afterwards, and a second person to check them. The server maker's technician had followed the server procedure correctly; the gap was in the facility side of the handover.

02:15: CDU high-temperature warning

At 02:15 DCIM raises a warning: CDU-D3A coolant supply 32.1 °C (89.8 °F) against its setpoint of 30 °C (86 °F). Two minutes later CDU-D3B, its partner on row 3, raises the same warning, and CDU-D4A follows. Jamal is at the desk this time. He resists the urge to run to the gallery and spends sixty seconds reading:

• Three CDUs, on two different rows, warming at the same time: the cause is probably something they share.
• Each CDU's facility-side valve is 100 percent open: they are asking for more cooling than they are getting.
• Facility water entering the CDUs: 28.9 °C (84.0 °F), up from its usual 26 °C (79 °F).
• CDU approaches: about 3 K (5.4 °F), normal for these units.

The CDUs themselves are working correctly. Their heat exchangers are clean and their valves are wide open; they simply cannot make 30 °C (86 °F) coolant from 28.9 °C (84 °F) facility water. The problem is upstream, in the plant that makes the facility water.

# Separate a CDU fault from a supply problem: approach = coolant supply - facility water in
setpoint = 30.0
cdus = {  # CDU: (facility water in, coolant supply, facility valve %)
    "CDU-D3A": (28.9, 32.1, 100), "CDU-D3B": (28.9, 31.9, 100), "CDU-D4A": (28.9, 32.0, 100), "CDU-D1A": (28.9, 30.0, 92),
}
for name, (fw, sup, valve) in cdus.items():
    approach = sup - fw
    if sup > setpoint and valve >= 99 and approach < 4.0:
        verdict = "CDU healthy; facility water too warm"
    elif sup > setpoint and approach >= 4.0:
        verdict = "CDU heat exchanger or strainer suspect"
    else:
        verdict = "OK"
    print(f"{name}: facility {fw:.1f}, supply {sup:.1f}, valve {valve} %, approach {approach:.1f} -> {verdict}")

CDU-D3A: facility 28.9, supply 32.1, valve 100 %, approach 3.2 -> CDU healthy; facility water too warm
CDU-D3B: facility 28.9, supply 31.9, valve 100 %, approach 3.0 -> CDU healthy; facility water too warm
CDU-D4A: facility 28.9, supply 32.0, valve 100 %, approach 3.1 -> CDU healthy; facility water too warm
CDU-D1A: facility 28.9, supply 30.0, valve 92 %, approach 1.1 -> OK

Jamal tells Dmitri what he has found and that the warning is a plant problem, and opens the plant screen. One of the four banks of dry coolers serving the Hall D loop shows a fault: its fans are stopped. With the warm night and a quarter of the dry cooler capacity gone, the remaining banks cannot hold the facility water at 26 °C (79 °F).

02:25: the plant response

The licensed plant operator on duty confirms the fault: a fan drive on dry cooler bank 3 has tripped on a fault code. The plant has two responses. The first is to switch on the adiabatic pre-cooling pads on the remaining three banks, which wets the incoming air and lowers its temperature by several degrees on a warm, fairly dry night. The second is to bring the trim chiller on line to cool the Hall D facility water directly. The operator does both, following the plant's high-load procedure, and calls the dry cooler service contractor.

Within fifteen minutes the facility water entering the CDUs falls back to 26.4 °C (79.5 °F), and the CDU supply temperatures return to setpoint. The highest coolant supply reached was 32.6 °C (90.7 °F), inside the warning band, below the critical threshold, and well within the facility water class the Hall D servers are rated for. Carmen reports that no accelerators throttled.

Jamal's part in this was small but important. He identified a shared cause from the pattern of alarms, did not chase three separate CDU problems, gave the shift lead a clear explanation within minutes, and kept the hall's rack temperatures under watch while the plant team worked. He also checked that the CDUs' own redundancy was intact, since a pump failure during a high-temperature event would have been much worse.

05:30: writing it up

Before the morning handover Jamal writes the shift report. Roz has taught him the format: what happened, when, what was found, what was done, what remains, and what should change. His summary for the two events:

The last row is his own addition. The rack's flow had been low for eight hours, since the afternoon tray change, and no facility alarm had sounded, because the low-flow threshold was set as a fixed value well below the normal flow. Corvid's chip monitoring caught it first. A deviation alarm comparing each rack with its row median, like the one Jamal used for inlet sensors in Lesson 11, would have caught it within minutes of the change.

from statistics import median
# Flow-deviation alarm: flag any rack whose coolant flow differs from the row median by more than 10 %
row3 = {"D3-01": 168, "D3-02": 165, "D3-03": 167, "D3-04": 166, "D3-05": 169,
        "D3-06": 167, "D3-07": 128, "D3-08": 166, "D3-09": 164, "D3-10": 168}   # L/min
mid = median(row3.values())
for rack, flow in row3.items():
    dev = (flow - mid) / mid
    if abs(dev) > 0.10:
        print(f"ALARM {rack}: {flow * 1:.0f} L/min is {dev:+.0%} from the row median of {mid * 1:.0f} L/min")
print("checked", len(row3), "racks")

ALARM D3-07: 128 L/min is -23% from the row median of 166 L/min
checked 10 racks

Review: what the night used

At the end of the term, Roz goes through the night with Jamal and ties each step to the course:

• Heat equals power, and heat equals flow times specific heat times temperature rise: the heat balance showed the rack's heat had not changed, so the problem was flow (Lessons 1 and 9).
• Compliance and chip health depend on the right measurement: air inlets were normal, so the hot spot was not an air problem (Lesson 3).
• Compare with neighbours and with related readings before acting: one rack versus its row, three CDUs versus the plant (Lesson 11).
• Shared causes show as patterns: several units warming together points upstream (Lessons 4 and 5).
• The approach temperature separates a CDU problem from a supply problem (Lesson 9).
• Only authorized people operate liquid cooling valves and couplings; technicians observe, record and report (Lessons 8 and 9).
• Every event ends with a record and, where needed, a change to procedures or alarms (Lesson 11).

Roz signs off Jamal's cooling term. His next course, DCT-310, moves from the facility to the equipment inside the racks.