Monitoring temperature, humidity, pressure and leaks

Most cooling problems at Stonebridge are found not by someone walking past a rack but by a number changing on a screen. YUL-2 and IAD-3 each have several thousand cooling sensors: rack inlets, cooling unit supply and return air, underfloor and containment pressures, humidity, chilled-water and coolant temperatures, pressures and flows, and kilometers of leak detection cable. They feed the building management system, BMS, which controls the plant, and the data center infrastructure management system, DCIM, which shows the halls rack by rack. Dmitri puts Jamal on the monitoring desk for two night shifts. "The sensors don't lie," he says, "but they don't always tell the truth either. Your job is to know which." This lesson covers the sensors, the alarms they raise and how a technician responds. DCT-330 covers DCIM, ticketing and change management in depth.

What is measured

A typical air-cooled hall at Stonebridge measures:

• Rack inlet temperature at three heights on the front of every rack, and sometimes rack exhaust temperature at the back.
• Supply and return air temperature at every cooling unit, plus fan speed, valve position and filter pressure drop.
• Underfloor pressure at several points, or pressure across containment in slab halls.
• Room humidity and dew point at a few representative points, not at every rack.
• Chilled-water supply and return temperatures and pressures at the hall's main pipes and often at each unit.
• Leak detection along pipe runs, under cooling units and under liquid-cooled racks.

A liquid-cooled hall adds CDU coolant supply and return temperatures, flows, pressures, reservoir levels and coolant quality, and often per-rack coolant temperatures and flows. The servers themselves report chip temperatures, fan speeds and inlet temperatures through their management controllers, which customers watch in their own tools and which some operators also collect.

Sensors and their limits

Each type of sensor has its own weaknesses:

• Temperature sensors are accurate but sensitive to where they are mounted. A rack inlet sensor taped to a door that is sometimes open, or placed behind a cable bundle, reads air that is not what the servers see.
• Humidity sensors drift over time and must be checked against a reference. Their readings are only as good as their last calibration.
• Pressure sensors measure very small differences, a few pascals, and are affected by air currents near their tubes. A kinked or disconnected tube gives a reading of zero.
• Flow meters on coolant lines can read wrong if air is trapped in the pipe or if the flow is outside their range.
• Leak detection cable can be triggered by condensation, cleaning water or a mop resting on it, and can be damaged by being stepped on or pinched under equipment.

Wireless sensors, common on rack doors, depend on batteries and radio coverage. A sensor that has stopped reporting shows its last value, which can look normal while the rack overheats. DCIM marks stale readings, and technicians replace batteries on the schedule.

Alarm levels

Not every abnormal reading needs the same response. Stonebridge's alarm philosophy uses three levels, and each sensor type has its own thresholds:

• Advisory: a reading outside its normal band but inside safe limits, such as a rack inlet above the recommended range but inside the allowable range. Investigate during the shift; no immediate action.
• A warning: a reading that needs action soon, such as a cooling unit fault that reduces redundancy, or an inlet approaching the allowable limit. Investigate now and report to the shift lead.
• Critical: a reading that threatens equipment or service, such as a leak alarm, a CDU flow failure or inlet temperatures above the allowable limit. Respond at once by the emergency procedure.

Thresholds are set to give warning before trouble while avoiding nuisance alarms. An alarm that sounds every day for no reason trains people to ignore it; good operators review alarm counts each month and fix nuisance alarms at their cause rather than by raising the threshold.

# Grade readings into advisory, warning and critical using the site's thresholds (illustrative)
thresholds = {  # sensor type: (advisory above, warning above, critical above)
    "rack inlet F": (80.6, 86.0, 89.6), "CDU supply F": (87.8, 91.4, 95.0),
}
readings = [("rack inlet F", "A-22-03 top", 81.7), ("rack inlet F", "B-04-09 top", 87.4), ("CDU supply F", "CDU-D2B", 92.1), ("CDU supply F", "CDU-D1A", 86.2),]
for kind, where, value in readings:
    adv, warn, crit = thresholds[kind]
    level = "critical" if value > crit else "warning" if value > warn else "advisory" if value > adv else "normal"
    print(f"{where:22s} {kind:16s} {value:6.1f}  {level}")

A-22-03 top            rack inlet F       81.7  advisory
B-04-09 top            rack inlet F       87.4  warning
CDU-D2B                CDU supply F       92.1  warning
CDU-D1A                CDU supply F       86.2  normal

Rate-of-change alarms

A reading can be dangerous because it is changing fast even when it is still inside its limits. A rack inlet rising 5.4 °F in five minutes, a coolant supply temperature climbing steadily, or a reservoir level falling slowly all point to a problem developing. Rate-of-change alarms watch the slope of a reading rather than its level. They give earlier warning of failures such as a cooling unit stopping, a valve sticking or a slow leak.

Rate alarms need care to avoid false alarms from normal events. When a cooling unit starts or stops as part of the normal rotation, nearby inlet temperatures shift for a few minutes. Operators set rate thresholds and averaging times so that these normal steps do not trip them.

# Rate-of-change alarm on a rack inlet: averaged slope over the last three samples (one minute apart)
limit = 0.9    # F                     # per minute
series = {
    "unit stopped":  [73.4, 73.6, 73.4, 75.0, 76.8, 78.4, 80.2],
    "standby swap":  [73.4, 73.6, 75.0, 75.6, 74.5, 73.8, 73.6],
}
for name, values in series.items():
    tripped = None
    for i in range(3, len(values)):
        slope = (values[i] - values[i - 3]) / 3
        if slope > limit and tripped is None:
            tripped = i
    print(f"{name:13s} readings {values}")
    print(f"{'':13s} rate alarm: {'at minute ' + str(tripped) if tripped is not None else 'not tripped'}")

unit stopped  readings [73.4, 73.6, 73.4, 75.0, 76.8, 78.4, 80.2]
              rate alarm: at minute 4
standby swap  readings [73.4, 73.6, 75.0, 75.6, 74.5, 73.8, 73.6]
              rate alarm: not tripped

The first series is a rack whose inlet climbs steadily after a cooling unit stops; the rate alarm trips at minute 4, while the inlet is still well inside the recommended range and several minutes before any level alarm would sound. The second series is the normal step when a standby unit swaps in; it rises briefly and settles, and the averaged rate stays under the threshold.

Is the sensor right?

When an alarm arrives, the first question is whether the reading is real. Dmitri's rules:

• Compare with neighbors: if one rack inlet reads 90 °F and its neighbors read 73 °F, either that rack has a local problem or its sensor is wrong. Go and measure.
• Compare with related readings: a cooling unit reporting a supply air temperature of 95 °F with a fully open valve and cold water entering is either badly faulty or misreporting.
• Check the history: a sensor that jumped instantly from normal to an extreme value, with no gradual change, often has a fault. Real temperatures rarely jump.
• Check whether it is stale: a reading that has not updated for an hour is not current information.
• Measure with a calibrated handheld when in doubt, and record both readings.

A faulty sensor is still reported and fixed, because a missing or wrong sensor is a gap in the protection. But a false reading does not trigger an emergency response once the technician has confirmed, with an independent measurement, that the air or water is normal.

from statistics import median
# Flag sensors that disagree with their neighbors by more than the site's tolerance
tolerance = 5.4
row = {  # rack: top inlet reading
    "A-25-03": 74.8, "A-25-04": 75.9, "A-25-05": 75.2, "A-25-06": 93.2, "A-25-07": 76.3, "A-25-08": 75.0,
}
mid = median(row.values())
for rack, value in row.items():
    dev = value - mid
    note = "  <- check: sensor or local problem" if abs(dev) > tolerance else ""
    print(f"{rack}: {value:5.1f} ({dev:+.1f} from row median){note}")

A-25-03:  74.8 (-0.8 from row median)
A-25-04:  75.9 (+0.3 from row median)
A-25-05:  75.2 (-0.4 from row median)
A-25-06:  93.2 (+17.6 from row median)  <- check: sensor or local problem
A-25-07:  76.3 (+0.7 from row median)
A-25-08:  75.0 (-0.6 from row median)

Leak alarms

Leak detection deserves its own procedure because water and electricity do not mix and time matters. When a leak alarm sounds:

• Note the zone and, if the system reports it, the distance along the cable to the wet point.
• Go to the location at once with a flashlight and absorbent pads. Tell the shift lead you are going.
• Find the wet point and the source: a dripping fitting, a condensate pan overflowing, a cleaning bucket, a roof leak.
• If equipment is at risk, protect it as far as can be done safely: place pads, move a portable item, close a drip tray. Do not touch electrical equipment that is wet.
• Isolate the source only if trained and authorized to operate that valve; otherwise call the facilities technician.
• Record the alarm, the cause and the actions; reset the cable only after it is dry.

Every leak alarm is visited, even at three in the morning, even after a false alarm on the same cable an hour earlier.

Trending and reporting

Many problems are found not in an alarm but in a trend. Technicians review weekly trend charts for slow changes: a CRAH whose valve opens a little more each week as its coil fouls; a CDU whose approach grows; a hall whose average inlet temperature creeps upward as racks are added; a filter whose pressure drop climbs toward its change point. Stonebridge's weekly cooling report lists the five units with the largest change in the week and the racks with the highest inlet temperatures.

Reports also go to customers. Many colocation contracts promise inlet conditions within the ASHRAE recommended range a stated percentage of the time, measured by the facility's sensors. Accurate, well-placed sensors protect both the customer and the operator, and good records settle disputes like the one in Lesson 3.

A night on the monitoring desk

Jamal's first night produces a typical list:

• 02:10, advisory: rack B-07-12 inlet top 81.3 °F. Walk-through finds a customer's empty slots without blanking; panels fitted from stock under the house rule; reading falls to 76.6 °F.
• 03:35, warning: CRAH-A07 communication lost. Local display shows it cooling normally; controls contractor ticket raised.
• 04:20, critical: leak alarm, Hall A zone 3, 60 ft along the cable. Visit finds a cleaning contractor's mop bucket overflowed near the cable; area dried, cable reset, contractor's supervisor told.
• 05:50, advisory: hall dew point 13.1 °F, just below the recommended lower limit, on a very cold, dry morning. Logged; no action under site policy.

None of the four needed heroics, but each needed a visit, a judgment and a record. That is what monitoring work mostly is.