Airflow management and liquid cooling

Lesson 5 explained how heat moves from servers to the outdoors and introduced hot and cold aisles, cooling units and liquid cooling. This lesson goes inside the white space in more detail. The first half deals with air: how much air the servers need, how it is delivered, how it goes wrong through bypass and recirculation, and how operators measure and fix it. The second half deals with liquid: how direct-to-chip systems are arranged, how coolant distribution units link the racks to the building, what temperatures and flows they use, and the practical issues of water quality, leaks and mixed air and liquid halls. Together they explain why a hall that looks adequate on paper can still have hot spots, and why AI has made liquid cooling routine.

Supplying the right amount of air

Every server draws a certain airflow through its fans, set by its power and temperature rise (Lesson 2). The cooling units must supply at least the total airflow of the servers, plus a margin, at a suitable temperature. Using \(\rho c_p \approx 1.2\) kJ/(m³·K) for air:

If the cooling units supply less air than the servers draw, the servers pull hot exhaust air around the ends and tops of the rows (recirculation), raising their inlet temperatures. If the units supply much more air than the servers draw, the excess bypasses the servers and returns to the units without picking up heat (bypass), wasting fan energy and lowering the return temperature, which reduces cooling unit capacity and efficiency.

Raised floors and slab floors

Older and some current data centres supply cold air through a raised floor: a plenum 450 mm to 1,200 mm deep beneath removable floor tiles, pressurized by the cooling units, with perforated tiles placed in front of racks. Each tile delivers a flow that depends on its open area and the plenum pressure; a typical 25% perforated tile delivers roughly 0.2 m³/s to 0.4 m³/s, enough for perhaps 3 to 6 kW of IT load, while high-flow grates deliver more. Cables and pipes under the floor obstruct airflow and cause uneven pressure, and tiles too close to cooling units may even draw air downward because of high velocity under the floor.

Many modern facilities use slab floors with air delivered from above or through fan walls on one side of the hall into contained cold aisles or flooded rooms. Slab floors carry heavy racks more easily, avoid underfloor obstructions and cost less; cables and busways run overhead. Raised floors remain useful where pipework for liquid cooling or legacy layouts favour them.

Measuring air management

Operators measure how well air is managed using sensors at rack inlets and outlets and at cooling units:

• Rack inlet temperature: the temperature that matters to the equipment, compared with the ASHRAE recommended range of 18 °C to 27 °C (Lesson 5).
• Temperature rise across the cooling units versus across the servers: if units see a much smaller rise than the servers produce, air is bypassing.
• Return Temperature Index (RTI): the ratio of the cooling units' temperature rise to the IT equipment's rise; 100% is balanced, below 100% indicates bypass, above 100% indicates recirculation.
• Rack Cooling Index (RCI): how many inlet temperatures fall outside the recommended range.

Computational fluid dynamics (CFD) models simulate airflow in a hall during design and before major changes, showing where hot spots will appear and testing remedies. Thermal imaging and wireless temperature sensors at the top, middle and bottom of rack fronts give operators continuous feedback.

Fixing hot spots: practical measures

Most air management problems have simple, inexpensive fixes:

• Install blanking panels in every empty rack unit and seal gaps between racks and at the bottom of racks.
• Seal cable openings in raised floors with brush grommets.
• Contain hot or cold aisles with doors and roofs; seal containment carefully, including above doors and around cable trays.
• Place perforated tiles only in cold aisles, matched to the load.
• Avoid placing high-density racks at the ends of rows, where recirculation is worst.
• Manage cables so they do not block server exhaust.
• Control cooling unit fans from rack inlet temperatures or containment pressure rather than return temperature.

With good air management, supply temperatures can rise toward the upper end of the recommended range, which extends economizer hours (Lesson 5) and reduces chiller energy.

Why liquid cooling

Water carries about 3,500 times more heat per unit volume than air for the same temperature rise. Air cooling becomes impractical above roughly 30 kW to 50 kW per rack because the airflow, fan noise and heat sink size grow too large, and because the processor chips themselves now dissipate 700 W to over 1,000 W each in small areas. Liquid cooling moves heat at the chip, at higher temperatures, with less pumping energy, and allows much denser racks.

Direct-to-chip cooling systems

In direct-to-chip (cold plate) cooling, the liquid loop is divided into two circuits separated by a heat exchanger:

• The technology cooling system (TCS) is the secondary loop running through the racks and cold plates. It uses treated water or a water-propylene glycol mixture, clean and filtered because the cold plates have fine channels, at controlled temperature and pressure.
• The facility water system (FWS) is the building's primary loop, connected to chillers, cooling towers or dry coolers.

A coolant distribution unit (CDU) contains the heat exchanger between the two loops, pumps, filters, expansion tanks and controls. CDUs may sit in each rack (rack-mounted), at the end of a row (in-row) or in a separate gallery serving many racks. Manifolds in each rack distribute coolant to each server through drip-free quick-disconnect couplings.

Cold plates usually capture 70% to 90% of a server's heat; the rest, from memory, power supplies, network cards and drives, still goes to air. A liquid-cooled hall therefore still needs air cooling, at a smaller scale, often through rear-door heat exchangers or fan walls.

Temperatures and warm-water cooling

Cold plates can operate with surprisingly warm coolant. ASHRAE defines liquid cooling classes by facility water supply temperature, from W17 (up to 17 °C) to W45 (up to 45 °C) and above. Many AI systems accept TCS supply temperatures of 30 °C to 45 °C. Warm water means that in most climates heat can be rejected with dry coolers or cooling towers without chillers for most or all of the year, cutting energy use; it also produces return water warm enough to be useful for heat reuse (Lesson 10). The approach temperature, the difference between the facility water and TCS temperatures across the CDU heat exchanger, is typically 2 K to 5 K.

Water quality, leaks and reliability

Liquid near electronics raises understandable concern, and good practice addresses it:

• Water quality: the TCS is filled with treated water or glycol mixtures with corrosion inhibitors and biocides, filtered to protect fine cold plate channels (often to 50 µm or finer), and tested regularly. Wetted materials must be compatible to avoid galvanic corrosion.
• Leak prevention and detection: drip-free quick disconnects, pressure testing during commissioning, leak detection cables under racks and manifolds, and controls that isolate a leaking rack automatically.
• Negative-pressure systems: some designs keep the TCS below atmospheric pressure so that a leak draws air in rather than pushing liquid out.
• Redundancy: CDUs and pumps are installed N+1 or better, and the cooling path is as important as the power path; an AI rack without coolant overheats within seconds to minutes.
• Thermal ride-through: when power fails, pumps must keep running on UPS power, or the chips overheat before generators start.

Immersion cooling

Immersion cooling submerges servers in a dielectric fluid. In single-phase immersion, the fluid (usually a hydrocarbon or synthetic oil) stays liquid and is pumped through a heat exchanger. In two-phase immersion, the fluid boils at the hot components and condenses on a coil above, carrying heat very effectively; however, many two-phase fluids are fluorinated chemicals (PFAS) facing regulatory restrictions, which has slowed adoption. Immersion removes nearly all heat to liquid and eliminates server fans, but tanks change floor layouts, the fluid adds weight, and servicing requires lifting equipment out of the fluid. It is used in some cryptocurrency, edge and high-performance installations, while direct-to-chip has become the mainstream approach for AI.

Planning for mixed halls

Most new halls must accommodate both air-cooled and liquid-cooled equipment, often in unknown proportions over their life. Common strategies include installing facility water piping and CDU connection points throughout the hall from the start, even where initial racks are air-cooled; designing air systems that can be reduced as liquid takes over; reserving floor space and structural capacity for CDUs; and choosing cooling plants able to supply both cold water for air cooling and warm water for liquid cooling. Retrofitting liquid cooling into an occupied hall is possible but disruptive, so these early decisions are among the most important in modern data centre design.