Direct-to-chip and immersion liquid cooling

A single AI accelerator now dissipates around 1000 W from a package the size of a palm, and a rack of them 100 kW or more. As Lesson 1 showed, air cannot carry that much heat through a rack at reasonable speeds, and fan power alone would become a large fraction of the load. Liquid cooling brings a liquid, which carries about 3500 times as much heat per unit volume as air, to the chips themselves. Once confined to supercomputers, it has become the default for AI training clusters and is reshaping how data centers are designed: warmer water, fewer chillers, new plumbing on the data hall floor, and new opportunities to reuse the heat.

This lesson covers direct-to-chip cooling with cold plates, coolant distribution units and the secondary loop, ASHRAE's liquid cooling classes and the temperature chain from chip to atmosphere, immersion cooling, the practical issues of leaks, water quality, materials and serviceability, and heat reuse.

Direct-to-chip cooling

In direct-to-chip (DTC) cooling, a cold plate, a metal block (usually copper) with internal channels or microfins, is clamped onto each processor or accelerator with a thermal interface material. Coolant flows through the cold plates, absorbing heat by conduction and convection, and carries it out of the server through flexible hoses, manifolds on the rack and quick-disconnect couplings.

• Single-phase DTC circulates water or a water-glycol mixture (often 25% propylene glycol, PG25, for freeze and biological protection) that stays liquid. It is the most common approach.
• Two-phase DTC pumps a dielectric refrigerant to the cold plates, where it boils; the latent heat of boiling absorbs large heat fluxes at a nearly constant temperature, and the vapor is condensed in a heat exchanger. It handles very high heat fluxes with low flow, and a leak of dielectric fluid does not short-circuit electronics, but the fluids and systems are more complex.

Cold plates typically capture 70 to 85% of a server's heat (the processors, accelerators and sometimes memory and voltage regulators). The remainder, from power supplies, network cards, storage and other components, still goes to air, so a liquid-cooled hall still needs air cooling, at a much lower density. Some designs extend cold plates to nearly every component to capture 95% or more.

Coolant distribution units and the secondary loop

IT equipment is not connected directly to the building's facility water. A coolant distribution unit (CDU) separates the two:

• The facility water system (FWS, or primary loop) connects the CDU to the plant: chillers, dry coolers or towers.
• The technology cooling system (TCS, or secondary loop) runs from the CDU to the racks and cold plates, with tightly controlled temperature, pressure, flow, filtration and fluid chemistry.

The CDU contains a liquid-to-liquid heat exchanger, pumps (usually redundant), filters (commonly 25 to 50 µm to protect the fine channels of cold plates), expansion and make-up provisions, and controls that hold the TCS supply temperature and pressure. In-rack CDUs serve one rack; in-row or floor-mounted CDUs serve a row or a group of racks, typically 500 kW to several megawatts each. Where no facility water is available, liquid-to-air CDUs (sometimes called sidecars) reject the heat from the TCS into the room air through a coil and fans, which lets a few liquid-cooled racks be added to an air-cooled hall, at the cost of a large local air load.

The temperature chain

Chip vendors specify the maximum coolant temperature at the cold plate inlet that keeps their silicon within limits at full power. Working backward from that limit through each heat exchanger's approach temperature determines the facility water temperature, and from it whether chillers are needed. ASHRAE TC 9.9 classifies liquid-cooled environments by the maximum facility water supply temperature:

Immersion cooling

In immersion cooling, entire servers are submerged in a dielectric (electrically non-conducting) liquid in a tank. Fans are removed, and the liquid contacts every component, capturing essentially all the heat.

• Single-phase immersion uses hydrocarbon or synthetic oils that stay liquid. Pumps circulate the fluid through a heat exchanger that transfers the heat to facility water. The fluids are relatively inexpensive and stable, but viscous, and servers emerge coated in oil for maintenance.
• Two-phase immersion uses engineered fluids, historically fluorocarbons, that boil at around 50 °C on hot components. The vapor rises, condenses on a water-cooled coil at the top of the tank and drips back, giving very high heat transfer with no pumps in the tank. The fluids are expensive, must be kept from escaping (vapor losses), and many are per- and polyfluoroalkyl substances (PFAS), which face increasing regulation; a major manufacturer of such fluids announced it would exit PFAS manufacturing by the end of 2025, pushing the industry toward alternative fluids.

Immersion changes the facility: tanks are heavy (a tank holding about a cubic meter of fluid can weigh well over a metric ton with its servers) and lie horizontally, so floor loading and layout differ from rack rows; servers must be lifted out with hoists for service; and server vendors must support and warrant their equipment for immersion. It is used in some cryptocurrency mining, edge and high-performance computing installations, while direct-to-chip has become the mainstream choice for AI clusters because it fits existing rack and server designs.

In the US, the EPA has issued reporting requirements for PFAS under the Toxic Substances Control Act, and several states, including Maine and Minnesota, have enacted laws restricting PFAS in products with phased deadlines; operators considering fluorinated immersion or two-phase fluids should check the current federal and state requirements.

Practical issues

Bringing liquid onto the data hall floor raises concerns that air cooling never had:

• Leaks: dripless quick-disconnect couplings, leak detection cable or sensors under racks and CDUs, drip trays, and controls that isolate a leaking branch. Negative-pressure TCS designs, which hold the loop below atmospheric pressure so that a breach draws air in rather than spraying coolant, are also offered.
• Fluid quality: cold plate channels are fine and easily fouled, so the TCS is filled with treated, filtered coolant and monitored for conductivity, pH, inhibitor levels and biological growth. Mixing incompatible metals (such as copper and aluminum) in one loop causes galvanic corrosion.
• Materials compatibility: hoses, seals and plastics must suit the coolant; dielectric fluids in immersion can swell or leach some cable jackets and labels.
• Redundancy and resilience: a liquid-cooled accelerator can overheat in seconds without flow, so CDU pumps are redundant and often on UPS power, and the TCS has little thermal mass to ride through interruptions.
• Commissioning and operations: flushing new loops, venting air, balancing flow between racks, and training staff in a discipline closer to plumbing than to traditional IT work.

Heat reuse

Liquid cooling returns heat at useful temperatures, 40 to 60 °C or more, compared with the 30 to 40 °C of air-cooled return air. That is warm enough for low-temperature district heating networks, greenhouses, aquaculture and some industrial processes directly, and a heat pump can raise it further for conventional heating networks very efficiently because the lift is small. Data centers in Northern Europe have heated thousands of homes this way. The obstacles are usually not technical: a heat customer must be nearby, the demand must match the supply across seasons (heating demand falls in summer when cooling demand rises), and the commercial arrangements must outlast both parties' plans.

Lesson 10 scales liquid cooling to whole AI halls, with CDU sizing, the temperature chain, transient loads and commissioning of liquid systems.