Removing heat

Lesson 1 established that essentially all the power delivered to IT equipment becomes heat. A 10 MW data hall is a 10 MW heater running all year, and if its cooling stopped, the room temperature could rise by several degrees per minute until servers shut themselves down to protect their chips. Removing that heat reliably, and with as little extra energy and water as possible, is the job of the mechanical system. Cooling is also the largest overhead in most facilities, so it decides much of the PUE.

Heat follows a chain just as power does, in the opposite direction: from the chip to a heat sink, from the heat sink to air or liquid, from the room to a cooling plant, and from the plant to the outdoors. This chapter follows that chain, introduces the one equation that sizes every part of it, and explains why the rise of AI is moving data centres from air cooling toward liquid cooling. DCE-330 covers the design in detail.

The equation that sizes cooling

When a fluid (air or water) flows past a heat source and warms up, the heat it carries away is

where \(\dot{Q}\) is the heat rate (W), \(\dot{m}\) the mass flow rate (kg/s), \(c_p\) the specific heat (J/kg·K) and \(\Delta T\) the temperature rise of the fluid. For air, \(c_p \approx 1005\) J/kg·K and density \(\rho \approx 1.2\) kg/m³; for water, \(c_p \approx 4180\) J/kg·K and \(\rho \approx 1000\) kg/m³. Water carries about 3500 times as much heat per unit volume as air for the same temperature rise. That single fact explains the trend to liquid cooling.

How warm can it be?

IT equipment is designed for a range of inlet air conditions. The industry reference is the thermal guideline published by ASHRAE Technical Committee 9.9. Its recommended envelope for air entering servers is 18 °C to 27 °C (64.4 °F to 80.6 °F), with a dew point between −9 °C and 15 °C and relative humidity no higher than 60%. Wider allowable ranges let equipment run outside the recommended envelope for limited periods: for class A1 equipment, 15 °C to 32 °C (59 °F to 89.6 °F), and up to 45 °C for class A4.

Older data centres were kept cold, around 18 to 20 °C, "to be safe". Running at the warmer end of the recommended range is one of the cheapest efficiency improvements: warmer supply air means warmer return air and warmer chilled water, which makes chillers more efficient and lets free cooling run for more hours of the year. The limit is that server fans speed up as inlet temperature rises, so the gain flattens out.

Air cooling in the white space

Hot aisles and cold aisles

Servers draw air in at the front and exhaust it at the back. Racks are therefore arranged in rows facing each other: fronts face a cold aisle where cool supply air arrives, and backs face a hot aisle where warm exhaust collects and returns to the cooling units. Mixing the two streams wastes cooling: if hot exhaust recirculates to server inlets, those servers overheat; if cold supply air bypasses the servers and returns directly, the cooling units move air that did no work.

Containment solves this with physical barriers: doors at the ends of the cold aisle and a roof over it (cold-aisle containment), or a chimney or enclosure over the hot aisle (hot-aisle containment). Blanking panels in empty rack spaces and brush grommets where cables pass through complete the separation. Good containment lets supply temperatures rise and fan speeds fall.

Cooling units

Air is cooled by units placed around the perimeter of the hall, in the rows between racks, or in separate galleries:

• A CRAC (computer room air conditioner) contains its own refrigeration compressor, like a large air conditioner, and rejects heat to an outdoor condenser.
• A CRAH (computer room air handler) contains only fans and a coil fed with chilled water from a central plant. Large data centres mostly use CRAHs or fan walls (banks of fans and coils that form one wall of the hall).

Air reaches the racks either through a raised floor (a plenum under perforated tiles) or, increasingly, by flooding the hall from the side with the racks on a slab floor and cables overhead. Slab floors carry heavy AI racks better and are cheaper to build.

Fans use a large share of cooling energy. Their power follows the fan affinity laws: flow is proportional to speed, and power to speed cubed. Slowing fans to 80% speed cuts their power to \(0.8^3 = 51\%\). Variable-speed fans controlled to match the actual load are standard in efficient designs.

The cooling plant: getting heat outdoors

Heat collected from the hall must be rejected to the outdoor air. There are three main routes, often combined.

• Mechanical refrigeration: chillers use the vapour-compression cycle to pump heat from chilled water (typically supplied at 15 to 20 °C in modern data centres, warmer than in office buildings) to a warmer condenser loop. Their efficiency is expressed as a coefficient of performance (COP): heat moved per unit of electrical energy used.
• Evaporative cooling: cooling towers spray the condenser water through moving air, and evaporation of a small fraction of the water cools the rest. Evaporation is very effective but consumes water.
• Dry coolers: radiators with fans that reject heat to the air without evaporation. They use no water but can only cool the fluid to a few degrees above the outdoor temperature.

An economizer uses outdoor conditions to cool without running compressors, whenever the weather allows: an air-side economizer brings filtered outdoor air into the hall; a water-side economizer cools the chilled water directly through a heat exchanger with the cooling tower or dry cooler loop. Hours of economizer operation are often called free-cooling hours.

Canada's climate is a real advantage here. In Edmonton, Montreal or most of the country, outdoor air is cool enough for economizers to carry most or all of the load for much of the year, and chillers may run only on warm summer days. Cold climates bring their own design problems, though: freezing protection for water systems, very dry winter air, and snow and ice on equipment.

Water

Evaporative cooling saves electricity but uses water, and water is becoming as important a siting constraint as power. Evaporating water absorbs about 2400 kJ/kg (its latent heat at typical tower temperatures).

The industry metric is water usage effectiveness: WUE = annual site water use (litres) ÷ IT energy (kWh). Facilities with dry coolers or closed-loop liquid cooling can approach zero water use, at the cost of more electricity on hot days. The right balance depends on local water scarcity, electricity carbon intensity and climate.

In Alberta, for example, the Bow, Oldman and South Saskatchewan river sub-basins have been closed to new surface water licences since 2006, so new users must buy or transfer existing licences. Projects in water-stressed regions increasingly choose dry or closed-loop cooling.

Liquid cooling

As rack power rises past what air can handle, liquid takes over, in stages:

• Rear-door heat exchangers replace the back door of a rack with a water coil, cooling the exhaust air before it enters the room. They extend air cooling to perhaps 30 to 50 kW per rack.
• Direct-to-chip (cold plate) cooling bolts a metal plate with internal channels onto each processor and GPU. A coolant (water or water–glycol) flows through the plates and carries 70 to 90% of the heat; fans remove the rest from memory, power supplies and other components. A coolant distribution unit (CDU) in the row or hall isolates this clean technology loop from the facility water loop through a heat exchanger.
• Immersion cooling submerges whole servers in a tank of dielectric (non-conducting) fluid. It removes almost all the heat but changes how servers are built and serviced.

Liquid cooling has another benefit: the coolant can return at 40 °C or more, warm enough to reject heat with dry coolers in almost any climate and useful for heating buildings. Its challenges are leaks near electronics, water quality, the cost of piping to every rack, and the need for operators and technicians with new skills.

Heat reuse

A data centre rejects enormous amounts of low-grade heat. Where there is a nearby user, such as a district heating network, greenhouses, aquaculture or industrial process heating, that heat can be sold instead of wasted, usually after a heat pump raises its temperature. Northern European data centres have connected to city heating networks for years; North American projects are beginning to do the same. Heat reuse works best with liquid cooling, because warmer return water needs less heat-pump work.