Heat in the data center

Jamal Pierce has been at Stonebridge Data for four months. He has learned the house rules, survived his first safety course and installed his first cabinets. On the Monday his cooling term begins, Roz Kerr meets him at the door of Hall A with two clipboards and a handheld thermometer. "For the next twelve weeks," she says, "you are going to learn where every watt in this building goes after the servers are finished with it. Spoiler: it all ends up as heat, and our job is to carry it outside before it cooks anything." This course follows Jamal through that term. It stays at the level of a technician who operates, inspects, maintains and troubleshoots cooling equipment; the design of cooling plants is covered in the Data Center Engineering school, in DCE-330.

Why every watt becomes heat

A server takes in electrical energy and does useful work with it: it switches billions of transistors, spins fans and drives network links. None of that work stores energy in a lasting form. When a processor adds two numbers, the energy used to flip its transistors ends up as a tiny rise in the temperature of the silicon. The fans push air, and the air's motion turns into heat by friction. The light from a network laser is absorbed at the far end of the fiber and warms the receiver. Over any period longer than a few seconds, the heat a rack gives off is equal to the electrical power it draws.

That simple rule is the starting point for all cooling work. If a rack's power distribution units show 12 kW, the rack is putting 12 kW of heat into the hall. If the whole of Hall D draws 7.9 MW, the cooling systems must remove about 7.9 MW of heat, every second of every day, plus the heat from lights, people, power losses in the UPS and transformers, and the heat that leaks in through the walls and roof on a summer afternoon.

Technicians sometimes hear a server described as "efficient" and assume an efficient server makes less heat. It does make less heat per unit of work, because it does more computing per watt. But a rack drawing 12 kW still makes 12 kW of heat whatever is running on it. The only way to reduce the heat is to draw less power.

Where the heat comes from in a building

The IT equipment is the largest source of heat, but not the only one. Roz has Jamal list the others for Hall A:

• IT load: servers, storage arrays and network switches. In Hall A this is about 1.2 MW, spread over roughly 200 racks.
• Power losses: UPS systems, transformers and power distribution units are not perfectly efficient. A UPS that is 96 percent efficient turns 4 percent of the power passing through it into heat in the electrical room.
• Lighting: LED fixtures in the hall, a few kilowatts in total.
• People: each person working gives off roughly 100 W while moving about. A crew of ten adds about a kilowatt, which only matters in small rooms.
• The building envelope: heat conducted through the roof and walls and brought in with outside air. This changes with the weather and the season.
• The cooling equipment itself: fans in cooling units turn their electrical power into heat inside the air stream, and pumps warm the water they move.

For most halls the IT load is more than 90 percent of the total. The cooling system is sized for the full IT load plus an allowance for the rest. When Jamal looks at a cooling unit's display, the heat it reports removing is mostly the heat of the servers in front of it.

Watts, kilowatts and heat rate

Heat is energy, and the rate at which heat is produced or removed is power. Both use the same units. A watt is one joule per second; a kilowatt is a thousand watts; a megawatt is a million. When a cooling unit is rated at 100 kW, it can remove 100 kilojoules of heat every second under its rated conditions.

Energy over time is measured in kilowatt-hours. A 10 kW rack running for 24 hours uses 240 kWh of electricity and produces 240 kWh of heat. The electricity bill counts kilowatt-hours, and so does the heat the cooling plant must carry away.

North American cooling equipment is often rated in other units as well, and technicians meet them every day on nameplates, drawings and monitoring screens:

• British thermal units per hour, BTU/h. One kilowatt is about 3,412 BTU/h.
• Tons of refrigeration. One ton is 12,000 BTU/h, about 3.517 kW. The name comes from the heat needed to melt a short ton of ice in a day.
• Kilowatts of cooling, often written kWr or kWth to show that the figure is heat, not electrical power.

A common source of confusion is that a chiller is rated both by the heat it removes, in tons or kilowatts of cooling, and by the electrical power it draws, in kilowatts. A 500-ton chiller removes about 1,760 kW of heat while drawing perhaps 300 kW of electricity. The ratio of the two is its efficiency, which is covered in Lesson 5.

# Heat from rack loads in kilowatts, BTU/h and tons of refrigeration
racks = {"Hall A cabinet": 6, "Hall B rack": 12, "Edge rack": 8, "Hall D AI rack": 132}
print(f"{'Rack':16s} {'kW':>6s} {'BTU/h':>10s} {'tons':>7s}")
for name, kw in racks.items():
    btu_h = kw * 3412
    tons = kw / 3.517
    print(f"{name:16s} {kw:6.0f} {btu_h:10,.0f} {tons:7.1f}")
hall_d_mw = 60 * 132 / 1000
print(f"Hall D, 60 racks: {hall_d_mw:.2f} MW of heat = {hall_d_mw * 1000 / 3.517:,.0f} tons")

Rack                 kW      BTU/h    tons
Hall A cabinet        6     20,472     1.7
Hall B rack          12     40,944     3.4
Edge rack             8     27,296     2.3
Hall D AI rack      132    450,384    37.5
Hall D, 60 racks: 7.92 MW of heat = 2,252 tons

The table shows how quickly the numbers grow. A single Hall D rack makes as much heat as about 37 tons of air conditioning, more than a typical office floor. Jamal writes the conversion factors on the back of his badge holder: 3,412 BTU/h per kilowatt, and 3.517 kW per ton.

How heat moves

Heat always moves from something hotter to something cooler. It travels in three ways, and all three matter in a data hall:

• Conduction: heat flows through a solid, such as from a processor through its thermal paste into a heat sink or a cold plate.
• Convection: a moving fluid, air or liquid, picks up heat from a hot surface and carries it away. Server fans and cooling unit fans drive forced convection; warm air rising on its own is natural convection.
• Radiation: hot surfaces give off infrared radiation. It matters little inside servers but explains why a sun-baked roof warms the space below it.

In an air-cooled server, heat is conducted from each chip into a finned heat sink, then carried away by air that the server's fans push across the fins. In a liquid-cooled server, heat is conducted into a cold plate and carried away by coolant flowing through channels inside it. In both cases the coolant, air or liquid, warms up as it absorbs the heat, and must be cooled again before it returns.

Sensible and latent heat

Heat that changes the temperature of air or water is called sensible heat, because a thermometer senses it. Heat that changes the moisture content of air, by evaporating or condensing water, is called latent heat. When a cooling coil is colder than the dew point of the air passing over it, water condenses on the coil and the coil removes latent heat as well as sensible heat.

Servers make almost only sensible heat; they do not give off moisture. A data center cooling system therefore wants to remove sensible heat and avoid condensing water out of the air, which would waste energy and dry the room. Data center cooling units are designed with a high sensible heat ratio, close to 1.0, meaning nearly all of their cooling goes into lowering temperature. Lesson 2 explains humidity and dew point in detail.

The coolant must carry the heat

A cooling system works by moving a coolant past the heat source, letting it warm up, and carrying it to a place where it can be cooled again. The amount of heat a stream of coolant carries depends on three things: how much of it flows, how much heat each kilogram absorbs for each degree of warming, and how many degrees it warms. Written as an equation:

Here Q is the heat rate in watts, ṁ is the mass flow in kilograms per second, cp is the specific heat in joules per kilogram per kelvin, and ΔT is the temperature rise. For air, cp is about 1,006 J/(kg·K) and a cubic meter weighs about 1.2 kg. For water, cp is about 4,186 J/(kg·K) and a cubic meter weighs about 1,000 kg.

That difference is the whole story of modern data center cooling. A cubic meter of water carries about 3,500 times as much heat as a cubic meter of air for the same temperature rise. Air is free, safe around electronics and easy to move with fans, so it has cooled data centers for decades. But as racks have grown from 5 kW to more than 100 kW, the volume of air needed has become impractical, and liquid has taken over for the densest equipment.

# Volume flow of air and of water needed to carry the same heat load
q_kw = 12.0                 # heat from one Hall B rack
dt_air = 12   # about 22 degrees F           # air temperature rise across the servers, K
dt_water = 6    # about 11 degrees F           # water temperature rise, K
rho_air, cp_air = 1.2, 1006        # kg/m3, J/(kg K)
rho_w, cp_w = 1000, 4186
air_m3s = q_kw * 1000 / (rho_air * cp_air * dt_air)
w_m3s = q_kw * 1000 / (rho_w * cp_w * dt_water)
print(f"Air:   {air_m3s:.3f} m3/s = {f"{air_m3s * 2118.9:,.0f} cfm"}")
print(f"Water: {w_m3s * 1000 * 60:.1f} L/min = {f"{w_m3s * 15850:.1f} gpm"}")
print(f"Air needs {air_m3s / w_m3s:,.0f} times the volume flow of water")

Air:   0.828 m3/s = 1,755 cfm
Water: 28.7 L/min = 7.6 gpm
Air needs 1,734 times the volume flow of water

The output shows the same 12 kW carried by air and by water. Even though the water is allowed to warm only half as much, air needs about 1,700 times the volume flow of water for the same job. That is why an air-cooled hall is full of large fans and why a liquid-cooled rack can be cooled by a hose the width of a garden hose.

What happens when cooling stops

Servers warm up very quickly when their cooling stops. The air in a hall has almost no capacity to store heat; the cooling system is what keeps the temperature steady. If every cooling unit in Hall A stopped while the servers kept running, the air temperature would rise by several degrees a minute. The servers would first throttle their processors to protect themselves, then shut down when their internal sensors reached their limits. Hardware can be damaged and, worse, customers lose service.

Hall D is even more sensitive. Its cold plates carry heat from processors drawing over a kilowatt each. If coolant flow stops, the chips reach their limits in seconds, not minutes. That is why the CDUs in Hall D run with spare pumps, why the coolant loops have buffer tanks, and why a CDU alarm is treated as urgent. Lesson 9 covers CDUs in detail, and Lesson 12 follows Jamal through a CDU alarm.

Designers protect against this with redundancy: more cooling units than the load needs, two pumps where one would do, chilled water stored in tanks, and cooling units and pumps on UPS power so they keep running through a utility failure until the generators start. Technicians protect against it by keeping that redundancy real: not leaving a spare unit switched off after maintenance, not leaving a valve closed, and reporting any alarm that reduces the margin.

# How fast the air in a hall would warm if all cooling stopped (air only, no thermal mass)
hall_area = 25800          # ft2
height = 15                # ft, floor to ceiling
it_kw = 1200                # heat still being produced, kW
volume_m3 = hall_area * height * 0.0283168
mass = volume_m3 * 1.2       # kg of air
rise_k_per_min = it_kw * 1000 / (mass * 1006) * 60
print(f"Air volume {hall_area * height:,.0f} ft3, air mass {mass:,.0f} kg")
print(f"Rise if nothing absorbed the heat: {rise_k_per_min * 1.8:.1f} F per minute")
for minutes in (1, 2, 5):
    print(f"  after {minutes} min: +{rise_k_per_min * 1.8 * minutes:.0f} F")

Air volume 387,000 ft3, air mass 13,150 kg
Rise if nothing absorbed the heat: 9.8 F per minute
  after 1 min: +10 F
  after 2 min: +20 F
  after 5 min: +49 F

Roz reminds Jamal that numbers like these are rough. Real halls have walls, floors, cabinets and equipment that soak up some heat, so the first minutes are slower than the simple calculation suggests. But the order of magnitude is right, and it explains why the operations team responds to a cooling failure as quickly as to a power failure.

Heat as a resource

Heat is not only a problem. The warm water leaving a liquid-cooled hall can heat buildings, greenhouses or swimming pools. Data centers in Europe and a growing number in North America send their waste heat to district heating systems or nearby buildings, and Stonebridge is studying whether the warm water from Hall D could heat offices on the campus. Lesson 10 returns to heat reuse.

For now, the lesson for a technician is simpler. Every kilowatt of IT load needs a path for its heat. When a customer adds servers, the cooling load rises by exactly the added power. When a cooling unit is down for maintenance, the remaining units must carry its share. Thinking about heat as a flow that must always have somewhere to go is the habit Roz wants Jamal to build.

A first walk with a thermometer

Roz and Jamal walk Hall A with the handheld thermometer. In the cold aisle the air at the floor tiles reads about 66 °F. At the top of a rack's front door it reads 73 °F. Behind the racks, in the hot aisle, the exhaust reads 91 °F at chest height and nearly 100 °F near the ceiling. At the return grilles of the CRAH units along the wall it reads about 86 °F.

Each reading tells part of the story. The cold air warms a little on its way up the front of the rack because some hot air mixes into it. The servers heat the air by about 22 °F on its way through. The hot air rises and collects near the ceiling, then flows back to the cooling units, cooling slightly as it mixes with cold air that bypassed the racks. Jamal records the readings on the clipboard, and Roz tells him that by the end of the term he will be able to read a hall's health from a dozen numbers like these.