Immersion cooling and heat reuse

Stonebridge has no immersion tanks in production yet, but the edge site at Saint-Hyacinthe (Winchester) has a pilot: two tanks holding sixteen servers each, submerged in a clear liquid that looks like light mineral oil. Anjali sends Jamal and Roz to spend a day with the pilot's vendor engineer, because the company is deciding whether to use immersion at future edge sites, and the technicians who would maintain it have opinions worth hearing. The same week, the company's sustainability lead visits YUL-2 to talk about sending Hall D's warm water to a neighbour. This lesson covers both: immersion cooling, the last of the main cooling methods, and the reuse of data centre heat. It is an overview; the design of both is covered in the DCE school.

What immersion cooling is

In immersion cooling, servers are submerged in a liquid that does not conduct electricity, called a dielectric fluid. Every component sits in the liquid, which absorbs heat directly from every surface. No server fans are needed. The warm liquid is cooled in a heat exchanger and returned to the tank. Two kinds of system are used:

• Single-phase immersion: the fluid stays liquid throughout. It is usually a synthetic hydrocarbon oil or a similar engineered fluid. Pumps circulate it from the tank through a heat exchanger, where it gives its heat to facility water, and back. The edge pilot is single-phase.
• Two-phase immersion: the fluid boils at a low temperature, around 50 °C (122 °F), on the hot components. The vapour rises, condenses on a cooled coil in the top of the tank and drips back. Boiling carries heat very effectively, but the fluids used have raised environmental and supply concerns, and the tanks must be sealed closely to keep vapour from escaping.

Immersion tanks usually lie on their backs, like chest freezers. Servers stand vertically in slots and are lifted out from the top for service. Tanks are heavy when full: a tank holding a few hundred litres of fluid plus servers can weigh more than 1,000 kg (2,200 lb), so floor loading is planned carefully.

Why operators consider immersion

Immersion has real advantages, which is why the pilot exists:

• High density: a tank can remove all the heat of very dense equipment, and no part of the server depends on air.
• No fans: server fans use several percent of a server's power; removing them saves energy and noise.
• Warm water: the fluid and the facility water can run warm, allowing free cooling almost everywhere and making the heat useful.
• Protection: components sit in a stable, clean liquid, away from dust, humidity and corrosive gases, which suits harsh edge locations.

It also has drawbacks a technician feels directly:

• Service is messier: a server lifted out drips fluid and must drain over a tray before work; components and cables come out slick.
• Compatibility: not every server, cable, label or thermal pad is designed to live in the fluid; some materials swell or leach. Equipment must be approved for immersion.
• Fluid management: fluid is lost on every service; it must be topped up, tested and kept clean. Spills must be cleaned at once because the floor becomes slippery.
• Weight and space: tanks are heavy and take floor area differently from racks.
• Warranty and support: server makers must support immersion for the equipment to stay under warranty.

Servicing an immersion tank

The vendor engineer shows Jamal and Roz the procedure for replacing a failed drive in the pilot:

• Check the ticket and the server's position; confirm with monitoring which server it is.
• Wear the specified gloves and eye protection; place an absorbent mat and a drip tray beside the tank.
• Open the lid; lift the server by its handles, or with the tank's hoist for heavy units, and hold it over the tank to drain for the time the procedure sets.
• Move it to the drip tray or service stand; let it drain further; replace the drive.
• Wipe connectors as the procedure directs; return the server, lower it into its slot, and reconnect.
• Close the lid; check the fluid level and top up from approved stock if needed; record the fluid used.
• Clean any drips from the floor and the tank edge at once.

Roz's verdict is that it is not hard, just different, and that the slip hazard is the part people underestimate. Jamal notes that the job took about twice as long as the same drive swap in a rack.

Monitoring immersion systems

The tank's controller reports fluid temperatures at the inlet and outlet, pump status and speed, fluid level, facility water temperatures and flow, and leak detection around the tank. Fluid quality is checked by sampling on a schedule: moisture content, acidity and particles. Water entering a dielectric fluid reduces its insulating ability and can harm components, so a rising moisture level is an alarm item even if temperatures are fine.

# Immersion tank health check: heat carried by the fluid and fluid-quality limits (illustrative)
fluid_rho, fluid_cp = 800, 2100       # kg/m3, J/(kg K), typical synthetic hydrocarbon
flow_lpm = 120
t_in, t_out = 35.0, 41.0
q_kw = flow_lpm / 60000 * fluid_rho * fluid_cp * (t_out - t_in) / 1000
print(f"Heat carried by the fluid: {q_kw:.1f} kW")
sample = {"moisture ppm": 62, "acid number": 0.04, "particles per mL": 1800}
limits = {"moisture ppm": 50, "acid number": 0.10, "particles per mL": 5000}
for k, v in sample.items():
    status = "OK" if v <= limits[k] else "ABOVE LIMIT - report"
    print(f"  {k:17s} {v:8} (limit {limits[k]}) {status}")

Heat carried by the fluid: 20.2 kW
  moisture ppm            62 (limit 50) ABOVE LIMIT - report
  acid number           0.04 (limit 0.1) OK
  particles per mL      1800 (limit 5000) OK

Comparing the cooling methods

By the end of the pilot visit Jamal can place every method he has met in this course on one table:

Most large sites now combine methods. Hall A uses perimeter air with a few rear doors and in-row units. Halls B and C use fan walls and hot-aisle containment. Hall D uses direct-to-chip with fan walls for the remaining air-side heat. The edge site is testing immersion.

Reusing data centre heat

All the heat a data centre removes is, in principle, available for use. The problem is its temperature. Air leaving a hall at 35 °C (95 °F) is too cool to heat most buildings without help. Water leaving a liquid-cooled hall at 40 °C (104 °F) or more is much more useful: it can feed a greenhouse, a swimming pool or a low-temperature district heating network directly, or a heat pump can raise it to the temperature a district system needs.

Some data centres in Quebec, as in Europe, already send heat to neighbouring buildings or greenhouses, and the province's low-carbon electricity makes heat pumps an attractive way to raise the temperature. Stonebridge's sustainability lead is studying a pipe from Hall D's warm loop to a greenhouse grower a short distance from YUL-2. In winter, when the greenhouse needs heat most, Hall D's dry coolers would otherwise throw that heat into the cold air.

Heat reuse is measured with a metric called energy reuse effectiveness, ERE, or with the share of energy reused, the energy reuse factor, ERF. A site that sends 20 percent of its heat to useful purposes has an ERF of 0.2. For a technician, heat reuse adds one more set of pumps, valves and heat exchangers to inspect, and one more customer, the heat user, who cares about the temperatures and reliability of the supply.

# Heat Hall D could send to a nearby greenhouse in winter
hall_d_kw = 60 * 132
to_liquid = 0.84
return_c = 40                   # warm return water from the CDUs, C
greenhouse_need_kw = 4500        # greenhouse heating demand on a cold day
available = hall_d_kw * to_liquid
used = min(available, greenhouse_need_kw)
print(f"Heat in Hall D's warm loop: {available:,.0f} kW at about {return_c} C")
print(f"Greenhouse demand on a cold day: {greenhouse_need_kw:,} kW -> supplied {used:,.0f} kW")
print(f"Share of Hall D's heat reused that day: {used / (hall_d_kw * 1.0):.0%}")
gas_m3 = used * 24 / 10.4 / 0.9   # m3 of natural gas a 90 % boiler would burn for the same heat
print(f"Natural gas avoided: about {gas_m3:,.0f} m3 per day")

Heat in Hall D's warm loop: 6,653 kW at about 40 C
Greenhouse demand on a cold day: 4,500 kW -> supplied 4,500 kW
Share of Hall D's heat reused that day: 57%
Natural gas avoided: about 11,538 m3 per day

The calculation shows why heat reuse works best with liquid cooling. Hall D's warm return water carries most of the hall's heat at a temperature a greenhouse can use directly. On a cold day it could cover the grower's whole heating demand, and the gas the grower would otherwise burn is a large daily quantity. Whether the project goes ahead depends on cost, contracts and how reliable the supply must be, decisions for the company rather than the technicians. But the technicians would run it.

What changes for technicians

Each new cooling method changes the daily work. With immersion and heat reuse, Jamal adds to his list:

• New hazards: slippery fluids, heavy lids and hoists, hot heat-recovery piping.
• New materials to control: dielectric fluids, glycol coolants, absorbent mats, fluid waste.
• New measurements: fluid moisture and acidity, heat meters on the reuse loop.
• New people: the vendor's engineer, the heat user's contact, the sustainability team.

The principles stay the same: heat must always have somewhere to go, every connection is a possible leak, alarms need a visit, and work follows a written procedure.

Fluids, waste and the environment

Every cooling fluid a technician meets has its own handling rules, set out in its safety data sheet. Glycol coolants are low in toxicity but must not go down floor drains; used coolant is collected in labelled containers for the approved waste contractor. Dielectric immersion fluids are handled the same way, and rags and absorbent mats soaked with them are collected separately. Some two-phase immersion fluids have come under scrutiny because of the chemical families they belong to, and several makers have withdrawn products, which is one reason most operators, Stonebridge included, are piloting single-phase systems.

Spill kits stand near every CDU gallery and next to the immersion pilot. Each kit has absorbent socks to stop a spill spreading, pads, bags, gloves and goggles. Technicians are trained to contain a small spill first, then clean it, then record it, and to call the facilities team for anything larger.

Choosing a cooling method is not a technician's call

Technicians often have strong views about which cooling method works best, and good operators listen to them, because they see the daily cost of each. But the choice depends on the equipment customers bring, the building, the climate, the contracts and the money. Jamal's report from the pilot visit lists what he and Roz found easy, what was slow, and what they would change in the service procedure. Anjali forwards it to the design team unchanged, which Roz tells him is the best compliment a technician's report can receive.