The power chain from grid to chip

Electricity reaches a processor after a long journey. It leaves a power plant at a few thousand volts, is stepped up to hundreds of thousands of volts for transmission, stepped down several times on its way into the building, converted from AC to DC and back again inside a UPS, stepped down again for the racks, converted to DC in each server's power supply, and finally reduced to less than one volt at the chip. Every step costs a little energy, and every step is a place where something can fail.

This chapter follows that chain from the transmission grid to the processor: what each piece of equipment does, the voltages used in each part, how much current flows, where the backup power comes in, and where the losses occur. DCE-310 and DCE-340 cover the design of each part in detail.

From the transmission grid

Large data centres connect directly to the transmission system. In Alberta, transmission lines operate mainly at 138 kV and 240 kV, with a 500 kV backbone; other provinces use similar levels (Ontario, for example, uses 115 kV, 230 kV and 500 kV). A campus of 100 MW or more usually needs its own substation, built by the transmission owner or by the developer to the utility's standards, where transformers step the voltage down to a medium voltage, commonly 25 kV in Alberta.

Why such high voltages? The power carried by a line is proportional to voltage times current, but the heat lost in the conductors is proportional to the current squared times the resistance (\(P_\text{loss} = I^2R\)). Raising the voltage tenfold cuts the current tenfold for the same power, and the losses a hundredfold. High voltage is how power travels efficiently; low voltage is how it is used safely.

Reliability starts at the connection. Many data centres ask for two utility feeds from different substations or different transmission lines, so that a single line outage does not reach the building. Whether that is possible, how long it takes and who pays for it are among the first questions in any project (Lesson 5).

Three-phase power

Almost all power in a data centre is three-phase alternating current: three conductors carry sinusoidal voltages 120° apart. Three-phase delivers power smoothly and uses less conductor material than single-phase for the same power. For a balanced three-phase load,

where \(V_L\) is the line-to-line voltage, \(I\) the line current and pf the power factor (close to 1 for modern IT equipment, whose power supplies correct it). Without the power factor, \(S = \sqrt{3} V_L I\) is the apparent power in volt-amperes (VA or kVA), the quantity transformers and UPS systems are rated in.

In a "wye" (star) system, each phase can also be connected to the neutral, giving a lower voltage \(V_L/\sqrt{3}\). That is where the familiar pairs of numbers come from:

• 600Y/347 V: the standard Canadian low-voltage system for commercial and industrial buildings (600 V line to line, 347 V line to neutral, used for lighting).
• 208Y/120 V: used for receptacles and much IT equipment in North America.
• 415Y/240 V: increasingly used for racks, because 240 V line to neutral suits IT power supplies directly while 415 V three-phase carries twice the power of 208 V on the same current.

Inside the building: switchgear and transformers

Medium-voltage power (for example 25 kV) enters the building or the yard through medium-voltage switchgear: an assembly of circuit breakers, protective relays and metering that can isolate faults and switch circuits. It feeds unit substations, each a transformer of typically 2 to 3.5 MVA that steps the voltage down to 600 V, bolted to low-voltage switchgear that distributes power to UPS systems and mechanical equipment.

Data centre electrical systems are usually divided into identical blocks (say, 2 or 3 MW each), each with its own transformer, generator and UPS. Blocks make it easy to build in phases, to repeat a proven design, and to arrange redundancy (Lesson 4).

Every point in the system must be protected: a short circuit can release enormous energy. Circuit breakers and fuses must interrupt the available fault current, and they must be coordinated so that the device nearest a fault trips first, leaving the rest of the system running. An arc flash, the explosive release of energy when current jumps through air between conductors, can kill workers; arc flash studies label every panel with its hazard so that workers wear the right protection.

Backup power: UPS and generators

When the utility fails, two systems take over.

Uninterruptible power supply (UPS)

A UPS sits between the switchgear and the IT load. The most common type in data centres, the double-conversion UPS, continuously converts incoming AC to DC (charging the batteries) and back to clean AC for the load. If the input fails, the batteries keep the output going with no interruption at all. The UPS also filters out voltage sags, spikes and frequency changes from the grid.

The batteries only need to last until the generators start and take the load, typically under a minute, but designers usually provide 5 to 10 minutes at full load for margin. Valve-regulated lead-acid batteries were the norm; lithium-ion batteries now dominate new installations because they are smaller, lighter and last longer, though they need careful fire protection (DCE-440).

Generators

Standby generators, usually diesel engines driving alternators of 2 to 3 MW each, carry the load through long outages. When the utility fails, the generators start, reach speed and voltage in about 10 seconds, and automatic transfer switches (or paralleling switchgear) move the load from the utility to the generators. When the utility returns and has been stable for some minutes, the load is transferred back.

Fuel is stored on site, typically for 24 to 72 hours at full load, with contracts for refuelling during long emergencies. Some operators now run generators on renewable diesel; others are installing natural gas engines, turbines or fuel cells that can run continuously, not just in emergencies (DCE-420).

Generators are tested regularly under load, because a generator that has not been exercised is the most likely piece of equipment to fail on the day it is needed.

To the rack

From the UPS, power flows to power distribution units (PDUs) in or near the white space. A PDU is a cabinet containing a transformer (where needed), a panelboard of breakers and monitoring. Each PDU feeds branch circuits to the racks, often through overhead busway: a prefabricated enclosed bar with tap-off boxes that can be plugged in wherever a rack needs power. Inside each rack, rack PDUs (power strips) distribute power to the individual pieces of equipment and usually meter each outlet.

Most IT equipment has two power supplies, fed from two different rack PDUs on two different upstream paths (often called A and B). If one path fails or is shut down for maintenance, the other carries the whole load.

Inside the server

The server's power supply unit (PSU) converts AC to low-voltage DC, traditionally 12 V and increasingly 48 V in high-power systems. On the motherboard, voltage regulator modules (VRMs) convert that to the roughly 0.7 to 1 V a modern processor core runs on. At these voltages the currents are enormous: a GPU drawing 1000 W at 0.8 V draws about 1250 A, which is why the regulators sit right next to the chip.

Each conversion loses energy. That has led the industry toward fewer conversions. One direction is 48 V distribution inside the rack, from a shared power shelf. Another is high-voltage DC: NVIDIA and partners have published designs for 800 V DC power to future AI racks approaching 1 MW, removing several AC-to-DC conversions and much of the copper that low-voltage distribution would need.

Where the energy goes

Because each stage is efficient but not perfect, the losses multiply.

Note that in the PUE of Lesson 1, losses before the IT equipment (transformers, UPS, PDUs) count as facility overhead, while losses in server power supplies count as IT energy, because they are measured inside the IT equipment.

Codes and safety

Every part of the installation must meet the Canadian Electrical Code, Part I (CSA C22.1) as adopted and amended by the province, and equipment must be certified to Canadian standards (look for the CSA, cUL or other accredited certification mark). Medium-voltage and high-voltage work, the utility interface, generator installations and battery rooms each bring additional rules. Workers follow CSA Z462, Workplace Electrical Safety, for arc flash and shock protection. Electrical permits and inspections are issued by the provincial or municipal electrical inspection authority.