Utility service and substations

Power is the first constraint on a modern data centre. A large campus can draw 100 to 1000 MW, comparable with a mid-sized city, and it draws that power around the clock with very little variation. Land, fibre and water can usually be found; a grid connection of hundreds of megawatts often cannot, at least not quickly. Power availability now decides where data centres are built, how fast they are built and how big they can grow.

DCE-200 traced the power chain from the transmission grid to the processor. This course designs it. It covers the electrical distribution of a data centre from the utility connection down to the rack, the calculations that size and protect it, and the codes that govern it. This first chapter starts at the top: how the grid delivers power, how a data centre connects to it, what is inside the substation that receives it, and how redundancy is built in from the very first breaker.

The grid in brief

Electricity is generated at power stations, typically at 13.8 to 25 kV, and immediately stepped up to transmission voltage. High voltage keeps the current, and therefore the \(I^2R\) losses, low over long distances. Transmission lines carry bulk power to substations near load centres, where transformers step it down to distribution voltages for delivery to customers.

In Canada, transmission voltages include 115, 138, 230 and 500 kV, and Hydro-Québec operates a 735 kV network to carry power from northern hydroelectric stations. Distribution is commonly at 25 kV or 12.5 kV in the west and Québec, and 27.6 kV or 13.8 kV in Ontario. Electricity is a provincial responsibility: each province has its own utilities and regulator, and in Ontario and Alberta an independent system operator (the IESO and AESO) runs the wholesale market and the grid.

Why does a data centre care about transmission voltage? Because the current for a given power falls in proportion to voltage, and the current sets the size of every conductor, breaker and transformer.

Service options

How a data centre connects depends mostly on its size.

• Low-voltage service: small server rooms and edge sites take power from a utility pad-mounted transformer at the same voltage as any commercial building.
• Medium-voltage (distribution) service: facilities up to roughly 10 to 30 MW are typically fed by one or more utility distribution feeders. The owner receives power at medium voltage and owns the switchgear and transformers that step it down.
• Transmission-level service: large facilities connect directly to the transmission network. Either the utility builds a dedicated substation next to the campus, or the owner builds and operates its own high-voltage substation. The point where ownership changes, the point of interconnection, is defined in the interconnection agreement.

Whatever the level, the utility will study the request before agreeing to it: whether the transmission lines and existing substations can carry the extra load, what upgrades are needed and who pays. For large loads these studies and upgrades can take years, which is why DCE-420 treats power procurement and interconnection as a project in its own right. A frequent strategy is to start a campus on a smaller interim supply while the full transmission connection is built.

Dual utility feeds

A single utility line is a single point of failure. Data centre owners therefore ask for two independent supplies, ideally from two different utility substations over separate routes, so that a fault on one line or one substation leaves the other in service. Utilities cannot always provide true independence: two feeders from the same substation still share that substation. The owner must understand exactly how independent the feeds are, the same shared-risk reasoning used for fibre routes in DCE-210. Generators and UPS systems (DCE-340) remain essential whatever the utility arrangement, because even two utility feeds can fail together during a wide-area disturbance.

Inside a substation

A substation switches, transforms, measures and protects. The main components of a customer-owned high-voltage substation are:

• Line entrances with disconnect switches, which provide visible isolation of each incoming line for maintenance.
• Circuit breakers, which interrupt current, including fault currents of tens of kiloamperes. High-voltage breakers use sulphur hexafluoride (SF₆) gas or vacuum to extinguish the arc; because SF₆ is an extremely potent greenhouse gas, SF₆-free designs are increasingly specified.
• Power transformers, which step transmission voltage down to the site's medium distribution voltage. They are rated in MVA and are usually oil-filled, with cooling classes such as ONAN (oil natural, air natural) and ONAF (oil natural, air forced), which add fans to raise the rating.
• Instrument transformers: current transformers (CTs) and voltage transformers (VTs or PTs) that scale currents and voltages down to safe levels for meters and relays.
• Protective relays, microprocessor devices that watch those signals and trip breakers within cycles when they detect a fault.
• Surge arresters to protect equipment from lightning and switching surges, a ground grid of buried conductors to limit dangerous step and touch voltages during faults, and a control building with batteries, communications and the protection and control panels.

Electrical engineers describe these systems with one-line (single-line) diagrams, which represent each three-phase circuit with a single line and each device with a standard symbol. The one-line diagram is the master document of any electrical design, from which equipment is specified, protection is studied and operators plan switching.

Bus configurations

The arrangement of breakers and buses decides how well the substation survives failures and maintenance. The common configurations, in rising order of cost and reliability, are:

• Single bus: all lines and transformers connect to one bus. Simple and cheap, but a bus fault or bus maintenance takes everything out.
• Main-tie-main (split bus): two buses, each fed by its own source, joined by a normally open tie breaker. If one source fails, the tie closes and the other source carries both buses. This is the workhorse of medium-voltage data centre distribution.
• Ring bus: breakers arranged in a loop, with each line or transformer connected between two breakers. Any breaker can be taken out for maintenance without interrupting a circuit.
• Breaker-and-a-half: two buses with three breakers for every two circuits. Very reliable and flexible, common at large transmission substations.

Sizing and redundancy of substation transformers

Substation transformers are large, expensive and slow to replace: lead times for large power transformers have stretched to two to four years in recent years, and a failed unit cannot be swapped quickly. They are therefore always installed with redundancy, so that the full load can be carried with one transformer out of service.

Transformers can carry more than their nameplate rating for limited periods, depending on cooling and the temperature of the oil and windings, which is one way the "one out of service" case is handled. Standards for transformer loading (IEEE C57.91 and IEC 60076-7) let engineers calculate how long a given overload can be sustained without excessive ageing of the insulation.

Power factor, metering and the bill

A utility measures both energy (kWh) and peak demand (kW, usually the highest 15- or 30-minute average in a month), and large customers often pay separately for each. Utilities also care about reactive power: a load with a low power factor draws more current, and therefore uses more transformer and line capacity, for the same real power. Many tariffs penalize power factors below about 0.9 to 0.95.

Modern servers have power supplies with active power factor correction, so IT load typically runs at a power factor of 0.95 to 0.99. Large motor loads, such as chillers and pumps, and lightly loaded transformers lower it. In AI campuses with fluctuating loads, the power factor and the voltage stability of the connection are studied carefully.

Power quality and grid interaction

Data centre electronics are protected from short disturbances by their UPS systems, so momentary voltage sags, which are common on any grid during lightning and faults, rarely affect the IT load. They do affect the grid, however. When a sag makes many data centres transfer to their UPS batteries and generators at the same moment, a large block of load disappears from the grid in seconds and returns later. In July 2024, a transmission fault in Northern Virginia caused about 1,500 MW of data centre load to drop off the grid at once, an event studied closely by NERC, the North American Electric Reliability Corporation, whose standards cover both Canada and the United States. Grid operators now increasingly ask large loads to ride through disturbances and to coordinate how they disconnect and reconnect. Lesson 8 examines power quality in detail.

The data centre, in turn, can disturb the grid. The power electronics in UPS systems and server supplies draw harmonic currents; very large AI training clusters can swing their power by tens of megawatts in seconds as jobs start, stop and synchronize; and large motor starts cause voltage dips. Interconnection studies examine these effects, and utilities may require harmonic filters, staged ramping, or energy storage to smooth the load seen by the grid.

How this course is organized

The course follows power from the grid to the rack and through the life of the facility:

• Lesson 2 estimates the load that everything else must serve.
• Lessons 3 and 4 cover the medium-voltage system: switchgear and transformers, then cables, duct banks and system grounding.
• Lessons 5 and 6 cover low-voltage distribution: switchboards, busway and PDUs, then conductors, raceways and circuit sizing.
• Lesson 7 covers short circuits, coordination and arc flash, Lesson 8 power quality and ride-through, and Lesson 9 grounding, bonding, lightning and surge protection.
• Lesson 10 covers metering, monitoring and control, Lesson 11 the electrical code and workplace safety rules, and Lesson 12 design deliverables, maintenance and the future of data centre power.