Design deliverables, maintenance and the future of data centre power · DCE-310 Lesson 12

The previous lessons covered the parts of a data centre power system one at a time. This final lesson looks at the system as a whole across its life: how an electrical design is developed and documented, which engineering studies prove it, how it is maintained once it is running, and how it is changing as AI racks approach a megawatt each and campuses approach a gigawatt. The thread running through all of it is that a data centre electrical system is never finished; it is designed, built, verified, operated, maintained and modified continuously, and good documentation and discipline at every stage are what keep it safe and available.

How an electrical design develops

Data centre electrical design follows the usual stages of building design, each ending in a review and a set of deliverables:

Concept or schematic design: the design basis (Lesson 2), utility strategy, redundancy architecture, preliminary one-line diagram and equipment room layout.
Design development: the detailed one-line diagram, equipment sizes, load schedules, preliminary studies and coordination with mechanical, structural, architectural and IT designs.
Construction documents: complete drawings (one-lines, plans, riser diagrams, panel schedules, grounding and lightning protection, details), specifications for every piece of equipment, and the final studies.
Construction support: reviewing contractor submittals and shop drawings, answering questions, and approving changes.
Record documents: as-built drawings and the final studies, which become the basis for operation and every future change.

Timeline of design stages from left to right: concept, design development, construction documents, construction, commissioning and operation. Under each stage, the main electrical deliverables are listed: design basis and preliminary one-line; detailed one-line, load schedules and preliminary studies; drawings, specifications and final studies; submittal reviews; test reports and as-built drawings; and maintenance records and updated studies. — Electrical deliverables accumulate through the life of the project and continue into operation.

The one-line diagram is the single most important document: it shows every source, transformer, breaker, bus and major load, with ratings, and is the map that operators, maintenance staff and emergency responders use. It must be kept current; an out-of-date one-line has misled people into working on circuits they believed were dead.

Engineering studies

A set of power system studies, usually performed with specialized software, proves that the design works:

Power system studies for a data centre.
Study	Question answered	Lesson
Load flow	Do voltages and loadings stay within limits in every operating configuration, including maintenance and failure cases?	2, 6
Short circuit	What fault current is available at each bus, and are all devices rated for it?	7
Protective device coordination	Does only the device nearest a fault operate?	7
Arc flash	What incident energy could a worker be exposed to at each point, and what labels and PPE are needed?	7, 11
Harmonic analysis	Will distortion meet limits at the utility connection and on generator? Is there resonance?	8
Motor starting and transient stability	Can generators and the utility accept large motor starts and load steps?	2, 8
Grounding	Are step and touch voltages at substations within limits?	9
Cable ampacity	Can duct banks carry their circuits without overheating?	4

Studies are not done once. They must be repeated whenever the system changes significantly, when the utility changes its system (which changes available fault current), and periodically: NFPA 70E requires the arc flash risk assessment to be reviewed at least every five years.

Worked problem

A change that invalidates a study

A utility replaces its substation transformer with a larger, lower-impedance unit, and the available fault current at a data centre's 480 V switchboard rises from 42 kA to 58 kA. Its feeder breakers are rated 50 kA interrupting. What must happen?

Show solution

The breakers are now applied above their interrupting rating (58 kA > 50 kA): they could fail violently during a fault (Lesson 7).
The arc flash study is also invalid: incident energies and labels change with fault current, sometimes increasing and sometimes decreasing depending on how clearing times change.
Remedies include replacing or upgrading the breakers, adding current-limiting fuses or reactors, or asking the utility about alternatives, followed by updated short-circuit, coordination and arc flash studies and new labels.

Answer: The equipment is no longer correctly rated and the studies are obsolete. Operators should request available fault current from the utility regularly and whenever the utility changes its system, and codes now require the available fault current to be marked on service equipment and updated when it changes (Lesson 11).

Maintenance

Electrical equipment that is not maintained fails, and when it fails it can injure people. Breakers that are never operated can stick; connections loosen and overheat; insulation degrades; batteries age. NFPA 70B, which became a mandatory standard (rather than a recommended practice) in its 2023 edition, requires an electrical maintenance program with documented procedures, equipment condition assessments and maintenance intervals based on equipment condition, criticality and environment. In Canada, CSA Z463 provides comparable guidance. Key activities include:

Infrared thermography of energized connections, which reveals loose or corroded joints by their temperature.
Exercising and testing circuit breakers, including primary injection testing of their trip units.
Insulation resistance and other tests on cables, transformers and switchgear (Lesson 4).
Transformer oil analysis (dissolved gas analysis) to detect internal faults early.
Battery testing for UPS and DC systems (DCE-340).
Cleaning, torque checks and inspection during planned shutdowns, which concurrently maintainable designs (Tier III and above) allow without interrupting the IT load.

Worked problem

Reading a thermographic survey

An infrared survey finds one bolted bus connection in a PDU at 71 °C, while the same connection on the other two phases, carrying similar current, is at 33 °C. Using a common severity guide (temperature difference between similar components under similar load: 1 °C to 3 °C possible deficiency; 4 °C to 15 °C probable deficiency, repair when possible; over 15 °C major discrepancy, repair immediately), what action is needed?

Show solution

Temperature difference: $71 - 33 = 38$ °C.
38 °C is well above 15 °C: a major discrepancy.

Answer: Immediate repair: plan a safe outage of that PDU (using its redundant partner to carry the load) to clean and re-torque or replace the connection. Left alone, a hot connection gets hotter as its resistance rises, and can fail in an arc flash. Thermography is cheap, quick and one of the highest-value maintenance tasks.

Worked problem

The cost of deferring maintenance

A survey of a facility's 400 low-voltage breakers suggests that, without exercise and testing, about 1% a year will fail to trip correctly when needed; with an annual maintenance program costing $60,000, that falls to 0.2%. If each failure to trip leads, on average, to an incident costing $250,000 (equipment damage and outage), is the program worth it, assuming each breaker is called on to trip once in ten years?

Show solution

Trip demands per year: $400/10 = 40$.
Expected failures without maintenance: $40 \times 0.01 = 0.4$ per year; cost $0.4 \times 250{,}000 = \$100{,}000$.
With maintenance: $40 \times 0.002 = 0.08$; cost $\$20{,}000$ plus \$60,000 program = \$80,000.

Answer: Maintenance saves about $20,000 a year on these illustrative numbers, before counting the value of avoided injuries and reputational damage, which usually dominate. The numbers are rough, but the method, comparing expected failure costs with and without maintenance, is how reliability-centred maintenance programs are justified.

Operating the system safely

Every change to an energized system carries risk. Data centres manage it with formal procedures: methods of procedure (MOPs) for planned work, written step by step and reviewed in advance; emergency operating procedures (EOPs) for failures; standard operating procedures (SOPs) for routine tasks; lockout and tagging; and switching orders that are read back and confirmed. Most serious data centre outages involve human error during maintenance or switching, so these procedures, and training on them, protect availability as much as safety (DCE-450).

The future of data centre power

AI is driving faster change in data centre power than at any time since the industry began. The main directions are:

Higher rack power: AI racks have gone from about 40 kW to over 100 kW in a few years, and designs for racks of several hundred kilowatts to around a megawatt are being published. At these levels, low-voltage AC distribution to the rack becomes unwieldy.
Higher-voltage DC distribution: industry groups, including the Open Compute Project, are developing rack and row power architectures at several hundred volts DC (such as ±400 V or 800 V), which cut current and copper and simplify integration with batteries.
Medium voltage closer to the load: placing MV-to-LV conversion near or within the data hall, and eventually solid-state transformers that convert MV directly to DC with power electronics, removing a conversion stage (ELE-202, Lesson 11).
Energy storage for grid support: large battery systems that smooth AI load swings (Lesson 2), provide ride-through, and earn revenue from grid services.
On-site and dedicated generation: gas turbines, fuel cells, and agreements to restart or build nuclear plants to supply data centres where the grid cannot deliver enough power quickly (DCE-420).

Worked problem

Current at a megawatt rack

Compare the current needed to supply a 1 MW rack at (a) 415 V three-phase AC at power factor 0.98 and (b) 800 V DC.

Show solution

(a) $I = 1{,}000{,}000/(\sqrt{3} \times 415 \times 0.98) = 1{,}420$ A per phase.
(b) $I = 1{,}000{,}000/800 = 1{,}250$ A.

Answer: About 1,420 A per phase at 415 V AC, and 1,250 A at 800 V DC, either of which is the current of a large building service delivered to a single rack. This is why rack power architectures are being redesigned around higher DC voltages, liquid-cooled busbars and conversion closer to the load, and why new safety standards for high-power DC in data halls are being developed.

Canadian codes and standards are adapting: the Canadian Electrical Code is revised every three years, and new requirements for DC distribution, energy storage and high-power IT equipment appear in successive editions; designers should check the edition adopted in each province.

Careers in data centre electrical engineering

Data centre growth has created strong demand for electrical engineers and technicians: design engineers in consulting firms, owner's engineers and project managers, commissioning agents, critical facilities engineers and electricians in operations, power systems and protection specialists, and engineers at equipment manufacturers. Licensure as a professional engineer is required to take responsibility for designs, and trades certification for electrical work; the courses of this program lead toward both.

Common mistakes

MistakeLetting the one-line diagram fall out of date.
InsteadUpdate it with every change; it is the map for safe work.
MistakeTreating studies as one-time design tasks.
InsteadRepeat them after system or utility changes and at least every five years for arc flash.
MistakeRunning equipment to failure.
InsteadMaintain to NFPA 70B or CSA Z463 with condition-based intervals.
MistakeIgnoring a large thermographic temperature difference.
InsteadRepair major discrepancies immediately, using redundancy to do it safely.
MistakePerforming switching from memory.
InsteadUse written, reviewed MOPs and switching orders.

Summary and key terms

Electrical design progresses from concept to construction documents and record drawings; the one-line diagram is central.
Load flow, short-circuit, coordination, arc flash, harmonic, motor starting, grounding and ampacity studies prove the design.
Studies must be updated when the system or utility changes.
NFPA 70B (a standard since 2023) and CSA Z463 guide maintenance; thermography and breaker testing are high-value tasks.
MOPs, EOPs, SOPs and switching orders control the risk of human error.
AI is pushing rack power toward a megawatt, higher-voltage DC, MV closer to the load, storage and on-site generation.

One-line diagram: A single-line representation of the power system's sources, buses, devices and loads.
Load schedule: A table of loads and the panels or buses that serve them.
Power system study: An engineering analysis proving a system's performance in an aspect such as faults or harmonics.
Record drawings: Drawings updated to show the system as built.
Infrared thermography: Imaging temperatures to find overheating connections.
Primary injection test: Testing a breaker by passing high current through it.
Dissolved gas analysis: Analysing gases in transformer oil to detect faults.
MOP: Method of procedure: a written, reviewed plan for a specific piece of work.
Solid-state transformer: A power-electronic converter that replaces a conventional transformer.
Concurrent maintainability: The ability to maintain any component without interrupting the critical load.

Practice problems

Work each problem on paper first, then reveal the answer to check.

What is the most important electrical document for operators?
Show answer
The current one-line diagram.
Which study checks that only the nearest device trips?
Show answer
Protective device coordination.
How often must the arc flash risk assessment be reviewed under NFPA 70E?
Show answer
At least every five years, and after significant changes.
Available fault current rises above a breaker's interrupting rating. Name two remedies.
Show answer
Replace the breakers with higher-rated ones; add current-limiting fuses or reactors.
Thermography: 52 °C versus 44 °C on similar phases. Severity?
Show answer
8 °C: probable deficiency, repair when possible.
What changed about NFPA 70B in 2023?
Show answer
It became a mandatory standard rather than a recommended practice.
Why do Tier III designs make maintenance easier?
Show answer
They are concurrently maintainable, so equipment can be isolated without dropping the load.
What is a MOP?
Show answer
A written, reviewed, step-by-step method of procedure for planned work.
Current for a 500 kW rack at 800 V DC?
Show answer
625 A.
Name two technologies reshaping data centre power for AI.
Show answer
Higher-voltage DC distribution, solid-state transformers, battery storage for load smoothing, on-site generation.