Running a data center: testing, operations and incidents

A data center is built in two or three years and then operated for twenty or more. Its reliability depends as much on how it is tested, operated and maintained as on how it was designed: industry surveys consistently find that a large share of serious outages involve human error or procedural failures, and that most could have been prevented with better management, training or processes. Lesson 11 introduced the commissioning levels and the main operating documents. This final lesson goes further: how integrated systems testing is planned and run, how a facility is staffed and operated day to day, the main maintenance strategies, how capacity is managed as load grows, how incidents are handled and learned from, and how facilities are eventually upgraded or retired.

Integrated systems testing in practice

Integrated systems testing (IST, commissioning Level 5) is the final exam for a new facility. With the building complete and before any customer equipment arrives, the commissioning team loads the data halls with load banks (portable resistive heaters that draw power and produce heat like servers) and runs scripted tests that simulate failures and maintenance events. A typical IST script includes:

• Utility failure: open the utility breakers and verify that UPS systems carry the load on battery, generators start and synchronize, transfer switches operate, the UPS walks in onto generator power, and cooling recovers without temperatures exceeding limits.
• Utility return: restore the utility and verify the retransfer sequence.
• Component failures: trip a UPS module, a generator, a chiller or a pump during operation and confirm that redundant equipment takes over.
• Maintenance scenarios: take each piece of equipment out of service using the written procedures to prove concurrent maintainability.
• Controls and monitoring: verify alarms, interlocks, building management and power monitoring responses.
• Pull-the-plug tests: some owners simulate the worst cases, such as losing a whole electrical or cooling path.

Each test has acceptance criteria (for example, rack inlet temperatures stay within the ASHRAE allowable range, no IT load is dropped, generator frequency stays within limits) and results are recorded, with deficiencies fixed and retested. IST often takes several weeks for a large facility and is followed by a period of burn-in operation.

Staffing and shifts

Most large data centers are staffed 24 hours a day, 7 days a week, by teams of critical facilities technicians and engineers, with specialists and vendors called in for complex work. A round-the-clock post requires about five full-time staff, because a year has 8,760 hours while each person works roughly 1,800 to 2,000 hours after vacation, training and sick time.

Operations teams work from a network or facility operations center that monitors alarms and dashboards, coordinates with IT and customers, and manages work on site. Clear roles, shift handovers with written logs, and regular drills for emergencies (such as a utility failure at night) keep the team ready.

Procedures and change management

Critical facilities run on written procedures:

• Standard operating procedures (SOPs) for routine tasks such as rounds, readings and generator tests.
• Methods of procedure (MOPs) for every planned activity on critical equipment, with step-by-step actions, checks, the expected state after each step, risks, and back-out steps if something goes wrong. A MOP is reviewed and approved before work and followed line by line, often with two people reading and confirming each step.
• Emergency operating procedures (EOPs) for failures, giving technicians a tested sequence to stabilize the facility under pressure.
• Site configuration procedures and up-to-date single-line diagrams, labeling and lock-out/tag-out procedures.

Change management requires every change to critical systems, including software and control settings, to be described, assessed for risk, approved and scheduled, often during low-risk windows, with customers notified when appropriate.

Maintenance strategies

Maintenance keeps equipment reliable. Several strategies are combined:

• Reactive (run to failure): acceptable only for non-critical items with redundancy.
• Preventive (time-based): tasks at fixed intervals, such as annual breaker testing, generator oil changes and filter replacement.
• Predictive (condition-based): monitoring condition and acting when it changes, using infrared thermography of electrical connections, vibration analysis of pumps and fans, oil analysis, battery impedance testing and partial discharge monitoring of medium-voltage equipment.
• Reliability-centered maintenance (RCM): a structured analysis of each asset's failure modes and consequences to choose the most effective tasks.

Maintenance itself carries risk: each intervention can introduce errors, so concurrently maintainable designs (Lesson 8) and well-written MOPs are essential. Over-maintenance can cause failures as well as prevent them.

Monitoring and capacity management

Facilities generate huge amounts of data from the building management system (BMS) for mechanical equipment, the electrical power monitoring system (EPMS), environmental sensors, security systems and IT equipment. Data center infrastructure management (DCIM) software brings these together with an inventory of every rack, device and connection, so operators can see:

• Power: the load on every UPS, PDU and circuit, and whether redundancy is still intact (for example, that no 2N path is loaded above 50%).
• Cooling: temperatures, airflow and liquid cooling flows by zone.
• Space and connectivity: free rack units, ports and floor positions.
• Trends and forecasts: when capacity will run out at current growth.

Capacity management prevents a common failure: stranded or overcommitted capacity. Power may be available in one area but cooling in another, or a customer may reserve capacity they never use. Operators track each resource and plan expansions before limits are reached.

Incidents and learning

Despite all precautions, incidents happen. Good incident management follows a pattern: detect and alert quickly; stabilize the facility using EOPs; communicate with customers and management; restore normal operation; and then investigate. A root cause analysis (RCA) looks beyond the immediate cause (for example, a technician opened the wrong breaker) to the conditions that allowed it (unclear labeling, a missing step in the MOP, fatigue, time pressure, a confusing design) and recommends changes that prevent recurrence. Blameless post-incident reviews, borrowed from aviation and software operations, encourage people to report mistakes and near misses honestly.

Uptime Institute's annual outage analyses have found that power problems remain the most common cause of significant on-site outages, that a large majority of operators believe their most recent serious outage was preventable, and that failures to follow procedures and flawed procedures are leading contributors to human-error outages. The lesson is consistent: reliability comes from people, processes and culture as much as from equipment.

Upgrades, refreshes and end of life

Over its life, a facility will see several generations of IT equipment and at least one major refresh of electrical and mechanical systems: UPS modules and batteries are replaced every 10 to 15 years (lead-acid batteries every 4 to 8), generators may last 25 years or more, and chillers and cooling units 15 to 25 years. Upgrading equipment in a live facility is among the riskiest work in the industry and is planned with the same care as construction. Many existing facilities are now being retrofitted for liquid cooling and higher densities to host AI equipment, sometimes by converting a hall at a time.

At the end of life, decommissioning includes secure destruction of data-bearing media, recycling of IT equipment, batteries, copper and steel, safe disposal of refrigerants and fuel, and, where possible, reuse of the building and its power connection, which is often the most valuable asset of an old site.

Bringing the course together

A data center is a system in which every part affects every other: the IT equipment sets power and heat (Lesson 2); power flows through redundant electrical chains (Lessons 3 and 4); heat is removed by air and liquid (Lessons 5 and 6); networks connect it to the world (Lesson 7); reliability comes from redundancy and maintainability (Lesson 8); the building protects it (Lesson 9); sustainability shapes its acceptance (Lesson 10); a project brings it into being (Lesson 11); and operations keep it running (this lesson). The remaining courses in the program study each of these systems in depth.