Maintenance strategies

A data centre that is well designed and thoroughly commissioned will still fail if it is poorly maintained. Breakers that are never exercised stick, batteries age, filters clog, connections loosen, fuel degrades and firmware becomes outdated. Yet maintenance itself is risky: a large share of data centre outages happen while equipment is being maintained, through switching errors, procedures that do not match the plant, or a second failure while redundancy is reduced. This lesson describes the main maintenance strategies, from reactive to reliability-centred, the techniques of condition monitoring, the standards that set maintenance requirements, how to choose replacement intervals, and how to perform maintenance safely on a live facility.

Four strategies

Maintenance strategies differ in when work is triggered:

• Reactive (run to failure): fix things when they break. Acceptable for cheap, non-critical items whose failure has no effect on the IT load, such as an office light, and wasteful or dangerous for anything critical.
• Preventive (time-based): perform tasks at fixed intervals, such as annual breaker servicing, quarterly filter changes or replacing UPS capacitors every few years, regardless of condition. Simple to plan and effective for wear-out failures with predictable timing, but it can replace healthy parts and, by disturbing equipment, introduce new faults.
• Predictive (condition-based): monitor the condition of equipment and intervene when the measurements show degradation. Thermography, oil analysis, vibration analysis, battery monitoring and partial discharge testing are typical techniques. It avoids unnecessary work but needs instruments, data and skilled interpretation.
• Reliability-centred maintenance (RCM): a structured process that chooses the right strategy for each failure mode of each asset, based on its consequences and how it can be detected. RCM builds on the FMEA of Lesson 7.

A mature data centre uses all four: run to failure for trivial items, time-based tasks where wear-out is predictable or the standards require them, condition monitoring for major electrical and rotating equipment, and RCM to decide which is which.

Condition monitoring techniques

The baseline from commissioning (Lesson 3) is what gives these measurements meaning: an insulation resistance of 800 MΩ means little alone, but a fall from 5,000 MΩ at acceptance to 800 MΩ over three years is a clear warning. Monitoring systems that continuously record battery resistance, breaker operations, temperatures and power quality allow trends to be watched without a technician visiting.

Maintenance standards

In Canada, CSA Z463, Maintenance of electrical systems, sets out an electrical maintenance programme: planning, inspection, testing, and maintenance intervals based on equipment condition and criticality. Emergency generators are inspected and tested to CSA C282, which sets a schedule of inspections and operating tests under load, from weekly and monthly checks to annual tests. Workplace safety during maintenance follows CSA Z462.

Both electrical maintenance standards tie the maintenance interval to equipment condition, assessed from its physical state, its criticality and its operating environment, and they expect results to be recorded and trended. Manufacturers' instructions, the NETA MTS testing standard and the owner's insurer add further requirements. Mechanical maintenance draws on manufacturers' recommendations and standards such as ASHRAE 180, Standard Practice for Inspection and Maintenance of Commercial Building HVAC Systems.

Choosing a replacement interval

For components that wear out, such as UPS DC capacitors and fans, generator starter batteries, and bearings, the question is when to replace them before they fail. Replacing too early wastes money and introduces disturbance; replacing too late lets failures occur in service, where they are far more costly. The Weibull distribution models wear-out well. Its shape parameter β describes the failure pattern: β < 1 means early failures, β = 1 means a constant rate (random failures), and β > 1 means wear-out, with failures concentrated around the characteristic life η.

An age replacement policy replaces a component at age T or on failure, whichever comes first. The best T minimizes the long-run cost per year: the expected cost of a cycle (a planned replacement if the part survives to T, a failure replacement if it does not) divided by the expected length of a cycle.

import numpy as np
from scipy.integrate import quad

# Age replacement of UPS DC capacitors: replace at age T, or on failure if sooner
eta, beta = 9.0, 3.5          # Weibull scale (years) and shape: wear-out failures
cost_planned = 6000           # planned replacement during a maintenance window
cost_failure = 60000          # failure in service: emergency call-out plus risk to the load

def R(t):
    return np.exp(-(t / eta) ** beta)

def cost_rate(T):
    expected_cycle = quad(R, 0, T)[0]           # mean time per replacement cycle
    expected_cost = cost_planned * R(T) + cost_failure * (1 - R(T))
    return expected_cost / expected_cycle

for T in (3, 4, 5, 6, 7, 8, 10, 15):
    print(f"Replace every {T:>2} years: {cost_rate(T):7.0f} per year, P(fail first) = {1 - R(T):.3f}")

grid = np.arange(2, 15.01, 0.1)
best = grid[np.argmin([cost_rate(T) for T in grid])]
print(f"Lowest cost interval: about {best:.1f} years ({cost_rate(best):.0f} per year)")
print(f"Run to failure only: {cost_failure / quad(R, 0, np.inf)[0]:.0f} per year")

Replace every  3 years:    2392 per year, P(fail first) = 0.021
Replace every  4 years:    2297 per year, P(fail first) = 0.057
Replace every  5 years:    2566 per year, P(fail first) = 0.120
Replace every  6 years:    3089 per year, P(fail first) = 0.215
Replace every  7 years:    3789 per year, P(fail first) = 0.340
Replace every  8 years:    4585 per year, P(fail first) = 0.484
Replace every 10 years:    6098 per year, P(fail first) = 0.764
Replace every 15 years:    7394 per year, P(fail first) = 0.997
Lowest cost interval: about 3.7 years (2280 per year)
Run to failure only: 7409 per year

With illustrative data for UPS capacitors (characteristic life 9 years, β = 3.5, a failure in service costing ten times a planned replacement), replacing every 3.7 years minimizes cost at about 2,280 a year, roughly a third of the cost of running to failure. The curve is flat near the optimum, so a round 4-year interval costs almost the same and is easier to schedule. Notice that at the optimum, about 5 % of capacitors still fail before replacement: preventive maintenance reduces failures but does not eliminate them, which is why redundancy is still needed. The analysis only makes sense for β > 1; for random failures, replacing a working part on a schedule does nothing to reduce the failure rate.

Maintenance risk and the method of procedure

Every maintenance task on critical infrastructure changes the plant's configuration, and configuration changes are where outages happen. Data centre operators control this with a written method of procedure (MOP) for every task that affects critical systems, and a site-specific emergency operating procedure (EOP) for responding if something goes wrong. A good MOP includes:

• Scope, the equipment affected and the risk assessment, including the effect on redundancy (for example, the facility will be at N rather than N+1 for four hours).
• Prerequisites: approvals, notification to customers or IT teams, spare parts on hand, weather and other work checked, no conflicting maintenance.
• Step-by-step switching and work instructions, with expected indications at each step and space to initial each step as completed.
• Back-out steps: how to return to a safe configuration at any point if something goes wrong.
• Restoration and verification steps, and sign-off.

The MOP is reviewed by a second qualified person, rehearsed for complex tasks, and executed with one person reading each step aloud and another performing it (sometimes called the challenge-and-response method). Many outages begin with a technician working from memory or from a generic manufacturer procedure that does not match the site's switching arrangement.

Spare parts and the CMMS

Spares directly reduce MTTR (Lesson 6). A spares strategy identifies critical parts with long lead times, such as UPS power modules, breaker trip units, control boards, generator starters and pump seals, and holds them on site or under vendor agreements with guaranteed delivery times. A computerized maintenance management system (CMMS) records every asset, its maintenance plan, work orders, results, parts used and failure history. The CMMS turns maintenance into data: failure records improve the reliability models, trends show deteriorating assets, and compliance reports show auditors and customers that required tasks were done.

Measuring maintenance

Useful key performance indicators include the percentage of scheduled preventive tasks completed on time, the ratio of planned to reactive work (a high reactive share signals poor maintenance), the backlog of outstanding work orders, mean time to repair by equipment class, repeat failures, and incidents caused by maintenance. These are reviewed regularly by operations management and feed back into the maintenance plan.