Operating and maintaining backup power

Design and commissioning make a backup power system capable; operations keep it that way for twenty years or more. Batteries age, fuel degrades, connections loosen, settings drift, staff change, and every maintenance activity on live critical equipment is a chance for error. Industry surveys of data center outages consistently find power problems to be the most common cause of significant outages, and human error, including procedures that were wrong or not followed, contributes to a large share of them. This lesson covers maintenance strategies, the care of each component, monitoring and trending, the procedures and training that make work on critical systems safe, emergency operations, and lessons from common failure patterns.

Maintenance strategies

• Preventive maintenance: tasks at fixed intervals, from manufacturers' recommendations and standards such as NFPA 70B (electrical equipment maintenance, now a standard in the US) and CSA Z463 in Canada: cleaning, inspection, torque checks, filter and fluid changes, testing.
• Predictive (condition-based) maintenance: monitoring condition and acting when indicators show deterioration, such as rising battery resistance, increasing vibration, hot connections on thermographic surveys, or oil analysis showing wear metals.
• Corrective maintenance: repairs after a failure or defect is found, ideally rare and planned rather than urgent.

A computerized maintenance management system (CMMS) schedules tasks, records work and findings, tracks spare parts and asset histories, and provides the evidence auditors and customers ask for.

Caring for each component

UPS

• Annual (or semi-annual) preventive maintenance by trained technicians: inspection, cleaning, capacitor and fan checks, firmware review, and functional tests of transfers.
• Replacement of components with limited life: DC and AC capacitors (often every 5 to 10 years), cooling fans (every 5 to 7 years), and, for some designs, power modules.
• Thermographic scans of connections under load.

Batteries

Batteries are the most failure-prone part of a UPS system. Care includes:

• Continuous monitoring of string and block voltages, temperatures and internal resistance, with alarms for outliers (Lesson 3).
• Temperature control: VRLA life halves for roughly every 8 to 10 °C above 25 °C.
• Periodic capacity tests and planned replacement of whole strings before capacity falls below the design margin, rather than waiting for failures.

import numpy as np

# Internal resistance of 40 VRLA blocks in a UPS string, measured quarterly (milliohms)
rng = np.random.default_rng(9)
baseline = rng.normal(3.0, 0.08, 40)
quarters = 12
history = np.zeros((quarters, 40))
for q in range(quarters):
    ageing = 1 + 0.012 * q                                   # normal slow rise for every block
    history[q] = baseline * ageing * rng.normal(1, 0.01, 40)
history[7:, 17] *= np.linspace(1.10, 1.60, quarters - 7)     # block 17 starts failing in quarter 8
history[10:, 31] *= np.array([1.22, 1.30])                   # block 31 deteriorates late

latest = history[-1]
vs_baseline = latest / baseline - 1
vs_string = latest / np.median(latest) - 1
flag = np.where((vs_baseline > 0.30) | (vs_string > 0.20))[0]
print(f"string median now {np.median(latest):.2f} mOhm (baseline median {np.median(baseline):.2f})")
for b in flag:
    print(f"block {b:2d}: {latest[b]:.2f} mOhm, +{100 * vs_baseline[b]:.0f}% vs its baseline, +{100 * vs_string[b]:.0f}% vs string median")
first = {int(b): int(np.argmax(history[:, b] / np.median(history, axis=1) > 1.20)) + 1 for b in flag}
print("first quarter above the 20% string-median alarm:", first)

string median now 3.41 mOhm (baseline median 3.00)
block 17: 5.64 mOhm, +81% vs its baseline, +65% vs string median
block 31: 4.31 mOhm, +48% vs its baseline, +26% vs string median
first quarter above the 20% string-median alarm: {17: 9, 31: 11}

A single failed block can disable a whole series string, because the string's current must pass through it, so finding and replacing weak blocks early protects the runtime of the entire battery. Replacing individual blocks in an old string has limits, however: a new block among aged ones is mismatched, and strings near end of life are usually replaced complete.

Generators

• Monthly runs under load and exercise of transfer equipment, as required by the applicable standard (Lesson 10).
• Starting systems: starting batteries tested and replaced on schedule; dead or weak starting batteries are one of the most common causes of generators failing to start. Many sites install redundant starters or batteries.
• Engine heaters, coolant, oil and filters maintained; oil and coolant analysis for wear and contamination.
• Fuel management: testing for water, microbial growth and degradation, polishing (filtering) stored fuel, and turning over stock; biodiesel blends degrade faster (Lesson 5).
• Annual load bank testing where monthly runs are at light load, to avoid wet stacking (unburned fuel accumulating in the exhaust).

Switchgear, transfer switches and breakers

• Periodic operation of breakers and transfer switches so that mechanisms and lubricants stay free.
• Testing of trip units and protective relays against settings from the coordination study (Lesson 8).
• Infrared thermography and, for medium-voltage gear, partial discharge testing to find insulation deterioration.
• Cleaning and torque checks during planned outages of each section, possible only if the design is concurrently maintainable.

Monitoring and trending

Modern facilities collect data continuously from UPS units, battery monitors, generator controllers, meters and relays through the building or electrical power monitoring system (EPMS). Value comes from using the data:

• Alarms with clear priorities and responses (DCE-330, Lesson 8), not floods.
• Trends of battery resistance, UPS efficiency and component temperatures, reviewed regularly.
• Event records synchronized in time across equipment, which allow post-incident analysis to reconstruct exactly what happened.
• Capacity reports: loading of each UPS and generator against its rating and redundancy, so that growth does not silently erode N+1.

Procedures and training

Most work on critical power systems is done while the facility is running. Errors in that work, opening the wrong breaker, missing a step, misreading a one-line diagram, cause many outages. Good practice includes:

• Methods of procedure (MOPs) for every planned activity, reviewed by operations and engineering, with each step, expected result, hold points, and back-out steps.
• Standard operating procedures (SOPs) for routine operations, and emergency operating procedures (EOPs) for failures.
• Labeling that matches the drawings exactly, with up-to-date one-line diagrams at the equipment.
• Two-person verification for critical switching, with one person reading the procedure and the other operating.
• Change management: no setting, firmware or configuration change without review and records.
• Training and drills: operators practice emergency procedures on simulators or during planned exercises, and new staff are qualified before working alone.
• Electrical safety programs (CSA Z462, NFPA 70E): energized work only when justified, with arc flash PPE, permits and boundaries.

Emergency operations

When something fails, operators need to act quickly and correctly:

• Emergency operating procedures for each likely event: utility failure with a generator that will not start, UPS module failure, battery thermal event, fuel supply problem, loss of cooling.
• Clear authority and communication: who decides, who is informed (customers, management, the utility, emergency services), and how.
• Manual operation skills: operators who can manually start and parallel generators, operate transfer switches in bypass, and put a UPS into maintenance bypass, if automatic controls fail.
• Fuel contracts and priority delivery agreements for extended outages, tested during regional emergencies.

After every incident, including near-misses, a post-incident review finds root causes and contributing factors without blame, and changes procedures, training or design to prevent recurrence.

Spare parts and vendor support

Maintenance plans are only as good as the parts and expertise behind them. A failed UPS power module, generator controller or breaker trip unit with a twelve-week lead time can leave a site at reduced redundancy for months. Operators therefore keep a critical spares list, based on the failure modes that would reduce redundancy and the lead time to replace each part: typical items include UPS power and control modules, fans and capacitors, generator starter motors, batteries for starting, fuel filters, sensors, and trip units or whole spare breakers for common frames.

Service contracts with equipment manufacturers define response times, parts availability and access to firmware updates. As equipment ages, manufacturers announce end of support, after which parts and expertise become scarce; tracking these dates lets the owner plan replacements as projects rather than emergencies. Firmware itself needs management: updates to UPS and paralleling controls can change behavior, so they are tested, scheduled under a MOP and recorded like any other change.

Common failure patterns

Experience across the industry shows recurring causes of backup power failures:

• Generator fails to start: weak starting batteries, failed starters, fuel contamination, or a control issue such as a switch left in the "off" or "manual" position after maintenance.
• UPS failures: aged capacitors or fans, battery strings with a failed block, firmware faults, or a static switch problem.
• Transfer failures: ATS mechanical problems, or control sequences that fail in conditions never tested.
• Breaker trips during transfer: settings that coordinate on utility power but trip on generator power (Lesson 8).
• Human error during maintenance: wrong breaker opened, redundancy lost unknowingly during work, procedures not followed.
• Cascading failures: one failure overloading the redundant equipment, for example when a UPS module fails during a period when another is in maintenance.