Maintenance and incident response review

Maintenance keeps redundancy real, and incident response limits the damage when something fails anyway. DCT-340 taught both at Stonebridge: the maintenance program, condition monitoring, emergency operating procedures, the incident process, root cause analysis and shift hand-offs. Exams test these ideas through scenarios more than definitions, so this review gives you the vocabulary, the arithmetic of reliability, and a set of worked cases built from incidents earlier in the program.

Kinds of maintenance

• Preventive maintenance (PM) is done on a schedule, by time or run hours, whether or not anything looks wrong: filter changes, belt checks, battery inspections, breaker exercising.
• Predictive or condition-based maintenance acts on measurements that show wear: infrared scans of electrical joints, battery ohmic readings, vibration on fans and pumps, oil analysis on generators.
• Corrective maintenance repairs something that has failed or is out of specification.
• Run-to-failure is a deliberate choice for cheap, non-critical items whose failure does no harm, such as some lamps; it is never chosen for anything in the critical path.

A maintenance program lists every asset, the tasks for each, their intervals, who does them (staff or vendor) and the procedures. Electrical maintenance programs in North America often follow NFPA 70B, which became a standard with mandatory requirements in its 2023 edition, and testing specifications such as NETA MTS. Generators are exercised under load monthly and load-banked annually under the site's program, in line with CSA C282 (DCT-220 lesson 7). Batteries are inspected and measured to IEEE practices or the maker's instructions (DCT-220 lesson 5).

Concurrent maintenance in practice

A Tier III site such as Stonebridge is designed so that any component can be removed for maintenance without shutting down IT load. That does not mean maintenance is free of risk. While a unit is out, the system runs with less redundancy, often at N, so maintenance is planned as a change with a MOP, scored for risk in the reduced state, and scheduled in a change window (lesson 8, DCT-330). PM compliance, the share of scheduled tasks done on time, is one of the main measures of a maintenance program, and a growing backlog of overdue PMs is an early sign of trouble.

Condition monitoring

Infrared thermography finds loose or corroded electrical connections, which run hotter than their neighbours. Scans are usually done under load, through infrared windows or by qualified staff with covers removed, and results are graded by the temperature rise over a similar component or over ambient. Battery ohmic readings (internal resistance or conductance) trend upward as cells age, and a rise of a set percentage over the cell's baseline triggers replacement; DCT-220 lesson 5 found UPS-A2 string 2's failing blocks from the runtime trend. Vibration on fan and pump bearings rises before they fail.

Reliability arithmetic

Mean time between failures (MTBF) is the average operating time between failures; mean time to repair (MTTR) is the average time to restore service after one. Availability from these two is MTBF / (MTBF + MTTR). Shortening MTTR, with spares on site, clear procedures and trained staff, improves availability as much as making failures rarer.

Failure rates are not constant over a component's life. The bathtub curve shows early failures in the first weeks (infant mortality, from defects and installation errors), a long period of low, steady random failures, and a rise at the end of life as wear sets in. Good practice is burn-in testing for new equipment (as in the DCT-310 capstone) and replacement before the wear-out period for items such as batteries, capacitors and fans.

# Reliability review: availability from MTBF and MTTR, and PM compliance
def availability(mtbf_h, mttr_h):
    return mtbf_h / (mtbf_h + mttr_h)
for mttr in (24, 8, 2):
    a = availability(50000, mttr)
    print(f"MTBF 50,000 h, MTTR {mttr:>2} h -> {a * 100:.4f} %, {(1 - a) * 8760 * 60:.0f} min a year down")

events = [  # (unit, hours in service, failures, total repair hours) over one year
    ("CRAH fans", 12 * 8760, 3, 18), ("CDU pumps", 18 * 8760, 2, 7), ("Rack PDUs", 400 * 8760, 5, 10)]
for unit, hours, n, rep in events:
    print(f"{unit:<10} MTBF {hours / n:>10,.0f} h  MTTR {rep / n:4.1f} h")

scheduled, on_time, late = 412, 391, 15
print(f"PM compliance {on_time / scheduled * 100:.1f} %, overdue backlog {scheduled - on_time - late} not started")

MTBF 50,000 h, MTTR 24 h -> 99.9520 %, 252 min a year down
MTBF 50,000 h, MTTR  8 h -> 99.9840 %, 84 min a year down
MTBF 50,000 h, MTTR  2 h -> 99.9960 %, 21 min a year down
CRAH fans  MTBF     35,040 h  MTTR  6.0 h
CDU pumps  MTBF     78,840 h  MTTR  3.5 h
Rack PDUs  MTBF    700,800 h  MTTR  2.0 h
PM compliance 94.9 %, overdue backlog 6 not started

Emergency operating procedures

An emergency operating procedure (EOP) tells the desk and the floor what to do when a specific failure happens: loss of utility, UPS on bypass, CRAH failure, leak in Hall D, fire alarm, generator failure to start. EOPs are short, written for stress, and practised in drills. They begin with immediate actions that protect people and load, then diagnosis, then recovery, and they say who to call at each step. The DCT-220 capstone storm night ran on the loss-of-utility EOP, and the GEN-D2 failure to start was handled by its contingency step.

Incident response

An incident is any unplanned event that affects or threatens service. The response has a common shape across standards and frameworks such as ITIL: detect, respond and contain, communicate, recover, and learn (DCT-340). One person takes the role of incident lead and coordinates; others work the problem; one person keeps a timeline of what happened and when. Customers and leadership hear early and regularly, even if the update is that the cause is not yet known.

Useful timing measures are time to detect (from the event to the first alarm or report), time to acknowledge, time to contain (when the load is safe) and time to resolve. Tracking them over many incidents shows where the response is slow.

Root cause analysis

Root cause analysis (RCA) looks past the immediate failure to the conditions that let it happen. Common tools are the five whys (asking why repeatedly until the answer is something the organization controls), the fishbone or Ishikawa diagram (grouping causes under headings such as people, procedures, equipment, environment and materials), and fault trees (showing how combinations of failures lead to the top event through AND and OR gates). A good RCA ends in corrective actions with owners and dates, not in blaming a person.

Shift hand-offs

Many incidents begin, or are made worse, at shift change, when knowledge passes from tired people to fresh ones. A good hand-off is written and spoken, follows a fixed order (safety first, then redundancy state, open incidents, changes in progress or planned, open tickets near their SLA, and anything unusual), and gives the incoming shift time to ask questions. Dmitri's whiteboard of the redundancy state is the first thing the incoming shift lead reads.

Exam traps on maintenance and incidents

• Concurrent maintenance reduces redundancy during the work; treat it as a change.
• Availability = MTBF / (MTBF + MTTR); faster repair helps as much as fewer failures.
• New equipment fails more often at first; burn-in catches early failures.
• Contain first, then diagnose; the load's safety comes before finding the cause.
• RCA aims at system causes and corrective actions, not at blame.
• An EOP is used during an emergency; a MOP is used for a planned change.

Battery maintenance in practice

UPS batteries are the most failure-prone part of the power chain and one of the most tested topics. Valve-regulated lead-acid (VRLA) batteries have a design life often quoted as 5 to 10 years at 25 °C (77 °F), but heat shortens it sharply: a common rule of thumb is that every sustained rise of about 8 to 10 K (15 to 18 °F) above 25 °C (77 °F) roughly halves battery life. Lithium-ion batteries, now common in new UPS systems, last longer and tolerate heat better, and they come with a battery management system that monitors each cell. Both need regular inspections and measurements.

The figures in this example are an illustrative site rule; follow the battery maker's guidance and the site program for real limits.

Spares and MTTR

MTTR is the sum of several smaller times: time to detect, time to diagnose, time to get the part, time to repair and time to verify. Spares on site cut the part time from days to minutes. Stonebridge keeps critical spares for each system class, chosen by asking which failures would take longest to fix without a spare: UPS control boards, CRAH fan motors, CDU pumps, ATS controllers, breakers of each frame size, and drives and power supplies for the most common customer servers by contract. Each spare is logged in the maintenance system, checked periodically and replaced as soon as it is used.

Fault trees and combined failures

A fault tree shows how failures combine to cause a top event. An AND gate means all inputs must fail; an OR gate means any one is enough. For independent events with small probabilities, the probability through an AND gate is the product of the inputs, and through an OR gate it is close to the sum.

Communicating during an incident

Clear communication is part of incident response. Stonebridge's incident updates follow a fixed form: what happened and when, what is affected, what is being done, what customers need to do (often nothing), and when the next update will come. Updates go out at fixed intervals even when nothing has changed, because silence makes customers assume the worst. Technicians do not speculate about causes in customer conversations; they pass questions to the incident lead, who answers from the facts.

Post-incident reviews

After any significant incident or near miss, the team holds a review within a few days, while memories are fresh. The review rebuilds the timeline from logs, alarms and interviews, identifies what went well and what did not, finds root causes and contributing factors, and agrees corrective actions with owners and dates. Reviews are blameless: they ask how the system let a mistake happen and how to make the right action easy, rather than who to punish. People who fear blame hide mistakes, and hidden mistakes come back as outages. An exam that offers disciplining the technician as the main corrective action is offering the wrong answer.