Preventive, predictive and corrective maintenance

In his first week on the rotation Jamal spends a morning with Morgan Tremblay, the campus maintenance planner, in a small office lined with binders and two large screens. One screen shows the computerized maintenance management system, the CMMS, with a list of the week's work orders; the other shows the monitoring trends Morgan uses to decide what needs attention. Morgan's job is to keep about four thousand pieces of equipment on the campus in good condition at a reasonable cost, without the maintenance itself causing the outages it is meant to prevent. That balance is the subject of this lesson: the different ways of maintaining equipment, what each is good for, and how a facility like Stonebridge's mixes them.

Four ways to maintain equipment

Every maintenance strategy answers the same question differently: when should we do something to this piece of equipment? There are four broad answers.

• Run to failure, also called reactive maintenance: do nothing until it breaks, then repair or replace it.
• Preventive maintenance, also called time-based or scheduled maintenance: service it at fixed intervals of time or use, whether it seems to need it or not.
• Predictive maintenance, also called condition-based maintenance: measure its condition and act when the measurements show deterioration.
• Corrective maintenance: repair a fault that has been found, whether by an inspection, a measurement or a failure.

The first three are strategies chosen in advance for a type of equipment. Corrective maintenance is what happens when any of them finds something: a technician on a preventive visit finds a cracked belt, a predictive measurement shows a hot connection, or a unit simply fails. In the CMMS each kind gets its own work order type, so the planner can see how much of the team's time goes into planned work and how much into fixing things.

Run to failure

Run to failure sounds careless, but it is the right strategy for some equipment. A light fitting in a corridor, a desk fan in an office or a monitor in the operations centre fails without harming anything, is cheap to replace and gives little warning that maintenance could catch. Spending a technician's time inspecting it would cost more than replacing it when it fails. In a data centre, run to failure is also accepted for some redundant, low-cost parts with spares on the shelf, such as individual server fans in a tenant's equipment, which the tenant replaces when they report failure.

Run to failure is wrong for anything whose failure harms people, drops the critical load, removes redundancy that is hard to restore or causes expensive secondary damage. A chilled-water valve that fails by leaking, a breaker that fails to open on a fault or a generator that fails to start are not candidates. The decision is made deliberately, item by item, and written into the maintenance program, as Lesson 3 describes. What is not acceptable is run to failure by neglect: equipment that was supposed to be maintained and simply was not.

Preventive maintenance

Preventive maintenance replaces wear parts, cleans, lubricates, adjusts and tests equipment at fixed intervals. Most of the maintenance in a data centre is preventive, because manufacturers specify it, standards require much of it and it is easy to plan. Typical examples on the campus are quarterly filter changes in the air handlers, semi-annual inspections of the UPS systems by the manufacturer's field engineers, monthly generator runs under load, annual inspections of switchgear and annual tests of the fire alarm and suppression systems.

Preventive maintenance works best for equipment whose failures are related to age or use, so that replacing a part before it wears out prevents the failure. Filters load with dust at a fairly predictable rate. Belts stretch and crack. Batteries lose capacity with age and temperature. Generators' fuel degrades, their starting batteries age and their seals dry out if they never run. For all of these, time or running hours is a reasonable guide to condition.

Its weakness is that it is blind. A preventive task is done whether or not the equipment needs it, so some work is wasted, and it does not catch failures that happen between visits. Worse, every maintenance visit is itself a risk. Opening equipment, removing parts and putting them back can introduce faults that were not there before: a connection not torqued, a valve left closed, a filter fitted backwards, a cover not replaced. These maintenance-induced failures are one of the most common causes of trouble in critical facilities, and one reason the industry has moved towards doing maintenance only when it adds value and doing it carefully when it does.

Maintenance engineers draw the failure rate of an item over its life as a bathtub curve: high at the start, when early failures appear, low and flat through a long useful life, and rising again at the end as parts wear out. The shape is worth keeping in mind on every job.

The bathtub curve explains both sides. Early in life, failures come from manufacturing defects and installation mistakes, which is why new equipment is commissioned and tested thoroughly. In the long flat middle, failures are mostly random, and time-based maintenance does little to prevent them. At the end of life, wear-out raises the failure rate again, and timely replacement or overhaul helps a great deal. A maintenance visit that disturbs equipment restarts a small piece of the early-failure part of the curve, which is why the work must be done to procedure and tested afterwards.

Predictive maintenance

Predictive maintenance measures the condition of equipment and acts only when the measurements show it deteriorating. It catches the failures that happen between preventive visits and avoids work that is not needed. Its tools are the subject of Lesson 5: infrared thermography for hot connections, ultrasound for electrical discharge and leaks, vibration analysis for rotating machines, oil analysis for engines and gearboxes, and ohmic measurements and trend analysis for batteries. The monitoring system is itself a predictive tool: a trend of rising temperature, falling flow or increasing current is often the first sign of trouble.

The idea that makes predictive maintenance work is that most failures do not happen suddenly. They develop over time, and there is a point at which the developing failure becomes detectable, long before the point at which the item actually fails. Maintenance engineers call these points P, for potential failure, and F, for functional failure, and the curve between them the P-F curve.

The P-F curve sets the rule for inspection intervals: to catch a failure before it happens, you must inspect more often than the time between P and F. If a loose busway joint takes about a year to go from a detectable temperature rise to failure, an annual infrared scan might just miss it; a scan every six months will catch it with time to spare. A failure that gives only a week's warning cannot be caught by a monthly inspection at all, and needs continuous monitoring or a different approach.

Reliability-centred maintenance

Reliability-centred maintenance, RCM, is a structured way of choosing the right strategy for each item. It asks, for each piece of equipment: what is it for, how can it fail, what happens when it fails, how much does that matter, and what task, if any, would prevent or detect the failure in time? The answers lead to a strategy: predictive where a measurable warning exists, preventive where failures are related to age, run to failure where failure is cheap and harmless, and redesign where no maintenance task can make the risk acceptable. Large operators apply RCM thinking to their programs, often with the help of engineers like those trained in the DCE school. Technicians do not usually run RCM studies, but they supply the most important input: what actually fails, how it fails and what it looked like beforehand.

Corrective maintenance and the backlog

Corrective work comes from three places: failures, inspection findings and predictive findings. Each finding becomes a corrective work order with a priority. At Stonebridge the priorities are emergency (work now, the load or safety is at risk), urgent (within a day, redundancy is lost or soon will be), routine (within the next planned window) and deferred (wait for the next scheduled outage of the equipment). The open corrective work orders form the backlog, and the planner watches its size and age. A backlog that grows steadily means the team is finding problems faster than it fixes them, and eventually the unfixed ones turn into failures.

Corrective work is not always a repair. Sometimes the right correction is to change the maintenance program: a filter that clogs before its quarterly change needs a monthly change, a valve type that keeps failing needs replacing across the campus, a task that keeps causing trouble needs a better procedure. Lesson 10 returns to this when it looks at corrective actions after incidents.

Counting failures

Morgan measures the program by what it prevents. One measure is the share of corrective work that was found by inspection or prediction rather than by failure: a high share means problems are being caught early. Another is the mean time between failures of each type of equipment, worked out from the event log. Jamal learns to calculate it on a sample of the log for the campus's CRAH units.

# MTBF, MTTR and availability for a group of units from an event log (hours since the start of the year)
events = [  # unit, failed at, restored at, how found
    ("CRAH-A03", 410.0, 414.5, "failure"),
    ("CRAH-A07", 1290.0, 1291.0, "inspection"),
    ("CRAH-A11", 2105.0, 2112.0, "failure"),
    ("CRAH-A07", 3380.0, 3381.5, "predictive"),
    ("CRAH-A02", 5022.0, 5024.0, "inspection"),
    ("CRAH-A09", 6650.0, 6668.0, "failure"),
]
units, period = 12, 8760.0
down = sum(r - f for _, f, r, _ in events)
up = units * period - down
mtbf = up / len(events)
mttr = down / len(events)
print(f"events {len(events)}, unit-hours down {down:.1f}")
print(f"MTBF per unit ~{mtbf:,.0f} h, MTTR {mttr:.1f} h, availability {mtbf / (mtbf + mttr) * 100:.3f} %")
early = sum(1 for e in events if e[3] != "failure")
print(f"found before failure: {early} of {len(events)} ({early / len(events) * 100:.0f} %)")

events 6, unit-hours down 34.0
MTBF per unit ~17,514 h, MTTR 5.7 h, availability 99.968 %
found before failure: 3 of 6 (50 %)

Half of the events were caught before they became failures, and the average repair took just over five hours. The longest repair, eighteen hours on CRAH-A09, pulls the average up; Morgan looks at it and finds the unit waited a day for a fan motor that was not stocked on site. Adding one spare motor to the store is a small corrective action that will shorten the next repair. Because the hall has spare CRAH capacity, none of these events affected the load, but each one removed some of the redundancy for its duration.

How the strategies fit together

No facility uses one strategy. Stonebridge's program is mostly preventive, with predictive methods layered on top for the equipment where they work well, run to failure for a small list of low-consequence items and a steady flow of corrective work from everything else. The trend over the years has been to move tasks from the preventive column to the predictive one where measurements make that possible, and to cut intrusive preventive tasks that add risk without adding value. A UPS inspection that involved opening every cabinet every quarter, for example, was replaced by quarterly external inspection, continuous monitoring of the UPS's own diagnostics and a full internal inspection once a year.