Learning from failure: investigation and root cause analysis

Engineering advances by learning from failure. The Quebec Bridge collapses of 1907 and 1916 helped shape engineering codes of ethics; the Tacoma Narrows Bridge taught structural engineers about aerodynamics; boiler explosions in the nineteenth century produced the first pressure vessel codes; and every major aircraft accident has made flying safer. The lessons are learned only if failures are investigated honestly and thoroughly, their real causes found, and the findings turned into changes in designs, codes, procedures and organizations. This lesson covers how engineering failures are investigated: preserving evidence, building a timeline, finding root causes with tools such as the five whys, cause-and-effect diagrams and fault trees, understanding the human and organizational factors behind most accidents, and making sure lessons are acted on. Four cases, from a bridge that twisted itself apart to a runaway train, show the methods at work.

Why investigate?

After a failure, the instinctive questions are "who is to blame?" and "how do we get running again?" Investigation asks a different question: "why did this happen, and how do we stop it from happening again, here and elsewhere?" Blame tends to stop at the last person who touched the system, an operator or a technician, and leaves the conditions that set them up to fail in place. Good investigations look past the immediate cause to the design decisions, procedures, training, maintenance practices, incentives and oversight that allowed it.

Investigation is not only for disasters. Near misses, events that could have caused harm but did not, have the same causes as accidents and far more of them occur. Studying them is the cheapest way to prevent accidents. Many organizations track near misses, minor incidents and serious accidents together, recognizing that a rising count of small events signals a growing chance of a large one.

In Canada, the Transportation Safety Board (TSB) investigates air, marine, rail and pipeline occurrences, independently of the regulators, to advance safety rather than to assign blame or liability. Provincial occupational health and safety agencies investigate serious workplace injuries, coroners investigate deaths, and engineering regulators investigate the conduct of their members.

Securing the evidence and building the timeline

Evidence disappears quickly: debris is cleared, parts are repaired, data are overwritten and memories fade or merge with what people later hear. Investigators therefore secure the scene, photograph everything before it is moved, tag and preserve failed components, download data from control systems, recorders and sensors, collect documents such as drawings, maintenance records, permits and logs, and interview witnesses separately and early.

A detailed timeline is the backbone of the investigation. It lists every relevant event, condition and decision in order, with its source, often going back months or years before the failure: when the design was changed, when a warning was given, when an inspection was skipped. Gaps and contradictions in the timeline show where more evidence is needed. Failed parts are examined in laboratories: fracture surfaces reveal whether a part broke in a single overload, by fatigue (with characteristic beach marks), by corrosion or by brittle fracture, and chemical analysis confirms whether the material met its specification.

Immediate causes and root causes

Investigators distinguish levels of cause. The immediate cause is the event that directly produced the failure: a bolt fractured, a valve was left open. Contributing causes are conditions that made the failure possible or worse. Root causes are the underlying, usually organizational, reasons: the deepest causes that the organization can control and whose correction would prevent this and similar failures. A useful test for a root cause is that fixing it would prevent a whole class of failures, not just the one that happened.

The five whys is the simplest way to dig down: ask why the failure happened, then why that cause occurred, and so on, typically about five times, until you reach something the organization can change.

The five whys is quick but has limits: real failures usually have several causes, and the method follows only one chain. Cause-and-effect diagrams, also called fishbone or Ishikawa diagrams, organize possible causes into categories such as people, methods, machines, materials, measurements and environment, prompting the team to consider all of them.

Fault trees

For complex systems, engineers use fault tree analysis, which works backward from an undesired top event through logic gates to the basic events that could cause it. An OR gate means any one of its inputs causes the output; an AND gate means all inputs must occur together. Fault trees are used both after a failure, to show how it happened, and before, in design, to find weaknesses and estimate how often a serious event might occur. They complement the failure modes and effects analysis of Lesson 6, which works forward from each component failure.

For independent events with probabilities p₁, p₂, …, an AND gate gives the product p₁p₂…, and an OR gate gives 1 − (1 − p₁)(1 − p₂)…, which is close to the sum when the probabilities are small.

def OR(*p):                       # at least one independent event occurs
    q = 1.0
    for x in p:
        q *= 1 - x
    return 1 - q

def AND(*p):                      # all independent events occur
    r = 1.0
    for x in p:
        r *= x
    return r

# Basic events (probability per year, or per demand for the relief valve)
valve_fails_open, operator_error, outlet_blocked = 0.10, 0.05, 0.02
rv_stuck, rv_isolated, rv_undersized = 0.010, 0.005, 0.001

pressure_rises = OR(valve_fails_open, operator_error, outlet_blocked)
relief_fails = OR(rv_stuck, rv_isolated, rv_undersized)
top = AND(pressure_rises, relief_fails)
print(f"Overpressure demand  {pressure_rises:.4f} per year")
print(f"Relief system fails  {relief_fails:.4f} per demand")
print(f"Vessel rupture       {top:.2e} per year (1 in {1 / top:.0f} years)")

# Option 1: lock the block valve open, so the relief valve cannot be isolated
top1 = AND(pressure_rises, OR(rv_stuck, 0.0005, rv_undersized))
# Option 2: add a second, independent relief valve
two = AND(relief_fails, relief_fails)
top2 = AND(pressure_rises, two)
# Option 2 with common-cause failure: 10 percent of failures hit both valves
beta = 0.10
two_cc = AND((1 - beta) * relief_fails, (1 - beta) * relief_fails) + beta * relief_fails
top3 = AND(pressure_rises, two_cc)
for name, p in [("Lock block valve open", top1), ("Second relief valve", top2), ("Second valve, common cause", top3)]:
    print(f"{name:28s} {p:.2e} per year (1 in {1 / p:,.0f} years)")

Overpressure demand  0.1621 per year
Relief system fails  0.0159 per demand
Vessel rupture       2.58e-03 per year (1 in 387 years)
Lock block valve open        1.86e-03 per year (1 in 537 years)
Second relief valve          4.12e-05 per year (1 in 24,295 years)
Second valve, common cause   2.92e-04 per year (1 in 3,429 years)

The calculation shows why fault trees are valuable. The base design ruptures about once in 400 years, which sounds rare but is unacceptable for a hazard that could kill workers; across a company with a hundred such vessels, it means a rupture every four years. Locking the block valve open helps a little. A second relief valve helps enormously in theory, but only if the two valves fail independently. If one in ten failures affects both valves at once, for example because both are blocked by the same deposit or were tested by the same faulty procedure, the benefit shrinks by a factor of seven. Common-cause failures defeat redundancy, which is why critical systems use diverse designs, separate locations and independent maintenance (Lesson 6).

Human and organizational factors

Most accident investigations find human error somewhere in the chain. Treating "operator error" as the conclusion is a mistake, because people make predictable errors and well-designed systems anticipate them. Human factors engineering studies how equipment, procedures and environments can be designed so that errors are less likely and less harmful: clear displays, controls that cannot easily be confused, alarms prioritized so that the important ones are not lost in a flood, procedures that match how work is actually done, and workloads and shift patterns that do not exhaust people.

The psychologist James Reason described accidents with the Swiss cheese model. Each layer of defence (design, maintenance, procedures, training, supervision, alarms) is like a slice of cheese with holes, some from active failures by people at the sharp end and many from latent conditions created long before by decisions in design, management and regulation. Usually the holes do not line up and an error is caught. An accident happens when holes in every layer momentarily align. Investigation looks for all the holes, especially the latent ones, because they will line up again with different people.

Organizations that learn well practise a just culture: honest mistakes are treated as learning opportunities and reported freely, while reckless conduct and wilful violations are still held to account. Aviation's voluntary, confidential reporting systems, in which pilots and controllers report their own errors without fear of punishment, are a major reason commercial flying is so safe.

Case: the Tacoma Narrows Bridge, 1940

The Tacoma Narrows Bridge in Washington State, then the third-longest suspension span in the world, was unusually slender, with a shallow, solid-sided deck girder chosen to save money and look elegant. Even during construction it moved up and down in light winds, earning the nickname "Galloping Gertie". Four months after opening, on November 7, 1940, in a steady wind of about 64 km/h, the deck began to twist violently and tore itself apart; the only death was a dog left in a car. Film of the collapse is still shown to engineering students. The investigation found that the deck's shape and flexibility allowed aeroelastic flutter, a self-exciting interaction in which the twisting motion changed the wind forces in a way that fed more energy into the motion. Designers had extrapolated successful experience with stiffer bridges without recognizing a new failure mode. Since then, long-span bridges have been tested in wind tunnels and designed with stiff, aerodynamically shaped decks.

Case: the Columbia shuttle, 2003

During the launch of the Space Shuttle Columbia in January 2003, a piece of foam insulation broke off the external tank and struck the left wing's leading edge. Sixteen days later, on re-entry, hot gas entered the damaged wing and the shuttle broke apart, killing all seven astronauts. Foam strikes had occurred on many earlier flights; like the O-ring erosion before Challenger (Lesson 5), they had come to be treated as a maintenance issue rather than a safety threat. During the mission, engineers asked for satellite images of the wing, but managers declined, partly because they believed nothing could be done. The Columbia Accident Investigation Board concluded that NASA's organizational culture was as much a cause as the foam, finding that the lessons of Challenger had not been institutionalized, and recommended an independent technical authority to separate safety decisions from schedule pressure.

Case: Lac-Mégantic, 2013

In the early hours of July 6, 2013, an unattended train of 72 tank cars of crude oil, parked on a slope above Lac-Mégantic, Quebec, ran away, derailed in the town centre and exploded, killing 47 people and destroying much of the downtown. The train had been left by its single operator with too few hand brakes applied, relying partly on air brakes on the locomotive; when a fire on the locomotive was put out and the engine shut down, the air pressure leaked away and the train began to roll. The Transportation Safety Board identified 18 causes and contributing factors, including the railway's weak safety culture and training, its practice of single-person crews, the lack of a physical defence against runaway trains, the vulnerability of the older tank cars, the misclassification of the volatile crude, and ineffective oversight by Transport Canada. The reforms that followed included rules on securing unattended trains, tougher tank car standards and changes to rail safety oversight.

Turning findings into action

An investigation is worthless if its recommendations are filed and forgotten. Good practice assigns each recommendation to a named owner with a deadline, tracks progress, and verifies that the change actually works. Findings should be shared widely, inside the organization and across the industry, because the same latent conditions usually exist elsewhere. Codes and standards are revised after major failures; professional regulators publish discipline decisions so that other engineers can learn; and many companies keep lessons-learned databases and require new projects to review them.

For an individual engineer, the habits are the same at a smaller scale: when something goes wrong in your own work, find out why, fix the cause rather than the symptom, tell others who could make the same mistake, and treat your own errors as information rather than embarrassment. Lesson 8 turns to communicating such findings clearly, and Lesson 12 asks you to apply these methods in a course project.