Learning from failure: investigation and root cause analysis

Engineering advances by learning from failure. The Quebec Bridge collapses of 1907 and 1916 helped shape engineering codes of ethics; the Tacoma Narrows Bridge taught structural engineers about aerodynamics; boiler explosions in the nineteenth century produced the first pressure vessel codes; and every major aircraft accident has made flying safer. The lessons are learned only if failures are investigated honestly and thoroughly, their real causes found, and the findings turned into changes in designs, codes, procedures and organizations. This lesson covers how engineering failures are investigated: preserving evidence, building a timeline, finding root causes with tools such as the five whys, cause-and-effect diagrams and fault trees, understanding the human and organizational factors behind most accidents, and making sure lessons are acted on. Four cases, from a bridge that twisted itself apart to a runaway train, show the methods at work.

Why investigate?

After a failure, the instinctive questions are "who is to blame?" and "how do we get running again?" Investigation asks a different question: "why did this happen, and how do we stop it from happening again, here and elsewhere?" Blame tends to stop at the last person who touched the system, an operator or a technician, and leaves the conditions that set them up to fail in place. Good investigations look past the immediate cause to the design decisions, procedures, training, maintenance practices, incentives and oversight that allowed it.

Investigation is not only for disasters. Near misses, events that could have caused harm but did not, have the same causes as accidents and far more of them occur. Studying them is the cheapest way to prevent accidents. Many organizations track near misses, minor incidents and serious accidents together, recognizing that a rising count of small events signals a growing chance of a large one.

In the United States, the National Transportation Safety Board (NTSB) investigates aviation, rail, highway, marine and pipeline accidents, and the Chemical Safety and Hazard Investigation Board (CSB) investigates chemical accidents, both independently of the regulators and without assigning blame or liability. OSHA investigates workplace fatalities and serious injuries, and state licensing boards investigate the conduct of engineers.

Securing the evidence and building the timeline

Evidence disappears quickly: debris is cleared, parts are repaired, data are overwritten and memories fade or merge with what people later hear. Investigators therefore secure the scene, photograph everything before it is moved, tag and preserve failed components, download data from control systems, recorders and sensors, collect documents such as drawings, maintenance records, permits and logs, and interview witnesses separately and early.

A detailed timeline is the backbone of the investigation. It lists every relevant event, condition and decision in order, with its source, often going back months or years before the failure: when the design was changed, when a warning was given, when an inspection was skipped. Gaps and contradictions in the timeline show where more evidence is needed. Failed parts are examined in laboratories: fracture surfaces reveal whether a part broke in a single overload, by fatigue (with characteristic beach marks), by corrosion or by brittle fracture, and chemical analysis confirms whether the material met its specification.

Immediate causes and root causes

Investigators distinguish levels of cause. The immediate cause is the event that directly produced the failure: a bolt fractured, a valve was left open. Contributing causes are conditions that made the failure possible or worse. Root causes are the underlying, usually organizational, reasons: the deepest causes that the organization can control and whose correction would prevent this and similar failures. A useful test for a root cause is that fixing it would prevent a whole class of failures, not just the one that happened.

The five whys is the simplest way to dig down: ask why the failure happened, then why that cause occurred, and so on, typically about five times, until you reach something the organization can change.

The five whys is quick but has limits: real failures usually have several causes, and the method follows only one chain. Cause-and-effect diagrams, also called fishbone or Ishikawa diagrams, organize possible causes into categories such as people, methods, machines, materials, measurements and environment, prompting the team to consider all of them.

Fault trees

For complex systems, engineers use fault tree analysis, which works backward from an undesired top event through logic gates to the basic events that could cause it. An OR gate means any one of its inputs causes the output; an AND gate means all inputs must occur together. Fault trees are used both after a failure, to show how it happened, and before, in design, to find weaknesses and estimate how often a serious event might occur. They complement the failure modes and effects analysis of Lesson 6, which works forward from each component failure.

For independent events with probabilities p₁, p₂, …, an AND gate gives the product p₁p₂…, and an OR gate gives 1 − (1 − p₁)(1 − p₂)…, which is close to the sum when the probabilities are small.

def OR(*p):                       # at least one independent event occurs
    q = 1.0
    for x in p:
        q *= 1 - x
    return 1 - q

def AND(*p):                      # all independent events occur
    r = 1.0
    for x in p:
        r *= x
    return r

# Basic events (probability per year, or per demand for the relief valve)
valve_fails_open, operator_error, outlet_blocked = 0.10, 0.05, 0.02
rv_stuck, rv_isolated, rv_undersized = 0.010, 0.005, 0.001

pressure_rises = OR(valve_fails_open, operator_error, outlet_blocked)
relief_fails = OR(rv_stuck, rv_isolated, rv_undersized)
top = AND(pressure_rises, relief_fails)
print(f"Overpressure demand  {pressure_rises:.4f} per year")
print(f"Relief system fails  {relief_fails:.4f} per demand")
print(f"Vessel rupture       {top:.2e} per year (1 in {1 / top:.0f} years)")

# Option 1: lock the block valve open, so the relief valve cannot be isolated
top1 = AND(pressure_rises, OR(rv_stuck, 0.0005, rv_undersized))
# Option 2: add a second, independent relief valve
two = AND(relief_fails, relief_fails)
top2 = AND(pressure_rises, two)
# Option 2 with common-cause failure: 10 percent of failures hit both valves
beta = 0.10
two_cc = AND((1 - beta) * relief_fails, (1 - beta) * relief_fails) + beta * relief_fails
top3 = AND(pressure_rises, two_cc)
for name, p in [("Lock block valve open", top1), ("Second relief valve", top2), ("Second valve, common cause", top3)]:
    print(f"{name:28s} {p:.2e} per year (1 in {1 / p:,.0f} years)")

Overpressure demand  0.1621 per year
Relief system fails  0.0159 per demand
Vessel rupture       2.58e-03 per year (1 in 387 years)
Lock block valve open        1.86e-03 per year (1 in 537 years)
Second relief valve          4.12e-05 per year (1 in 24,295 years)
Second valve, common cause   2.92e-04 per year (1 in 3,429 years)

The calculation shows why fault trees are valuable. The base design ruptures about once in 400 years, which sounds rare but is unacceptable for a hazard that could kill workers; across a company with a hundred such vessels, it means a rupture every four years. Locking the block valve open helps a little. A second relief valve helps enormously in theory, but only if the two valves fail independently. If one in ten failures affects both valves at once, for example because both are blocked by the same deposit or were tested by the same faulty procedure, the benefit shrinks by a factor of seven. Common-cause failures defeat redundancy, which is why critical systems use diverse designs, separate locations and independent maintenance (Lesson 6).

Human and organizational factors

Most accident investigations find human error somewhere in the chain. Treating "operator error" as the conclusion is a mistake, because people make predictable errors and well-designed systems anticipate them. Human factors engineering studies how equipment, procedures and environments can be designed so that errors are less likely and less harmful: clear displays, controls that cannot easily be confused, alarms prioritized so that the important ones are not lost in a flood, procedures that match how work is actually done, and workloads and shift patterns that do not exhaust people.

The psychologist James Reason described accidents with the Swiss cheese model. Each layer of defense (design, maintenance, procedures, training, supervision, alarms) is like a slice of cheese with holes, some from active failures by people at the sharp end and many from latent conditions created long before by decisions in design, management and regulation. Usually the holes do not line up and an error is caught. An accident happens when holes in every layer momentarily align. Investigation looks for all the holes, especially the latent ones, because they will line up again with different people.

Organizations that learn well practice a just culture: honest mistakes are treated as learning opportunities and reported freely, while reckless conduct and wilful violations are still held to account. Aviation's voluntary, confidential reporting systems, in which pilots and controllers report their own errors without fear of punishment, are a major reason commercial flying is so safe.

Case: the Tacoma Narrows Bridge, 1940

The Tacoma Narrows Bridge in Washington State, then the third-longest suspension span in the world, was unusually slender, with a shallow, solid-sided deck girder chosen to save money and look elegant. Even during construction it moved up and down in light winds, earning the nickname "Galloping Gertie". Four months after opening, on November 7, 1940, in a steady wind of about 64 km/h, the deck began to twist violently and tore itself apart; the only death was a dog left in a car. Film of the collapse is still shown to engineering students. The investigation found that the deck's shape and flexibility allowed aeroelastic flutter, a self-exciting interaction in which the twisting motion changed the wind forces in a way that fed more energy into the motion. Designers had extrapolated successful experience with stiffer bridges without recognizing a new failure mode. Since then, long-span bridges have been tested in wind tunnels and designed with stiff, aerodynamically shaped decks.

Case: the Columbia shuttle, 2003

During the launch of the Space Shuttle Columbia in January 2003, a piece of foam insulation broke off the external tank and struck the left wing's leading edge. Sixteen days later, on re-entry, hot gas entered the damaged wing and the shuttle broke apart, killing all seven astronauts. Foam strikes had occurred on many earlier flights; like the O-ring erosion before Challenger (Lesson 5), they had come to be treated as a maintenance issue rather than a safety threat. During the mission, engineers asked for satellite images of the wing, but managers declined, partly because they believed nothing could be done. The Columbia Accident Investigation Board concluded that NASA's organizational culture was as much a cause as the foam, finding that the lessons of Challenger had not been institutionalized, and recommended an independent technical authority to separate safety decisions from schedule pressure.

Case: East Palestine, 2023

On February 3, 2023, a Norfolk Southern freight train derailed in East Palestine, Ohio, after a wheel bearing on one car overheated and failed; cars carrying vinyl chloride and other hazardous materials were involved in a fire. Fearing that tank cars of vinyl chloride would explode, officials vented and burned their contents three days later, sending a huge plume of smoke over the area and forcing evacuations. The NTSB found that the trackside detectors that monitor bearing temperatures were too far apart to catch the rapidly overheating bearing in time and that the alerting thresholds and response procedures were inadequate. It also concluded that the vent and burn had not been necessary, because evidence from the tank car manufacturer and others showed that the dangerous reaction feared was not occurring, and that this information had not reached the decision-makers. Recommendations addressed detector spacing and standards, tank car design and how emergency decisions are made.

Turning findings into action

An investigation is worthless if its recommendations are filed and forgotten. Good practice assigns each recommendation to a named owner with a deadline, tracks progress, and verifies that the change actually works. Findings should be shared widely, inside the organization and across the industry, because the same latent conditions usually exist elsewhere. Codes and standards are revised after major failures; professional regulators publish discipline decisions so that other engineers can learn; and many companies keep lessons-learned databases and require new projects to review them.

For an individual engineer, the habits are the same at a smaller scale: when something goes wrong in your own work, find out why, fix the cause rather than the symptom, tell others who could make the same mistake, and treat your own errors as information rather than embarrassment. Lesson 8 turns to communicating such findings clearly, and Lesson 12 asks you to apply these methods in a course project.