Root cause analysis: 5 whys, fishbone and fault trees

The breaker incident of Lesson 7 was contained in ten minutes and recovered in under an hour and a half, but it left a question. A short circuit on one branch circuit should have tripped that branch's own breaker and nothing else. Instead the main breaker of the whole panel opened, taking the A side away from 14 cabinets instead of one. Something in the protection did not work as designed. Finding out what, and why, is root cause analysis, usually shortened to RCA. This lesson follows the analysis of that incident and introduces the three tools Stonebridge uses most: the 5 whys, the fishbone diagram and the fault tree.

What a root cause is

Every incident has a chain of causes. At one end is the triggering event, the thing that happened just before the failure: here, a crushed cord shorting line to ground. At the other end are the deeper causes that allowed the trigger to turn into an incident, and that will allow the next trigger to do the same unless they are fixed. A root cause is a cause that, if removed, would prevent the incident from happening again, and that the organization can actually do something about. Most incidents have one or two root causes and several contributing factors, conditions that made the incident more likely or worse without causing it alone.

The distinction matters because fixing only the trigger rarely prevents the next incident. Replacing the crushed cord fixes this cord. It does nothing about the next crushed cord, or the next fault of any kind on any branch of a panel whose main breaker trips first. A good analysis keeps asking until it reaches causes that are about systems, procedures and conditions rather than about one part or one person.

Principles

Stonebridge's RCA procedure sets out a few principles, shared with most of the industry.

• Start from evidence, not opinion: the timeline, the logs, photographs, the failed parts, the records and what people saw. The containment habits of Lesson 7 exist to preserve this evidence.
• Be blameless. The aim is to understand why things happened, not who to punish. People who fear blame hide information, and hidden information means the real cause is never found. If a person made a mistake, the analysis asks why that action made sense to them at the time and what in the system allowed it.
• Look for causes the organization controls: procedures, training, design, maintenance, communication, supervision. A cause such as bad luck or human error is a place to start asking questions, not an answer.
• Involve the people who were there and the people who know the equipment. Technicians, vendors, engineers and, where their equipment is involved, tenants.
• Check the causes against the evidence: each proposed cause must explain what happened, and the chain must hold together.

Technicians are central to RCA even though engineers and managers lead it. Jamal's part in this analysis is to gather the evidence from the floor: the photographs taken during the incident, the trip unit's event log, the work order history of the panel, the cage's access records and his own notes from the shift log.

The 5 whys

The simplest tool is to ask why, repeatedly, until the answer is a cause that can be fixed. Five is a rule of thumb, not a limit; some chains need three questions, others seven. The 5 whys works well for straightforward incidents and as a first pass on complex ones. Its weakness is that it follows one chain, and real incidents usually have several, so it is often combined with a fishbone diagram.

Anjali leads the RCA meeting two days after the incident, with Dmitri, Roz, Jamal, the electrical contractor's engineer and Halvorsen Logistics' engineer. The evidence includes the trip unit's log, which the contractor's engineer has downloaded: the main breaker tripped on its instantaneous function at 14:11:37, with a fault current far below what the coordination study expected the main to see before the branch breaker cleared.

# Record a 5 whys chain and check each answer is backed by evidence
chain = [
    ("Why did 14 cabinets lose their A-side feed?",
     "The main breaker of RPP-A2-08 tripped.", "monitoring alarm 14:12; breaker found tripped"),
    ("Why did the main trip instead of the branch breaker?",
     "Its instantaneous trip setting was low enough to trip on a branch fault.", "trip unit log: instantaneous trip 14:11:37"),
    ("Why was the setting that low?",
     "The trip unit was replaced last year and left at its factory default settings.", "work order history; settings read from unit"),
    ("Why was it left at factory defaults?",
     "The replacement procedure had no step to apply and record the coordination study settings.", "the MOP used, revision 2"),
    ("Why was that step missing?",
     "The MOP template for trip unit work was written for like-for-like swaps, which keep their settings.", "MOP template review"),
]
for i, (q, a, ev) in enumerate(chain, 1):
    ok = "evidence: " + ev if ev else "NO EVIDENCE - check before accepting"
    print(f"{i}. {q}\n   {a}\n   ({ok})")
print()
print("candidate root cause:", chain[-1][1])

1. Why did 14 cabinets lose their A-side feed?
   The main breaker of RPP-A2-08 tripped.
   (evidence: monitoring alarm 14:12; breaker found tripped)
2. Why did the main trip instead of the branch breaker?
   Its instantaneous trip setting was low enough to trip on a branch fault.
   (evidence: trip unit log: instantaneous trip 14:11:37)
3. Why was the setting that low?
   The trip unit was replaced last year and left at its factory default settings.
   (evidence: work order history; settings read from unit)
4. Why was it left at factory defaults?
   The replacement procedure had no step to apply and record the coordination study settings.
   (evidence: the MOP used, revision 2)
5. Why was that step missing?
   The MOP template for trip unit work was written for like-for-like swaps, which keep their settings.
   (evidence: MOP template review)

candidate root cause: The MOP template for trip unit work was written for like-for-like swaps, which keep their settings.

The chain reaches a cause the organization controls: a procedure template that did not fit the job it was used for. A new trip unit, unlike the old one, did not keep its settings when installed, and nothing in the procedure caught that. But the chain stops at one line of causes. It does not ask why the cord was crushed, why the device in A-26-05 had only one cord, or why nobody noticed the wrong settings for a year. For those, the team draws a fishbone.

The fishbone diagram

A fishbone diagram, also called an Ishikawa or cause-and-effect diagram, puts the incident at the head of a fish and groups possible causes along bones by category. The categories prompt the team to look in places it might otherwise miss. Stonebridge uses six: people, procedures, equipment, materials, environment and management.

The meeting fills in each bone. Under procedures: the trip unit MOP had no settings step, and Stonebridge's acceptance of tenant power strips did not include a check of cord routing. Under equipment: the new trip unit model ships with conservative defaults, unlike the old one. Under people: the electrician who replaced the unit followed the MOP exactly as written, and had no reason to expect the settings to change. Under materials: the customer's power strip had a stiff inlet cord that would not bend into the cable manager. Under environment: the cabinet door closes tightly against cords routed at the front. Under management: no audit of protective device settings is done after protection work, so the wrong settings went unnoticed for a year, and the single-corded device's exposure was known but accepted without a transfer switch.

# Count fishbone causes by category, and mark those the team agreed are root or contributing
causes = [
    ("procedures", "trip unit MOP has no settings step", "root"),
    ("procedures", "tenant power strip acceptance has no routing check", "contributing"),
    ("equipment", "new trip unit model ships with low default settings", "contributing"),
    ("people", "electrician followed the MOP as written", "none"),
    ("materials", "stiff inlet cord on customer strip", "contributing"),
    ("environment", "cabinet door closes against front-routed cords", "contributing"),
    ("management", "no settings audit after protection work", "root"),
    ("management", "single-corded device accepted without transfer switch", "contributing"),
]
by_cat = {}
for cat, text, kind in causes:
    by_cat.setdefault(cat, []).append(kind)
for cat, kinds in by_cat.items():
    print(f"{cat:12} {len(kinds)} cause(s): " + ", ".join(k for k in kinds))
roots = [t for _, t, k in causes if k == "root"]
contrib = [t for _, t, k in causes if k == "contributing"]
print(f"\nroot causes ({len(roots)}):", "; ".join(roots))
print(f"contributing factors: {len(contrib)}")

procedures   2 cause(s): root, contributing
equipment    1 cause(s): contributing
people       1 cause(s): none
materials    1 cause(s): contributing
environment  1 cause(s): contributing
management   2 cause(s): root, contributing

root causes (2): trip unit MOP has no settings step; no settings audit after protection work
contributing factors: 5

The team agrees on two root causes, both about systems rather than people: the procedure that missed the settings, and the absence of any check afterwards that would have caught the mistake. Five contributing factors made the incident possible or worse. The people bone shows something worth noticing: the person closest to the mistake did nothing wrong. He followed the procedure he was given. A blame-seeking analysis would have stopped at him, and the next trip unit replacement would have gone the same way.

Fault trees

A fault tree works in the opposite direction. It starts from an undesired top event, such as loss of power to a cabinet, and works down through the combinations of failures that could cause it, joined by logic gates. An OR gate means any one of the events below causes the event above; an AND gate means all of them must happen together. Fault trees are drawn by engineers when designing systems and studying risk, but technicians meet them in incident reports and maintenance planning, and they explain clearly why redundancy works.

The tree shows the arithmetic of redundancy. If the chance of losing either feed in a year is small, the chance of losing both together, for a dual-corded cabinet, is the product of two small numbers, which is very small indeed, provided the two sides fail independently. The tree also shows the conditions that break this: shared causes that hit both sides at once, and times when one side is already lost, such as during maintenance or after an incident like this one.

# Fault tree for 'cabinet loses all power': AND of the two feeds, each an OR of three causes (yearly probabilities)
def OR(*p):
    q = 1.0
    for x in p:
        q *= (1 - x)
    return 1 - q

def AND(*p):
    q = 1.0
    for x in p:
        q *= x
    return q

panel, upstream, ups = 0.004, 0.002, 0.001       # illustrative yearly probabilities for one side
side = OR(panel, upstream, ups)
print(f"one feed lost in a year:            {side:.4f}")
print(f"both feeds lost (independent):      {AND(side, side):.6f}")
# A panel whose main trips on any branch fault is more likely to trip
side_bad = OR(0.04, upstream, ups)
print(f"both feeds, one panel miscoordinated: {AND(side_bad, side):.6f}")
# A single-corded device depends on one feed only
print(f"single-corded device on that panel:   {side_bad:.4f}")

one feed lost in a year:            0.0070
both feeds lost (independent):      0.000049
both feeds, one panel miscoordinated: 0.000300
single-corded device on that panel:   0.0429

The numbers are illustrative but the pattern is the lesson. A miscoordinated panel makes its own side about six times more likely to fail, and that multiplies the risk to every cabinet it feeds. For the single-corded device, which has no AND gate to protect it, the panel's weakness passes straight through. This is why the analysis treated the single-corded device without a transfer switch as a contributing factor, not as the customer's private business.

Writing it up

The RCA ends in a written report, the basis of the final incident report to the tenant described in Lesson 8. Stonebridge's format is short: a summary, the timeline, the impact, the triggering event, the root causes and contributing factors with their evidence, and the corrective actions with owners and dates. It describes what happened in neutral language and names roles, not people. The report for the breaker incident ran to four pages. Its key sentence read: a branch circuit fault caused the panel main breaker to trip because the main's trip unit had been left at factory settings after replacement; the replacement procedure did not include applying the coordination settings, and no check of settings was made after the work.

Halvorsen's engineer asked one question when the draft was shared: was his company's damaged power strip the cause? The honest answer, which the report gives, is that the strip's damaged cord was the trigger, but any branch fault would have tripped the main. The root causes were Stonebridge's, and so were the main corrective actions.