Risk matrices: design, calibration and limits

The risk matrix is the most widely used risk tool in the world. Almost every company has one, usually a grid of 3 by 3, 4 by 4, 5 by 5 or 6 by 6 cells with likelihood on one axis, severity on the other and cells coloured green, yellow and red. HSE-103 showed how to use one. This lesson is about how to design one properly, how to calibrate its categories so that different teams get the same answer, and why risk researchers warn that poorly designed matrices can give answers that are worse than useless.

A matrix does three jobs. It ranks risks so that effort goes to the worst first. It links each risk level to a required response, such as stop work, senior management approval or routine control. And it gives people a shared language. It does these jobs well only if its categories are defined clearly, its scales match the kinds of events the organization faces, and its users understand its limits.

Defining the severity scale

Severity categories should describe outcomes in concrete terms, and most organizations use several consequence types side by side: people, environment, asset damage and business interruption, reputation and sometimes legal or community impact. The rule is to rate each scenario on every applicable column and use the worst. Each category should be clearly separated from its neighbours. Phrases such as "serious" or "significant" invite argument; descriptions such as "lost time injury" or "permanent disability or single fatality" do not.

Defining the likelihood scale

Likelihood scales are where most matrices go wrong. Words such as "likely" and "rare" mean very different things to different people: in studies, people asked to assign a probability to "likely" give answers from about 40 to 90 per cent. Good matrices attach numbers to each category, usually as frequency ranges per year, and make each category about ten times the one below. Many also give a plain-language anchor, such as "has happened at this site" or "has happened in the industry", to help people who are not comfortable with numbers.

With both scales logarithmic, each cell represents a band of risk. Moving one step up in severity or likelihood multiplies the risk by about ten. That is why cell colours should run along diagonals: a 4-A (major, rare) and a 2-C (moderate, possible) are roughly comparable if the scales are designed this way. It is also why multiplying the level numbers, such as 4 times 1, has no real meaning. If the scales are logarithmic, the levels should be added, not multiplied, to rank risk consistently, or better, the cells should simply be assigned a risk level directly.

Linking cells to responses

A matrix is only useful if each risk level triggers a defined response. A typical scheme has four levels. Low risk: manage with routine procedures. Medium risk: reduce as low as reasonably practicable, with supervisor approval. High risk: senior manager approval and an action plan with dates. Extreme or intolerable: stop or do not start the work until the risk is reduced. Some companies also have a rule that any scenario with a catastrophic severity is reviewed by a more detailed method, such as LOPA, regardless of its cell, because likelihood estimates for rare events are so uncertain.

Calibration

Calibration means making sure the matrix gives consistent answers and that its risk levels match the organization's risk criteria. One method is to place known reference events on the matrix. For example, if the company's tolerable risk of a single worker fatality is 1 in 10,000 per year per scenario, the cell at major severity and rare likelihood should be the boundary between medium and high. Another method is a calibration exercise: give several teams the same five scenarios and compare their ratings. Large differences point to unclear definitions or a need for training.

Known limitations

Risk analyst Louis Anthony Cox published a widely cited paper in 2008, "What's wrong with risk matrices?", showing several mathematical problems. Range compression: a matrix squeezes a huge range of real risks into a few cells, so risks that differ by a factor of 100 can share a cell. Ties and reversals: risks in different cells can be assigned the wrong order, especially when consequences and likelihood are negatively correlated, as they often are (big events are rare). Subjective inputs: the categories are judged, not measured, and people are poor at estimating rare frequencies. Ambiguous inputs and outputs: a scenario can have several outcomes with different severities, and the matrix forces the team to pick one.

Other practical problems are common. Teams game the matrix by lowering the likelihood until the colour changes to an acceptable one. Matrices are copied from other companies without checking that the categories fit. Severity is rated for the most likely outcome rather than the worst credible one, or for the worst conceivable rather than credible one, inconsistently. And the matrix encourages people to treat a number such as "12" as if it were a measurement rather than a label.

None of this means matrices should be abandoned. For screening and ranking large numbers of ordinary risks, nothing is faster, and IEC 31010 lists the consequence-likelihood matrix as a valid technique. The answer is to design matrices carefully, define the categories in numbers, use them for screening rather than final decisions on high-consequence risks, and escalate scenarios with catastrophic potential to more rigorous methods. ANSI/ASSP Z590.3 and CSA Z1002 both include example matrices and note that they should be adapted to the organization.

Multiple consequence types and outcomes

Most scenarios have more than one possible outcome. A small gas leak may usually disperse harmlessly, sometimes cause a flash fire and rarely cause an explosion. Rating only the most likely outcome hides the catastrophic tail; rating only the worst conceivable outcome makes every scenario red. A practical approach is to rate the worst credible outcome and, where the team is unsure, rate two or three outcomes separately, each with its own likelihood. The highest resulting risk level governs. This takes more time but avoids the ambiguity Cox described.