Audit sampling, evidence and findings

The first lockout audit at the Truro plant took two days and one night. The team, an auditor from Dartmouth named Carla Benoit and one of Aisha Rahman's quality auditors, Pierre Gagnon, could not check every machine, every lockout and every training record. The plant had 118 machine-specific lockout procedures, more than 2,900 lockouts a year recorded on permits, and 340 workers authorized to lock out. Like every audit, theirs rested on samples. This lesson follows the audit through its evidence, its sampling and its findings, and explains how auditors decide how much evidence is enough.

Kinds of evidence

Audit evidence comes in four main forms, and good auditors use all of them:

• Documents: the procedures, plans and forms that say how things should be done, such as the lockout program and the machine-specific procedures.
• Records: evidence that things were done, such as completed lockout permits, training records, verification checklists and periodic review sign-offs.
• Interviews: conversations with the people who do the work, supervise it or manage it, about what they actually do and why.
• Observation: watching the work being done and the conditions in which it is done.

Each form has weaknesses. Documents describe intentions, not practice. Records can be completed without the work being done, or the work done without the record. People in interviews may say what they think the auditor wants to hear. And people who know they are being watched often work differently. The defence against these weaknesses is triangulation: looking for the same fact in more than one form of evidence. A finding supported by a record, an interview and an observation is far stronger than one resting on any single source.

At Truro, triangulation produced the audit's most important finding. The lockout permits for the night sanitation shift were complete and correctly signed. In interviews, two sanitation workers said that on busy nights the crew lead applied one lock for the whole crew and everyone worked under it. On the night of the audit, Carla observed exactly that on the spiral freezer: one lock, one permit, four workers. The records alone would have shown full compliance.

Judgemental and statistical samples

Auditors choose samples in two ways. A judgemental sample is chosen by the auditor's judgement, to look where problems are most likely or most serious: the newest machines, the ones that have been modified, the night shift, the agency workers. A statistical sample is chosen at random, and its size is set so that the auditor can draw a conclusion about the whole population with a stated level of confidence.

Judgemental samples are efficient at finding problems. If an auditor checks 20 lockout procedures chosen because they relate to modified equipment, and three of them miss an isolation point, the audit has shown that the procedure review process is not working. One failure is enough to prove a nonconformity. But a judgemental sample cannot show that a population is in good order, because its items were not chosen to represent the population. To say with confidence that 'no more than 5 percent of procedures are wrong', the auditor needs a statistical sample.

How large a statistical sample?

For pass-or-fail items such as procedures, records or verifications, auditors often use attribute sampling. The auditor chooses a confidence level, such as 95 percent, and a tolerable failure rate, the highest rate at which the population would still be acceptable, such as 5 percent. The sample must be large enough that, if the true failure rate were as high as the tolerable rate, a sample with no failures would be unlikely. For a large population and no failures allowed, the sample size \(n\) satisfies \((1-p)^n \le 1 - C\), where \(p\) is the tolerable rate and \(C\) the confidence, which gives:

For small populations, fewer items are needed, because sampling without replacement removes uncertainty as the sample becomes a larger share of the population. Allowing one failure in the sample raises the sample size sharply. The script below calculates both cases.

# Audit sample sizes for attribute testing (pass or fail items)
import math

def n_infinite(conf, tolerable, expected_fails=0):
    # smallest n where seeing <= expected_fails failures is unlikely (prob <= 1 - conf)
    # if the true failure rate were as high as the tolerable rate
    n = 1
    while True:
        p = sum(math.comb(n, k) * tolerable**k * (1 - tolerable)**(n - k)
                for k in range(expected_fails + 1))
        if p <= 1 - conf:
            return n
        n += 1

def n_finite(conf, tolerable, N):
    # exact for a finite population with zero failures allowed (hypergeometric)
    bad = math.ceil(tolerable * N)
    for n in range(1, N + 1):
        p_zero = math.comb(N - bad, n) / math.comb(N, n)
        if p_zero <= 1 - conf:
            return n
    return N

print("zero failures allowed, 95 % confidence:")
for tol in (0.10, 0.05, 0.02):
    print(f"  tolerable rate {tol:4.0%}: sample {n_infinite(0.95, tol)}")
print("one failure allowed, 95 % confidence, tolerable 5 %:", n_infinite(0.95, 0.05, 1))
for N in (118, 340, 2900):
    print(f"population {N:>5}: zero-failure sample at 95 %, 5 % tolerable = {n_finite(0.95, 0.05, N)}")

zero failures allowed, 95 % confidence:
  tolerable rate  10%: sample 29
  tolerable rate   5%: sample 59
  tolerable rate   2%: sample 149
one failure allowed, 95 % confidence, tolerable 5 %: 93
population   118: zero-failure sample at 95 %, 5 % tolerable = 46
population   340: zero-failure sample at 95 %, 5 % tolerable = 54
population  2900: zero-failure sample at 95 %, 5 % tolerable = 58

For the 118 lockout procedures at Truro, a random sample of 46 with no failures would support the conclusion that fewer than 5 percent are wrong. For the 2,900 permits, 58 would be needed, close to the figure for an infinite population. These numbers explain why audits that check 'a few records' cannot support broad statements about a whole population, and why the audit report must say what kind of sample each conclusion rests on.

Writing a finding

An audit finding is a statement of what the evidence shows against the criteria. The Harrowgate Foods Ltd. audit procedure requires each nonconformity to have four parts:

• Criterion: the requirement, quoted or referenced, such as the clause of the lockout procedure that requires each worker to apply a personal lock.
• Condition: what the auditor found, with the evidence and its sources, stated factually and without exaggeration.
• Cause: where the auditor can determine it, why the condition exists. The auditee's own investigation will go further.
• Consequence: why it matters, in terms of risk or failure to meet the criterion.

Here is how Carla and Pierre wrote the group lockout finding: 'Criterion: section 5.2 of the lockout procedure requires each worker exposed to hazardous energy to apply a personal lock at each isolation point. Condition: on the night of the audit, four sanitation workers cleaned the spiral freezer under a single lock applied by the crew lead (observed). Two sanitation workers interviewed separately said this practice is used on busy nights. Permits for that night show four names under one lock number. Cause: crew leads stated that applying individual locks at three isolation points for four workers takes about twenty minutes, which the night schedule does not allow for. Consequence: workers can be exposed to restart if the single lock is removed while they are in the machine; this is the failure mode of last year's serious injury.'

The finding does not accuse anyone, does not speculate beyond the evidence, and makes the link to serious harm plain. A reader who was not there can see exactly what was found and how.

Grading findings

Findings are graded so that the response matches their importance. Harrowgate Foods Ltd. uses the grades common in management system auditing:

• Major nonconformity: a failure of a requirement that results in, or could result in, a serious risk, or the absence or total breakdown of a required process.
• Minor nonconformity: a failure of a requirement that is limited in scope and does not by itself indicate a breakdown of the system.
• Observation: a situation that is not yet a nonconformity but could become one.
• Opportunity for improvement: a suggestion where requirements are met but practice could be better.
• Good practice: something done well that other sites could learn from.

The lockout audit raised two major nonconformities, the group lockout on night shift and the failure to review 41 of the 118 procedures within the required interval, and three minor ones. It also recorded good practice: the bakery maintenance crew's use of photographs on each procedure showing every isolation point, which Pierre recommended for every site.

From findings to corrective action

An audit finding is not complete until it has been corrected and the correction verified. For each nonconformity, the auditee must investigate the cause, as HSE-403 described, propose corrective action, and set a date. The auditor or audit program owner reviews the proposed action to check that it addresses the cause, not just the symptom. When the action is complete, someone independent verifies that it has worked.

For the group lockout finding, the plant changed the night sanitation schedule to allow time for individual locks, installed group lockout boxes so that each worker could lock a box holding the isolation keys, and retrained crew leads. Three months later, Carla returned and observed eight night lockouts at random. All eight used individual locks on the group box. That was not proof that the practice had stopped everywhere, but together with the pulse survey and the new night verification checks, it was enough evidence for the audit program to close the finding.

Interviews that get to the truth

Interviews produced the evidence that the records hid, so the auditor course spent half a day on them. Carla's approach followed a few rules. She interviewed people at their place of work, not in an office, and asked them to show rather than tell: 'Walk me through how you would lock out this machine.' She asked open questions, such as 'What happens on a busy night?', rather than leading ones such as 'You always use your own lock, don't you?' She interviewed workers apart from their supervisors, and she explained at the start that the audit was looking at the system, not at individuals, and that names would not appear in the report.

She also listened for the difference between what people said they did and what they said others did. Workers who would not admit to a shortcut themselves would often say that 'some crews' took it. Those statements were not evidence on their own, but they told the auditors where to look and what to watch for on the night shift.