Performance evaluation, scorecards and close-out

Last August's shutdown ended, as most shutdowns do, in a rush. The restart date had already slipped four days because of the fire. Contractors packed up as their jobs finished, and some left before anyone from the plant had looked at their work area. On the first morning of production, a bakery line operator found a contractor's adjustable wrench on a conveyor guard above an open product zone. The line was stopped, the product from the first hour was held for inspection, and the quality team spent a day confirming nothing had fallen into the food. Nobody ever recorded which contractor the wrench belonged to, how the contractors had performed or which ones should be invited back.

The last two stages of the lifecycle, close-out and evaluation, are the ones most often skipped, because by then the work is done and everyone wants to move on. They are also the stages that protect the restart and improve the next job. This lesson covers how Harrowgate Foods Ltd. now closes out contractor work and how it evaluates contractor performance.

Closing out the work

Close-out is the process of confirming that a contractor's work is complete, that the area is safe to return to operations and that everything the job changed has been restored or handed over properly. At a food plant, close-out also has to protect the product. Harrowgate Foods Ltd.'s close-out checklist is completed by the job owner, the area owner and the contractor's supervisor together, at the job, before the area is handed back:

• Permits and isolations. Every permit for the job has been closed, every isolation removed in the right order by the people who applied it, and the equipment is ready for the area owner to restore.
• Safety systems. Guards, interlocks, alarms, detectors and fire protection that were removed or impaired have been restored and tested, and impairment certificates have been closed.
• Housekeeping and waste. Materials, scrap, packaging and waste have been removed, and hazardous waste has been disposed of properly.
• Temporary items. Scaffolds, barriers, temporary power, hoses, signs and tools have been removed or formally handed over.
• Food safety. A tool and parts count has been completed against the contractor's tool list, the area has been checked for foreign material, and the area has been cleaned and released by the quality team before production.
• Documents. Test certificates, inspection records, drawings showing the work as built and operating information have been handed over.
• Change and start-up. Any change has completed its pre-startup review, and the people who will operate the changed equipment have been trained.

The tool count was Aisha Rahman's addition, in direct response to the wrench on the conveyor guard. Contractors working in production areas now keep a tool list at the job, check their tools in and out of the area, and report any missing tool at once. A missing tool stops the hand-back of the area until it is found. In this year's shutdown, the count found two missing items, a socket and a box of cable ties, both recovered before the areas were released.

Handing the area back

Close-out ends with a formal handover: the area owner signs the checklist to accept the area back, and from that moment the area is under production control again. Equipment is restarted only by the plant's operators, following the plant's start-up procedures. Where contractors still need to be present at start-up, for example to commission the new freezer, their role at start-up is defined in the start-up plan, and they work under the plant's permits until the equipment is accepted.

Restart is a high-risk period in its own right. Equipment that has been worked on may behave differently, and people who have spent two weeks around stopped machinery may forget it can now move. Harrowgate Foods Ltd. holds a restart briefing for every shift on the first day of production, covering which equipment has changed, which areas still have contractors in them and which barriers are still in place. Contractor crews still working are told which areas are now live.

Why evaluate contractors

A contractor's performance on one job is the best evidence the host has about how it will perform on the next. Prequalification, as Lesson 5 explained, relies largely on what contractors say about themselves and on records from other sites. Performance evaluation records what the host has seen for itself. Over time, it becomes the most valuable input to the approved list.

Evaluation also helps the host show due diligence, because it shows that the host monitored contractor performance and used what it learned.

The contractor scorecard

Harrowgate Foods Ltd.'s scorecard rates each contractor on a medium- or high-risk job on six criteria. Each is scored from 1 to 5 using defined descriptions, so that different job owners score in the same way, and each has a weight reflecting its importance.

Elena Novak deliberately left injury counts out of the scorecard. A contractor with no injuries on one two-week job has not necessarily worked safely; it may have been lucky. More importantly, scoring contractors on injury counts encourages them not to report. The reporting and learning criterion rewards the opposite: a contractor that reports near misses promptly and acts on them scores higher than one that reports nothing. Serious incidents are not ignored, though. Any serious injury, fire, release or life-saving rule breach triggers a separate review, and the rating cannot be higher than 'conditional' until that review is complete.

# Weighted contractor scorecards for the shutdown
WEIGHTS = {"audits": 0.25, "permits": 0.25, "supervision": 0.15,
           "reporting": 0.15, "closeout": 0.10, "subs": 0.10}
scores = {
    "Bayview Mechanical": {"audits": 4, "permits": 4, "supervision": 5, "reporting": 4, "closeout": 4, "subs": 4},
    "Northline Refrig.":  {"audits": 5, "permits": 5, "supervision": 4, "reporting": 5, "closeout": 4, "subs": 3},
    "Atlantic Roofing":   {"audits": 3, "permits": 3, "supervision": 3, "reporting": 4, "closeout": 3, "subs": 3},
    "AirPro Duct":        {"audits": 4, "permits": 4, "supervision": 4, "reporting": 3, "closeout": 2, "subs": 3},
}
def band(s):
    return "preferred" if s >= 4.2 else "approved" if s >= 3.4 else "conditional" if s >= 2.6 else "remove"
for firm, s in scores.items():
    total = sum(WEIGHTS[k] * v for k, v in s.items())
    low = [k for k, v in s.items() if v <= 2]
    print(f"{firm:<19} {total:.2f}  {band(total):<11} {'weak: ' + ', '.join(low) if low else ''}")

Bayview Mechanical  4.15  approved    
Northline Refrig.   4.55  preferred   
Atlantic Roofing    3.15  conditional 
AirPro Duct         3.55  approved    weak: closeout

The weighted score places each contractor in a band that feeds back into the approved list. Preferred contractors are invited first for similar work and audited a little less often. Approved contractors continue as before. Conditional contractors must agree an improvement plan before their next job and are audited more often. Contractors in the lowest band are removed from the approved list for the work concerned, though they can reapply after showing improvement. Atlantic Roofing, new to the site, landed in the conditional band, mainly because of the warning line finding from Lesson 9 and two permit lapses, so it must agree an improvement plan, including a review of its edge protection planning, before its next roof job.

Doing the evaluation well

A scorecard is only as good as the evidence behind it and the conversation that follows it. Harrowgate Foods Ltd.'s job owners complete the scorecard within a week of close-out, using the audit log, the permit records, the incident log and their own observations. Elena's team reviews every high-risk scorecard to check that the scores match the evidence.

Each high-risk contractor then has a short debrief meeting with the job owner and someone from Elena's team. The meeting goes through the scorecard, what went well and what should change. Elena insisted that the debrief also ask the contractor to evaluate the plant. Were permits issued on time? Were isolations ready when promised? Was the site information accurate? Did the coordination meeting help? Contractors see things about the host's systems that the host cannot see itself, and they are often more willing to say so when asked directly. Northline's feedback this year, for example, was that permits for the engine room had sometimes been delayed by an hour because only one issuer was trained for the area. A second issuer was trained before the next planned outage.

Ratings are shared with the contractor in writing. A contractor that disagrees with a rating can ask for a review by Elena and Kevin Doucette. In the first year, two contractors asked for reviews; one rating was raised after the contractor showed that an audit finding had been corrected within the hour, which the record had not captured.

Evaluating resident and recurring contractors

Contractors who work on site all year, such as the sanitation contractor, are evaluated quarterly rather than per job, using the same criteria. Their evaluations are reviewed at an annual meeting with the contractor's management, which also covers their workers' incidents, training and turnover. For the sanitation contractor, the first annual review led to an agreement that its crew leaders would attend the plant's supervisor safety training, and that the plant's night supervisor would audit one sanitation task each night.

Using the results across the program

Individual scorecards matter most for the contractor concerned. Taken together, they also show how the program itself is working. After the first shutdown under the new program, Elena's team combined all nineteen scorecards and the audit log and looked for patterns. Close-out and housekeeping was the weakest criterion across almost every contractor, which suggested the problem lay partly with the host: the scopes had not said clearly enough what condition areas should be left in, and the plant had not provided enough bins, cleaning stations or laydown space. Subcontractor management scores were lower for firms using deeper chains, which supported the rule limiting chain depth.

The combined results went into a short report to Rachel Okoye and the joint occupational health and safety committee, with four changes for the next shutdown: a housekeeping section in every scope, more waste stations in the bakery and freezer halls, a second engine room permit issuer and a pre-award meeting for every subcontractor on a high-risk job, not just the prime contractors. The report also gave the leadership team its first real picture of contractor safety at the site: hours worked, audit results, incidents and near misses, all recorded separately from the plant's own statistics, as Lesson 1 had recommended.