Full mock exams

A full mock exam is a rehearsal of the real thing: the full number of questions in the full time, at a desk, with the permitted calculator, no notes, no phone and no breaks except those the real exam allows. For Theo, that meant a NATE core and specialty pair back to back, and a separate open-book journeyman mock with the code books. Dana scheduled them for weeks 8, 10 and 11, each on a Saturday morning at the time his real exams would start. "A short set is practice," she said. "A full exam is a different animal. You'll find out what happens to your reading near the end, and you'd rather find out here."

This lesson covers how to run a mock exam so that its result means something, how to read the result block by block, how much a score can vary by chance and when a candidate is ready.

Running a mock properly

A mock exam is only useful if it is run like the real one. The questions should be new to the candidate, not ones already practiced, and of the same type and balance as the real exam: drawn across the blueprint in the right proportions. The timing should be exact, with the candidate's checkpoint card in view. Food, water and breaks should match the real rules. Dana marked the mocks herself, by block, the same day.

Conditions matter more than candidates expect. Theo sat his first mock at the kitchen table at home and was interrupted twice; his concentration fell apart after question 80. The second and third he sat in Northfield's empty training room, the closest thing to an exam center available, and the difference was obvious in the last hour.

Northfield's mocks drew questions from the practice bank in the proportions of Dana's blueprint for the NATE exams, and from the Kentucky code books for the journeyman mock, so that each mock had the balance of the real exam. Each mock used different questions.

Reading the result

The total score matters, but the block scores matter more, because they show where the remaining marks are. Theo's three mocks went like this.

The first mock set, at 72.8 %, was a pass, but a narrow one, with two weak areas: electrical and controls at 65 % and gas heat and hydronics at 50 %. Dana moved hours to both, as lesson 2 described. By the third mock, every area was at 72 % or above and the total was 82.8 %. Refrigeration circuits and heat pumps, already strong, stayed level; the gains came where the hours went.

How much a score can vary

One question worried Theo: if he scored over 80 % on a mock, how sure could he be of passing the real exam? The answer depends on how much a score varies by chance. A candidate's true ability on a topic is not one fixed number on every question; each exam samples a different set of questions, and on any given day some will fall on topics the candidate knows better or worse. A simple way to see the effect is to treat each question as a coin toss with the candidate's true accuracy as its chance of coming up right, and to work out the chance of reaching the pass mark.

# Chance of passing: if Theo's true accuracy is p, what is the chance of scoring at least 70 %?
from math import comb, ceil
def p_pass(n, p, mark=0.70):
    need = ceil(round(n * mark, 6))
    return sum(comb(n, k) * p ** k * (1 - p) ** (n - k) for k in range(need, n + 1)), need
for n in [50, 100, 60]:
    for p in (0.66, 0.70, 0.74, 0.78, 0.82):
        prob, need = p_pass(n, p)
        print(f"{n:3d} questions, need {need:3d}: true accuracy {p:.0%} -> chance of passing {prob:6.1%}")

 50 questions, need  35: true accuracy 66% -> chance of passing  33.2%
 50 questions, need  35: true accuracy 70% -> chance of passing  56.9%
 50 questions, need  35: true accuracy 74% -> chance of passing  79.2%
 50 questions, need  35: true accuracy 78% -> chance of passing  93.3%
 50 questions, need  35: true accuracy 82% -> chance of passing  98.8%
100 questions, need  70: true accuracy 66% -> chance of passing  23.2%
100 questions, need  70: true accuracy 70% -> chance of passing  54.9%
100 questions, need  70: true accuracy 74% -> chance of passing  84.7%
100 questions, need  70: true accuracy 78% -> chance of passing  97.7%
100 questions, need  70: true accuracy 82% -> chance of passing  99.9%
 60 questions, need  42: true accuracy 66% -> chance of passing  30.6%
 60 questions, need  42: true accuracy 70% -> chance of passing  56.3%
 60 questions, need  42: true accuracy 74% -> chance of passing  80.5%
 60 questions, need  42: true accuracy 78% -> chance of passing  94.6%
 60 questions, need  42: true accuracy 82% -> chance of passing  99.3%

The results are sobering. A candidate whose true accuracy is exactly 70 % has only about an even chance of passing any of these exams, because the score on the day could fall either side. At 74 %, the chance rises to about 80 to 85 %; at 78 %, to well over 90 %. Shorter exams vary more, so the 50-question core is the least predictable. A candidate scoring around the pass mark in practice is gambling. That was Dana's argument for aiming at 80 % in practice, not 70: a margin of ten points turns a coin toss into a near certainty.

The model is simple, and real exams are not quite coin tosses: questions differ in difficulty and topics cluster. But the lesson holds. A score is a sample, and a sample near the threshold says little about which side of it the next sample will fall.

When is a candidate ready?

Dana's readiness rule had three parts. Two consecutive full mocks at 80 % or above, under real conditions. No block below 70 % on the latest mock. And the error log showing mostly knowledge errors on rare topics, not misreads or calculation slips, which can be fixed and should have been. Theo met all three after the third mock.

A candidate who is not ready by the booked date has a choice. Many exam bodies allow a booking to be moved with notice, and a failed sitting costs a fee, a waiting period and some confidence. A short delay to close a clear gap is usually better than a sitting with a high chance of failure. Dana had seen candidates refuse to delay out of pride and regret it; she had also seen candidates delay repeatedly out of fear when they were ready. The readiness rule was meant to take that decision out of feelings and put it on evidence.

After each mock

The work after a mock is as important as the mock. Within a day, while the questions are fresh, every missed and guessed question goes into the error log with its cause. The block scores go into the study plan, moving hours toward the weakest blocks. The pacing goes on the chart from lesson 10: where did Theo fall behind, and why? And the conditions get a note: what was the hardest part of sitting for four hours, and what would help? Theo found that his reading slipped after about three hours, so he practiced a two-minute stretch at the 50 and 100 question checkpoints, which the real exam room would allow at his desk.

The last mock should come about ten days before the real exam, close enough to be a fair test of readiness and far enough away to leave time to fix anything it reveals. After that, the job is review and rest, the subject of lesson 12.

A worked example: reading Theo's second mock

Dana's notes on the second mock set show how a result becomes an action list. The total was just over 80 %. Gas heat and hydronics was still the weakest at 65 %, and the error log showed why: four of its errors were venting and combustion air questions from the National Fuel Gas Code, and in three of them Theo had found the right table but read the wrong row. That was a lookup habit, and it went onto the code-drill list with a rule to read the row and column headings aloud. Electrical and controls had improved to 76 %, but three of its errors were three-phase power questions where he had left out the 1.732. Those went onto the daily calculation drill and the spaced review cards.

The rest of the error log was small and scattered: a handful of rare facts, two misreads and one question that Dana agreed was badly written. That pattern, a few concentrated weaknesses and a scatter of one-off errors, is what a candidate close to readiness usually looks like. The concentrated weaknesses can be fixed in a week or two. The scatter mostly cannot, and does not need to be; it is the normal cost of a long exam.

Stamina and the long exam

Long exams test stamina as well as knowledge. Concentration falls with time, and the errors in the last hour of a four-hour sitting are different from those in the first: more misreads, more calculation slips, more answers chosen too quickly. Theo's first mock showed it clearly. In the first 60 questions he missed 12 %; in the last 40, he missed 31 %. By the third mock, after practicing in proper conditions and taking short stretch breaks at his checkpoints, the gap had nearly closed.

Simple things help. A normal meal beforehand, water during the exam if allowed, and a two-minute pause at planned points to stretch and breathe. Looking away from the page for a few seconds every half hour rests the eyes. Keeping to the checkpoints prevents the panic that comes from realizing the time is short. None of these is clever, and all of them were worth marks in Theo's mocks.

When a mock goes badly

Mocks sometimes go badly, and a bad mock is not a disaster. It is information. The first question is whether the result reflects a real weakness or a bad day: an interrupted sitting, illness, a night of little sleep or a mock with an unbalanced set of questions. A real weakness shows up in the error log as a concentration of errors in one area or of one cause. A bad day shows up as errors scattered across everything, often with many misreads. The response to a weakness is to change the plan; the response to a bad day is to sit another mock under better conditions before drawing conclusions.

Dana had one more rule about mocks: never sit one when you are already exhausted, and never sit two in one weekend. A mock is a measurement, and a measurement taken in bad conditions is a bad measurement, like a pressure read before the system has settled.

The mocks also taught Theo something about himself that no score showed. He tended to rush when he was ahead of his checkpoints, as if banking time were the goal, and his misreads clustered in those stretches. Being ahead of the pace, he learned, is a reason to slow down slightly and read more carefully, not to go faster.

For Theo's US exams, the challenge was the sequence rather than the length. The NATE core and specialty can be sat on the same day, and the open-book Kentucky exam asks for a different way of working, with the code books open. Dana had him sit one mock pairing the core and specialty with a short break between them, as he planned to on the day, so that he learned how his concentration held across both.