Practice exam strategy and error logs

By week 5 Kai had reviewed every area once and his topic scores were near target, but his mixed practice sets were stuck in the mid-sixties. Aggie was not worried. She had seen the pattern many times: an apprentice who knows the trade but loses marks to the way the questions are read and answered. The cure is a strategy for taking the exam and an error log that records not just which questions were wrong but why. This lesson covers both, using Kai's first three Saturday practice sets.

How multiple-choice questions are built

Each question has a stem, which asks the question, and four options, one correct answer and three distractors. Good distractors are not random; they are the answers that people who make a particular mistake would choose. A calculation question will include the answer you get by using the radius instead of the diameter; a code question will include the value from a nearby row of the table; a procedure question will include the step that is right but in the wrong order. Knowing this changes how options are read. An option is not right because it looks familiar. It is right because it answers exactly what the stem asks.

Stems also contain key words that change the answer: first, best, most likely, not, except, always, never. A question that asks what to do first when a leakage test fails has several options that are reasonable steps, but only one comes first. Kai's habit, taught by Aggie, was to underline the key word on the scrap paper and to say the question back to himself in his own words before looking at the options.

A method for every question

Aggie gave Kai a five-step method and asked him to use it on every practice question until it was automatic. Read the stem fully and identify what it is asking, including any key word. Predict the answer before reading the options, if possible. Read all four options, even if the first looks right. Eliminate the options that are clearly wrong, which usually leaves two. Choose between the remaining two by going back to the stem and asking which one answers it exactly.

Pacing

The committee's final written exam allows three hours for 100 questions, 1.8 minutes per question on average. Recall questions take well under a minute and calculations take two or three, so the average works only if the easy questions are answered quickly. Kai planned to pass checkpoints: 25 questions by 40 minutes, 50 by 80 minutes, 75 by 120 minutes and all 100 by 160 minutes, leaving 20 minutes to go back to marked questions.

# Pacing checkpoints: where you should be, with time reserved for review
questions = 100
minutes = 180
review = 20                      # minutes kept for marked questions at the end
per_q = (minutes - review) / questions
print(f"working pace {per_q:.2f} min per question")
for frac in (0.25, 0.5, 0.75, 1.0):
    q = round(questions * frac)
    print(f"question {q:3d} by minute {q * per_q:5.0f}")

working pace 1.60 min per question
question  25 by minute    40
question  50 by minute    80
question  75 by minute   120
question 100 by minute   160

The rule for a hard question is simple: if it is not solved in about twice the average time, mark it, pick the best answer so far, and move on. A blank answer scores nothing; a marked guess has a one-in-four chance at worst, and better odds if two options have been eliminated. Coming back with fresh eyes often makes the answer obvious. Kai's practice sets showed that his low scores came partly from spending five or six minutes on two or three hard calculations and then rushing the last twenty questions.

The error log

After each practice set, Kai entered every wrong answer in an error log. For each one he recorded the question number, the area, the topic, the right answer, and above all the cause: why he got it wrong. Aggie gave him six causes to choose from. Misread question: the knowledge was there, but he answered a different question from the one asked. Formula slip: the method was right but a step went wrong. Code or table value: he did not know or misremembered a value. Units: the units were mixed or not converted. Guessed: he did not know the topic. Out of time: he did not reach the question or rushed it.

The cause matters more than the topic, because each cause has a different cure. Misreading is cured by the five-step method; formula slips by writing the formula first and estimating; code values by flash cards; units by converting first; guessing by more study of the topic; and running out of time by pacing. Without the cause, the obvious response to a wrong answer is to reread the topic, which cures only one of the six.

# Error log Pareto: which causes account for most of the lost marks
from collections import Counter
log = (["misread question"] * 14 + ["formula slip"] * 9 + ["code value"] * 7 +
       ["units"] * 5 + ["guessed"] * 4 + ["out of time"] * 3)       # 42 errors in three sets
counts = Counter(log).most_common()
total = sum(n for _, n in counts)
cum = 0
for cause, n in counts:
    cum += n
    print(f"{cause:17s} {n:3d}  cumulative {100 * cum / total:5.1f} %")
top = [c for c, _ in counts[:3]]
print("focus next fortnight on:", ", ".join(top))

misread question   14  cumulative  33.3 %
formula slip        9  cumulative  54.8 %
code value          7  cumulative  71.4 %
units               5  cumulative  83.3 %
guessed             4  cumulative  92.9 %
out of time         3  cumulative 100.0 %
focus next fortnight on: misread question, formula slip, code value

The Pareto chart surprised Kai. He had assumed his problem was knowledge, but only four of his 42 errors were true guesses. A third of them were misread questions, which is good news, because misreading is the easiest cause to fix. The next two causes, formula slips and code values, were also habits rather than gaps. For the next fortnight he used the five-step method on every question, wrote formulas before numbers, and spent his Thursday flash card session on the values he had missed.

Reviewing a practice set

A practice set is only half done when the last answer is marked. The review is the other half, and Aggie insisted it take at least as long as the set itself. For each wrong answer, Kai worked out the right answer without looking first, then checked it, then wrote the cause in the log. For each right answer that had been a guess, he logged it too, as a lucky guess, because it would not be lucky next time. Once a fortnight he went back through the log and reworked every old error, crossing out the ones he now got right. The errors that kept coming back went on a short list for the final weeks.

Building practice sets

Practice questions come from several places: the self-assessment and sample questions published for the trade, commercial question banks, the training provider's banks, and questions written by the apprentice. Writing questions turned out to be one of Kai's most useful exercises. To write a good distractor he had to understand the mistake that would lead to it, and once he understood the mistake he stopped making it. Aggie also warned him about two risks: sets that repeat until the answers are memorized, and banks of poor quality whose answers are wrong. A score that rises because the questions are familiar is not a sign of readiness.

Guessing well

Every candidate meets questions they cannot answer with certainty, and how they guess matters. A blind guess among four options has a 25 percent chance; after eliminating one wrong option it rises to 33 percent, and after eliminating two it is 50 percent. Since there is no penalty for wrong answers, every question should be answered, and every guess should be made after as much elimination as the time allows. Options that are much longer or more specific than the others, options that use absolute words like always or never, and options that contradict a basic safety principle are often, though not always, distractors.

# Expected extra marks from guessing well on the questions you are unsure of
unsure = 12                       # questions with no confident answer
for removed in (0, 1, 2):
    p = 1 / (4 - removed)
    print(f"eliminate {removed} option(s): chance {p:.2f}, expected correct {unsure * p:.1f} of {unsure}")

eliminate 0 option(s): chance 0.25, expected correct 3.0 of 12
eliminate 1 option(s): chance 0.33, expected correct 4.0 of 12
eliminate 2 option(s): chance 0.50, expected correct 6.0 of 12

On a paper where the pass mark is close, those few extra expected marks can be the difference. Kai's log showed that his guesses after eliminating two options were right more than half the time, which told him his elimination was sound.

Reading drawings and tables in the exam

Some exam questions come with a drawing, a detail or a table. These take longer, and they are worth a method of their own. Read the question first, so that you know what to look for; then look at the drawing or table for that one thing, rather than studying the whole figure. Check the units and the scale, and check any notes beside the figure. For table questions, put a finger or the edge of the scrap paper on the row and the column, because the most common error is reading from the row above or below. Kai practiced this with photocopied tables from the duct construction standard, and his table questions went from his weakest type to one of his strongest.

Managing nerves

Most candidates feel nervous before and during the exam, and a little nervousness sharpens attention. Too much makes people misread questions and second-guess correct answers. Kai's practice sets were part of the cure, because by exam day the format was familiar. Aggie also taught him two simple techniques. Slow breathing, a few breaths in for a count of four and out for a count of six, calms the body quickly. And moving on, marking a question that causes panic and going to the next one, turns a stuck moment into a small task. Both are practiced in mocks, so that they are automatic when they are needed.

Aggie's last piece of advice on nerves was to stop comparing. Other candidates leaving early, or seeming relaxed, say nothing about their scores. Kai's plan was his own, his pace was his own, and the only checkpoints that mattered were the ones on his own scrap paper.