Practice exam strategy and error logs

By week 7, the content review was done and the plan switched to mixed practice. Theo sat his first timed set of 50 mixed questions on a Saturday morning and scored 71 %, just over the pass mark. Dana did not congratulate him. She asked to see every question he had got wrong and, for each one, why. "The score tells you where you are," she said. "The reasons tell you how to move. Most people only look at the score." This lesson covers the technique of answering multiple-choice questions well and the error log that turns practice into improvement.

Exam technique is not a trick for passing without knowledge. It is a way of making sure the knowledge a candidate has turns into marks: reading what the question actually asks, using the information it gives, managing time and not losing marks to carelessness. Theo's first practice set showed that about half his errors had nothing to do with what he knew.

Reading the question

The first rule is to read the whole stem before looking at the options, and to know what is being asked before deciding anything. A useful habit is to say the question back in plain words: "they want the first thing to check", "they want the most likely cause given high superheat and high subcooling", "they want the one that is not a requirement". Many errors come from answering the question the candidate expected rather than the one on the page.

Some words change a question completely, and question writers use them deliberately. First, initial and before ask for the order of steps, not the most important one. Most likely, best and primary ask for the strongest answer among several that might be partly right. Not, except and least ask for the one option that does not fit. Always and never are rare in correct answers, because few things in the trade are absolute, except in safety rules, where they often are. Theo started circling these words on paper practice sets, and on screen pausing on them, until it became automatic.

The second rule is to use every piece of information in the stem. A stem that mentions the system has a TXV, that the weather is cold, that the equipment is new or that a repair was just done gives that information for a reason. If an answer ignores a fact in the stem, it is probably a distractor. Conversely, do not add information the stem does not give. If it does not say the filter is dirty, do not assume it is.

Choosing the answer

When the answer is not obvious, elimination works. Read each option and decide whether it is definitely wrong, possibly right or definitely right. Cross out the definitely wrong ones. With two options left, compare them against every fact in the stem: usually one fits all of them and the other fits most. Choosing between two plausible options after eliminating two doubles the chance of a correct guess even when knowledge runs out.

Some patterns help when two options remain. On safety questions, the safe option is nearly always right. On order-of-steps questions, the option that checks the simple and quick things first, or makes the situation safe first, is usually right. On calculation questions, an estimate rules out options far from it, and the remaining options often differ by a common error: the gauge-for-absolute mistake, the missing 1.732, the wrong saturation point. Knowing the common errors shows which option is the trap. And options that are much longer or more detailed than the others are not more likely to be right; question writers try to avoid that pattern.

Changing answers is a common worry. The evidence from exam research is fairly consistent: when candidates change an answer after reconsidering, they more often change a wrong answer to a right one than the reverse. The rule Dana used was simple: change an answer when you have found a reason, such as a fact in the stem you missed or a calculation error, and not because of a vague feeling.

Managing time

Lesson 1 set Theo's checkpoints. Lesson 10's addition was a rule for hard questions. On the first pass, answer every question that can be answered in about the average time. For a question that will take much longer, choose the best answer so far, flag it and move on. At the end of the first pass, use the reserve time on the flagged questions. Never leave a question blank: there is no penalty for wrong answers on these exams, so a blank is a certain zero and a guess has at least a one-in-four chance, more after elimination.

The other time trap is the familiar question answered too fast. Questions on topics a candidate knows well are where careless reading costs the most marks, because the candidate stops reading halfway through the stem. Theo's practice sets showed that his slowest questions were the ones he got right, and his fastest were the ones he got wrong on topics he knew. Slowing down slightly on easy questions turned out to be worth more than speeding up on hard ones.

The error log

The most effective single habit in Theo's preparation was the error log. For every question he got wrong, and every one he got right by guessing, he wrote down four things: the topic, the correct answer, why the correct answer is right, and why he chose what he did. The last one was the important one. He gave each error a cause from a short list: misread the stem, did not know the rule, calculation slip or wrong unit, knew the theory but chose a distractor, ran out of time, or a topic he had never studied.

After three practice sets, the causes could be counted. The script sorts them into a Pareto table, largest first, with a running total, and then shows the dates for reviewing each missed question again at increasing intervals.

# Error log: sort missed questions by cause, then schedule spaced reviews
from datetime import date, timedelta
errors = {  # cause: number of missed questions over three practice sets
    "misread the stem (missed NOT, FIRST, MOST)": 14,
    "did not know the code or rule": 11,
    "calculation slip or wrong unit": 9,
    "knew the theory but chose a distractor": 6,
    "ran out of time": 4,
    "guessed on a topic never studied": 3}
total = sum(errors.values()); run = 0
for cause, n in sorted(errors.items(), key=lambda kv: -kv[1]):
    run += n
    print(f"{n:3d}  {n / total:5.1%}  cumulative {run / total:5.1%}  {cause}")
start = date(2027, 1, 11)
print("review dates for each missed question:", ", ".join(str(start + timedelta(days=d)) for d in (1, 3, 7, 14, 30)))

 14  29.8%  cumulative 29.8%  misread the stem (missed NOT, FIRST, MOST)
 11  23.4%  cumulative 53.2%  did not know the code or rule
  9  19.1%  cumulative 72.3%  calculation slip or wrong unit
  6  12.8%  cumulative 85.1%  knew the theory but chose a distractor
  4   8.5%  cumulative 93.6%  ran out of time
  3   6.4%  cumulative 100.0%  guessed on a topic never studied
review dates for each missed question: 2027-01-12, 2027-01-14, 2027-01-18, 2027-01-25, 2027-02-10

The table was a surprise to Theo. The largest single cause of his errors, nearly 30 %, was misreading the stem, and the third, nearly 20 %, was calculation slips. Together, half his errors were not about knowledge at all. Missing rules, the cause he had assumed was his main problem, was second, at under a quarter. The Pareto table changed the plan for the next weeks: the reading habits and the daily calculation drills from lesson 9 got as much attention as the remaining content.

The review dates use spacing. A missed question is reviewed the next day, then after three days, a week, two weeks and a month. Each review means answering the question again without looking, and checking the reason. Questions answered correctly at every review drop out; those missed again go back to the start. The method is sometimes called spaced repetition, and it is one of the most reliable ways known of moving material into long-term memory. Theo kept his on index cards in a box with dividers for the review dates; an app does the same job.

Practice that counts

Not all practice is equal. Practice questions should be of the same type and difficulty as the real exam, and Dana was wary of question banks that asked mostly simple recall, because they give flattering scores. Northfield's bank had been built over years by mechanics who had written the exams and remembered the kinds of questions asked, never the questions themselves; reproducing real exam questions is prohibited and pointless, because the questions change. NATE publishes its knowledge area outlines and offers practice materials, and the Kentucky code books themselves are the best practice material for an open-book exam. Practice under exam conditions, timed, without notes or phone, gives scores that mean something; practice with the book open and the phone on does not.

Theo's scores on mixed sets over weeks 7 to 10 rose from 71 % to 76 %, then 79 % and 81 %. The rise came mostly from fewer misread stems and fewer calculation slips, exactly as the Pareto table had predicted. His knowledge errors fell more slowly, as the spaced review worked through them.

Answering calculation questions

Calculation questions deserve their own routine because they take longest and invite the most slips. Theo's routine had five steps. Write down what is given, with units. Write down what is asked, with its unit. Pick the formula and write it. Substitute and calculate, keeping units. Compare the result with a rough estimate and with the options. If the result matches an option, check that the option does not belong to a common error, such as gauge instead of absolute; if it matches none, look for a unit or arithmetic slip before guessing. The routine adds perhaps twenty seconds to a question and saves far more than that in marks.

Scratch paper helps if it is organized. Theo divided his into numbered boxes, one per flagged or calculation question, so that when he returned to a flagged question his earlier work was there and he did not have to start again. On computer-based exams, the scratch paper or whiteboard rules vary, so it is worth knowing beforehand what will be provided.

Keeping the log honest

An error log only works if it is honest. The temptation is to label every error as a misread, which feels better than not knowing the rule, or to skip logging the questions that were guessed correctly. Dana checked Theo's log against his answers for the first two weeks, and twice found errors marked as misreads that were really gaps in knowledge. After that, Theo's own entries were reliable, and the Pareto table could be trusted to direct the study plan.

Dana also kept a short list of the habits that cost Theo marks in practice, written as reminders rather than rules: read to the end of the stem; check whether the question wants the first step or the most important one; write the units; estimate first; never leave a blank. He read the list before every practice set and, on exam day, before turning over the first page.