Practice set analysis

On Tuesday, November 10, 2026, before his study plan was finished, Andre sat a 40-question diagnostic practice set in 80 minutes at his kitchen table, with his phone in another room. He scored 27 out of 40. His first reaction was to look up the answers to the 13 he got wrong and move on. Noor stopped him. "The score is the least useful thing on that sheet," she said. "What matters is why you lost each point." This lesson shows how to analyse practice results so that every set changes what you study next.

What a practice score does and does not tell you

A raw practice score, 27 out of 40 or about 68 in 100, is not a prediction of an exam result. The real exam reports a scaled score from 100 to 1,000, and the relationship between the share of questions answered correctly and the scaled score is not published and may differ between versions. Practice questions also vary in difficulty and quality. A practice score is best treated as a rough reading on a trend: useful when compared with your own later scores on similar material, misleading when compared with the passing score.

Small samples need particular care. With about ten questions per domain, a single extra right or wrong answer moves a domain result by ten points out of 100. AWS gives the same caution for the section feedback on real score reports. Treat domain results from one set as a hint, and look for patterns that persist across several sets.

Step 1: results by domain

The domain pattern confirmed what Andre had guessed: governance was his weakest area. But it also showed that Domain 2, his supposed strength, was only middling. Three of the strategy questions he missed were business case questions he was sure he knew. That was the first clue that the problem was not always knowledge.

Step 2: an error log

For every wrong answer, and for every right answer that was a lucky guess, Andre recorded the question's topic, the option he chose, the correct option and, most importantly, the cause of the error. Noor suggested five causes.

Only 4 of the 13 lost points came from missing knowledge. The other 9, more than two thirds, came from reading, traps, second-guessing and time. That changed the plan. Extra reading alone would have fixed at most four points; practising the reading method and the trap list could fix more.

Step 3: look at the questions you got right

Analysis is not only about mistakes. Andre marked each correct answer as sure or unsure. Seven of his 27 correct answers were unsure: he had narrowed to two options and guessed. Those seven are fragile points that could easily be lost on exam day, so their topics went onto his review list alongside the errors. A candidate who scores 68 with many lucky guesses is less ready than one who scores 68 with confident answers.

Step 4: time

Andre finished in 74 of the 80 minutes, an average of 1.85 minutes per question. But his time was uneven. He spent more than four minutes on two long business case questions and then rushed the last five questions, guessing one without reading it properly. Timing data matters as much as accuracy: the fix is a checkpoint plan and a rule for when to flag a question and move on, which lesson 12 sets out.

Step 5: plan the next practice sets

Noor's rule for practice sets was "never the same set twice, and never without analysis". A repeated set measures memory. A set taken without analysis wastes the questions. Andre's plan for the remaining practice was:

The goal for the full sets was not a particular raw score. It was a stable pattern: errors mostly from genuinely hard questions, few from misreading or traps, consistent pacing with time to review flagged items, and no domain far behind the others.

Working through the error log

Noor asked Andre to take three entries from his error log and explain each one fully: what the question asked, why he chose what he chose, and what would have led him to the right answer. The aim was not to memorize those three questions, which would never appear again, but to see the thinking that went wrong.

Tracking progress across sets

One set is a snapshot. Several sets make a trend. Andre's plan is to keep a simple tracking table after each full set: the overall raw score, the score by domain, the number of errors by cause, the number of unsure correct answers, and minutes used. The numbers he cares most about are not the overall score but the error causes. If misreading and trap errors fall from 7 of 13 to 2 or 3 of a similar number of misses, the reading habits are working, even if the overall score moves only a little because the later sets are harder.

The targets are deliberately described as patterns rather than precise numbers, because practice sets differ in difficulty and the relationship to the scaled score is unknown. What matters is that the kinds of mistakes change.

When results stall

Sometimes scores stop improving despite steady study. Noor's checklist for a stall has four questions. Is the study aimed at the causes in the error log, or at comfortable topics? Are practice questions being reviewed fully, including the right answers? Is fatigue affecting the later questions in a set? And are the practice questions themselves good, or do their explanations contradict the official guide? A stall usually has one of these causes, and each has a different fix: redirect study, slow the review down, practise in shorter timed blocks before building up to full sets, or change practice material.

Andre wrote the checklist on the inside cover of his notebook. He hoped not to need it, but it was there if he did.