Practice set analysis

A practice set is only as useful as the analysis that follows it. Many candidates take a practice test, look at the score, feel either relieved or worried, and move on. Noor treats the score as the least interesting number on the page. On November 10, 2026, Jonah sat her 40-item diagnostic, written for this program and covering all five domains in proportion to their weights. He answered 27 correctly. This lesson shows how Noor and Jonah analyzed the 13 misses, and how you can do the same with any practice set you take, including the full sets in his plan for January and March 2027.

Why the score alone misleads

Twenty-seven out of 40 is about 68 in 100, which sounds close to a pass. But a raw proportion on an unofficial set cannot be converted to AWS's scaled score, and practice sets differ in difficulty from the real exam and from each other. A single score also hides which kinds of mistakes you make. Two candidates with the same score may need completely different study plans: one may lack knowledge of a domain, another may know everything but misread stems under time pressure. So the first rule of analysis is to stop looking at the total and look at the misses.

Classifying each miss

Noor asks Jonah to classify every wrong answer by its cause, using four categories. A knowledge gap means he did not know the fact or concept, for example what Amazon Macie does. A confusion means he knew both options but mixed them up, for example choosing Comprehend where Textract was needed. A reading error means he knew the answer but misread the stem, missed a constraint such as "without retraining", or chose too few options on a multiple-response item. A timing error means he rushed or guessed because time was running out. Each category has a different remedy, which is the whole point of classifying.

Reading the heat map

The heat map turns 13 misses into a picture. Two patterns stand out. First, confusions are spread across domains 1, 2, 3 and 5, five misses in all, which is why lesson 9 matters so much for Jonah: one drill on confused pairs fixes mistakes in four domains at once. Second, knowledge gaps cluster in domains 3 and 5, where he did not know specific items: the vector stores named in the guide, the difference between continued pre-training and fine-tuning, Amazon Macie, and AWS Artifact. Those need real study, not just drills. Reading and timing errors are few, but both happened in domain 3, on the longest scenario items, which suggests he needs a pacing rule for long stems.

Noor also looks at the miss rate per domain, not just the count. Domain 5 had six items and three misses, a rate of 0.5, the worst of any domain. Domain 3 had five misses, the most in number, but from eleven items, a rate of about 0.45. Domain 1 had one miss from eight. These rates fed the time allocation in lesson 10.

items  = {"D1": 8, "D2": 10, "D3": 11, "D4": 5, "D5": 6}
missed = {"D1": 1, "D2": 3,  "D3": 5,  "D4": 1, "D5": 3}
causes = {"knowledge": 5, "confusion": 5, "reading": 2, "timing": 1}
for d in items:
    print(d, "miss rate", round(missed[d] / items[d], 2))
print("total correct", sum(items.values()) - sum(missed.values()), "of", sum(items.values()))
top = max(causes.values())
print("most common causes:", [c for c in causes if causes[c] == top])

D1 miss rate 0.12
D2 miss rate 0.3
D3 miss rate 0.45
D4 miss rate 0.2
D5 miss rate 0.5
total correct 27 of 40
most common causes: ['knowledge', 'confusion']

Reviewing correct answers too

Analysis should include the questions you got right. Jonah marked each answer while taking the set with a confidence level: sure, fairly sure, or guessed. Four of his correct answers were guesses. A correct guess is a hidden gap, so those four items join the study list. Noor calls them "lucky points": they will not be lucky twice. Equally, an item answered correctly with high confidence for the wrong reason is a gap; reading the explanation for every item, right or wrong, catches this.

Writing explanations in your own words

For each miss and each lucky point, Jonah writes a two-line entry in his mistakes log. The first line states the correct idea in his own words, not copied from the explanation. The second line states the clue in the stem that should have led him there. For the fine-tuning item he misread, the entry reads: "RAG gives current facts without changing the model; fine-tuning changes behavior and needs labeled examples. Clue: 'without retraining' and 'latest policies'." Writing the clue trains the reading habit that prevents the next mistake.

A worked analysis of three misses

Noor walked through three of Jonah's misses in detail to show the method. The first was a domain 3 item: a company wants its assistant to answer from its latest policies and asks for the most cost-effective approach. Jonah chose fine-tuning. When he reread the stem, he saw the words "latest policies" and "without retraining", both of which point to retrieval augmented generation. He knew the difference between RAG and fine-tuning perfectly well, so this was a reading error. His log entry names the two clue phrases.

The second was a domain 5 item asking which service gives auditors AWS's compliance reports. Jonah chose AWS CloudTrail. He did not know AWS Artifact at all, so this was a knowledge gap, and it led to the governance-services session in his February plan. The third was a domain 2 item: a team needs managed access to foundation models without building infrastructure, and the options included Amazon Bedrock and SageMaker JumpStart. Jonah chose JumpStart. He knew both services but had not grasped that JumpStart deploys models onto SageMaker infrastructure that the team manages, while Bedrock is serverless. That is a confusion, and the pair went onto his list.

Three misses, three different causes, three different remedies: rereading the stem, studying a new service, and drilling a pair. If Noor had simply told him to "study more", he would have spent hours rereading material he already knew and fixed only one of the three.

Tracking analysis over time

Jonah keeps his analysis in a simple spreadsheet with one row per miss: date, practice set, domain, task statement, cause, the correct idea in his own words and the clue. After the first full practice set in January, he will add those rows; after the second in March, he will add more. Sorting by task statement shows whether misses cluster in particular parts of a domain, for example task 5.2 governance rather than task 5.1 security. Sorting by cause shows whether his problem is shifting from knowledge to reading, which is a sign of progress: knowledge gaps usually close first, and the misses that remain are about technique.

The spreadsheet also guards against a common illusion. After many weeks of study, candidates often feel that everything is weak, because they now know how much there is to know. The data shows otherwise. Seeing that domain 1 produced one miss in forty items, and that confusions fell after the pairs drills, keeps effort pointed at the real gaps and keeps morale steady.

Timing analysis

Timing deserves its own look, because the heat map shows only one timing miss but that hides how close Jonah came. Noor asked him to note the time at questions 10, 20 and 30 of the diagnostic. He reached question 20 at about 28 minutes, which on a 65-question exam in 90 minutes would put him well behind the pace needed for review time. Two long scenario items took more than three minutes each. That is why lesson 12 gives a flag-and-move rule: choose the best answer you have after about two minutes, flag it and come back. In the first full practice set planned for January, Jonah will record the same checkpoints and compare them with the targets in lesson 12.

Pace is also affected by item type. Ordering and matching items take longer than multiple choice because they ask for several decisions. Case studies take longer to read the first time but can be faster for later questions, because the scenario is already familiar. Knowing where your time goes is the first step to managing it.

One last caution about practice sets: they are written by people who sometimes make mistakes. If an explanation contradicts the exam guide or AWS documentation, log the item as disputed rather than as your miss, check the official source and move on. Arguing with a flawed practice item wastes time that is better spent on real gaps, and learning a wrong answer from it is worse than not having seen it at all.

When the analysis is done, turn it into no more than three actions for the coming week. Three is enough to make real progress and few enough to finish. For Jonah after the diagnostic, they were: study the six governance services and Macie; drill the five confused pairs; and practice underlining constraints on ten domain 3 stems. Each action maps directly to a cluster on the heat map, which is the point of the whole exercise.

Using a second practice set well

A second practice set is only informative if it is different from the first and taken under the same conditions: timed, closed-book and in one sitting. Retaking the same set measures memory of that set, not readiness. Jonah's plan places the second set in March, after his February review, so it can show whether the confusions and knowledge gaps have closed. If the heat map from the second set still shows knowledge gaps in domain 5, he will spend the buffer week there. If the misses have shifted to reading errors, the remedy is pacing and technique, not more content.

Noor's last instruction for this lesson is about emotion. A bad practice result is information, not a verdict. "The practice set is where you are allowed to be wrong," she wrote. "Every miss you analyze now is a point you collect in April."