JAVA-290 · Lesson 8 of 12 · 31 min

Testing and quality

▶ Video coming soon A video version of this lesson is on the way.

Sprint 4, from Thursday, July 13 to Wednesday, July 26, 2028, had a goal with two halves: customers get reminders and Odile gets her figures, and every page passes the accessibility checks. The second half was the reason Ingrid had pushed for it to be written into a goal. Testing had been part of every story since sprint 1, because the Definition of Done required a test for each acceptance criterion, but a capstone team under time pressure is always tempted to treat quality as whatever is left over at the end. Naming it in a sprint goal made it work the team would plan, estimate and demonstrate.

Thierry, who had found more of the team's bugs than anyone, became the sprint's quality lead. His first act was to write down what the team was already doing, because nobody had seen it in one place, and to ask for each kind of test what it was for, what it cost and what it could not find.

A strategy, not a pile of tests

The test pyramid, a picture Mike Cohn popularized around 2009, says that a healthy suite has many fast, small tests at the bottom, fewer tests that combine parts in the middle, and only a few slow tests through the whole system at the top. The shape follows from cost: a unit test of the cancellation rule runs in a millisecond and points at the broken line, while a browser test of the same rule takes seconds, needs a running application and a database, and when it fails can mean almost anything. Both are needed, in the right proportions.

A test pyramid in four bands. The wide base is unit tests of the domain and use cases, about 240, run in seconds. Above it, slice tests of controllers with MockMvc and repositories with Testcontainers, about 70. Above that, integration tests of the whole application with PostgreSQL in a container, about 25. At the narrow top, Playwright browser tests with axe-core accessibility checks, 12. Beside the pyramid, a separate column for checks that are not tests in the same sense: the contract test, the architecture rules, the load test and manual screen reader checks.
The team's tests at the end of sprint 4.
The team's automated tests on July 26, 2028.
KindWhat it checksToolCountTime in CI
UnitDomain rules and use cases with in-memory adapters and a fixed clockJUnit 6, AssertJ2416 s
Web sliceControllers, forms, validation messages, redirects, security rulesMockMvc449 s
Repository sliceQueries, migrations and constraints on real PostgreSQL@DataJpaTest, Testcontainers2724 s
IntegrationThe whole application: book, cancel, outbox, reminder sender@SpringBootTest, Testcontainers2538 s
Browser and accessibilityReal pages in Chromium, keyboard use, axe-core rulesPlaywright, axe-core1271 s
Contract and architectureThe 1.5.0 counter app contract; the dependency ruleJUnit, ArchUnit194 s

The table has no row for manual testing, which is not because the team did none. Exploratory testing, in which a person tries to break the software with no script, and checks with a real screen reader were part of every page story. But they cannot run on every pull request, so the strategy was to automate everything that a machine can judge and spend human attention on what it cannot: whether a page makes sense, whether an error message helps, whether the order a screen reader reads things in matches the order on the screen.

Test helpers

Good tests read like the acceptance criteria they check. That takes some helpers, and the team wrote them early so that every new test was cheap. A test data builder creates a valid booking with sensible defaults, so a test states only the fields that matter to it. A fake notifier records the messages it was asked to send instead of sending them. And a fixed clock makes now whatever the test says, so a test about reminders the evening before does not depend on when it runs. The example puts the three together in plain Java with a tiny check function in place of JUnit's assertions.

Java Example
import java.time.*;
import java.util.*;

public class Main {
    record Booking(String reference, String phone, LocalDateTime start, boolean remind, String status) {}

    static class BookingBuilder {
        private String reference = "K7QF-29XM", phone = "+15555550142", status = "BOOKED";
        private LocalDateTime start = LocalDateTime.of(2028, 7, 22, 10, 30);
        private boolean remind = true;
        BookingBuilder startingAt(LocalDateTime t) { start = t; return this; }
        BookingBuilder withoutReminder() { remind = false; return this; }
        BookingBuilder withStatus(String s) { status = s; return this; }
        Booking build() { return new Booking(reference, phone, start, remind, status); }
    }
    static BookingBuilder aBooking() { return new BookingBuilder(); }

    static class FakeNotifier { final List<String> sent = new ArrayList<>(); void text(String phone, String body) { sent.add(phone + ": " + body); } }

    static void sendDueReminders(List<Booking> bookings, Clock clock, FakeNotifier n) {
        LocalDateTime now = LocalDateTime.now(clock);
        for (Booking b : bookings) {
            LocalDateTime due = b.start().toLocalDate().minusDays(1).atTime(18, 0);
            if (b.remind() && b.status().equals("BOOKED") && !now.isBefore(due) && now.isBefore(b.start()))
                n.text(b.phone(), "Reminder: repair at Second Gear tomorrow at " + b.start().toLocalTime());
        }
    }

    static int failures = 0;
    static void check(String name, boolean ok) { System.out.println((ok ? "pass " : "FAIL ") + name); if (!ok) failures++; }

    public static void main(String[] args) {
        ZoneId zone = ZoneId.of("UTC");
        Clock fridayEvening = Clock.fixed(LocalDateTime.of(2028, 7, 21, 18, 1).toInstant(ZoneOffset.UTC), zone);
        Clock fridayAfternoon = Clock.fixed(LocalDateTime.of(2028, 7, 21, 17, 59).toInstant(ZoneOffset.UTC), zone);

        FakeNotifier n1 = new FakeNotifier();
        sendDueReminders(List.of(aBooking().build()), fridayEvening, n1);
        check("US-07.1 opted-in booking is reminded after 18:00 the evening before", n1.sent.size() == 1);

        FakeNotifier n2 = new FakeNotifier();
        sendDueReminders(List.of(aBooking().build()), fridayAfternoon, n2);
        check("US-07.1 no reminder before 18:00", n2.sent.isEmpty());

        FakeNotifier n3 = new FakeNotifier();
        sendDueReminders(List.of(aBooking().withoutReminder().build(), aBooking().withStatus("CANCELLED").build()), fridayEvening, n3);
        check("US-07.2 no reminder without consent or after a cancel", n3.sent.isEmpty());

        System.out.println(n1.sent);
        System.out.println(failures == 0 ? "all passed" : failures + " failed");
    }
}
Output
pass US-07.1 opted-in booking is reminded after 18:00 the evening before
pass US-07.1 no reminder before 18:00
pass US-07.2 no reminder without consent or after a cancel
[+15555550142: Reminder: repair at Second Gear tomorrow at 10:30]
all passed

What this code does

The builder starts from a valid opted-in booking for Saturday, July 22, so each test changes only what it is about, such as removing the consent or marking the booking cancelled. Two fixed clocks a minute either side of 18:00 on Friday test the boundary of the rule, and the fake notifier lets the test count the messages instead of sending any. Each check is named with the acceptance criterion it covers, as the real JUnit tests are, so the traceability check can read them.

The real versions are the same ideas with JUnit 6 and AssertJ. The builder lives in the test sources and is shared by unit and integration tests; the fake notifier is a bean that replaces the SMS adapter in the test profile; and the fixed clock is a Clock bean, which is why every class in the application takes a Clock in its constructor rather than calling now() itself. Thierry's rule in reviews was blunt: a call to LocalDateTime.now() without a clock argument in main code is a bug, even if no test has caught it yet.

A flaky test, and what it taught

In the second week of the sprint, one integration test failed on Thierry's pull request and passed when he ran it again. A test that sometimes fails without any change to the code is flaky, and flaky tests are corrosive: once people learn that a red build might mean nothing, they stop believing red builds. The team's rule was that a flaky test is a priority bug, never something to rerun and forget.

The test checked that a booking could be made for tomorrow's first free start. Hamza found the cause by reading it: the test used the system clock, not the fixed one, so tomorrow depended on the day it ran. The CI server runs in UTC, and the test had failed on a Friday evening in Hamilton, when it was already Saturday in UTC, so tomorrow was Sunday, the shop was closed, and there was no first free start. The example runs the same test logic for each day of one week.

Java Example
import java.time.*;
import java.util.*;

public class Main {
    static boolean open(LocalDate d) { return d.getDayOfWeek().getValue() >= 2 && d.getDayOfWeek().getValue() <= 6; }   // Tue..Sat

    static Optional<LocalDateTime> firstFreeStartTomorrow(LocalDate today) {
        LocalDate tomorrow = today.plusDays(1);
        return open(tomorrow) ? Optional.of(tomorrow.atTime(9, 0)) : Optional.empty();
    }

    static Optional<LocalDateTime> firstFreeStartAfter(LocalDate today) {
        for (LocalDate d = today.plusDays(1); d.isBefore(today.plusDays(15)); d = d.plusDays(1))
            if (open(d)) return Optional.of(d.atTime(9, 0));
        return Optional.empty();
    }

    public static void main(String[] args) {
        LocalDate monday = LocalDate.of(2028, 7, 17);
        System.out.println("old test, which assumed tomorrow is open:");
        for (int i = 0; i < 7; i++) {
            LocalDate d = monday.plusDays(i);
            System.out.println("  run on " + d.getDayOfWeek() + ": " + firstFreeStartTomorrow(d).map(t -> "pass").orElse("FAIL (no start tomorrow)"));
        }
        System.out.println("new test, fixed clock on Friday July 21, asks for the next open day:");
        System.out.println("  " + firstFreeStartAfter(LocalDate.of(2028, 7, 21)).orElseThrow());
        System.out.println("  and from Saturday July 22: " + firstFreeStartAfter(LocalDate.of(2028, 7, 22)).orElseThrow());
    }
}
Output
old test, which assumed tomorrow is open:
  run on MONDAY: pass
  run on TUESDAY: pass
  run on WEDNESDAY: pass
  run on THURSDAY: pass
  run on FRIDAY: pass
  run on SATURDAY: FAIL (no start tomorrow)
  run on SUNDAY: FAIL (no start tomorrow)
new test, fixed clock on Friday July 21, asks for the next open day:
  2028-07-22T09:00
  and from Saturday July 22: 2028-07-25T09:00

What this code does

The old test asks for a start tomorrow, which exists only when tomorrow is a day the shop is open, Tuesday to Saturday, so it fails when run on a Saturday or a Sunday by the clock it reads. In CI that included Friday evenings in the shop's time zone, because the server's date was already Saturday in UTC. The new test fixes the clock and asks for the next open day, so it has one answer that never depends on when it runs.

The fix took ten minutes, and the lesson took longer to sink in. Thierry added an ArchUnit rule that fails the build if main code calls the no-argument now() methods of the java.time classes, and Hamza searched the test sources for the same mistake and found two more tests that would have failed on the night of the clocks changing in November. None of them would have been found by rerunning.

Every criterion, traced

The traceability check from lesson 2 now had real tests to read. It ran in the build over the display names of every test, and from sprint 4 on it failed the build if any criterion of a Must story had no test. The output for the end of the sprint showed one gap, which was a good result: it was in a Should story still being built, and it was exactly the work left on the board.

Java Example
import java.util.*;
import java.util.regex.*;

public class Main {
    public static void main(String[] args) {
        Map<String, String> priority = new LinkedHashMap<>();
        Map<String, Integer> criteria = new LinkedHashMap<>();
        String[][] stories = {{"US-01", "MUST", "3"}, {"US-02", "MUST", "6"}, {"US-03", "MUST", "4"}, {"US-04", "MUST", "3"},
                              {"US-05", "MUST", "5"}, {"US-06", "MUST", "4"}, {"US-07", "SHOULD", "3"}, {"US-08", "SHOULD", "5"},
                              {"US-09", "SHOULD", "2"}};
        for (String[] s : stories) { priority.put(s[0], s[1]); criteria.put(s[0], Integer.parseInt(s[2])); }

        List<String> testNames = new ArrayList<>();
        int[][] testedUpTo = {{1, 3}, {2, 6}, {3, 4}, {4, 3}, {5, 5}, {6, 4}, {7, 3}, {8, 3}, {9, 2}};
        for (int[] t : testedUpTo)
            for (int c = 1; c <= t[1]; c++) testNames.add(String.format("US-%02d.%d some behavior", t[0], c));
        testNames.add("US-03.3 wrong reference gives the same page");   // a second test for one criterion

        Pattern id = Pattern.compile("^(US-\\d{2})\\.(\\d+) ");
        Map<String, Set<Integer>> covered = new TreeMap<>();
        for (String n : testNames) {
            Matcher m = id.matcher(n);
            if (m.find()) covered.computeIfAbsent(m.group(1), k -> new TreeSet<>()).add(Integer.parseInt(m.group(2)));
        }
        boolean mustGap = false;
        for (var e : criteria.entrySet()) {
            Set<Integer> got = covered.getOrDefault(e.getKey(), Set.of());
            List<String> missing = new ArrayList<>();
            for (int c = 1; c <= e.getValue(); c++) if (!got.contains(c)) missing.add(e.getKey() + "." + c);
            if (!missing.isEmpty() && priority.get(e.getKey()).equals("MUST")) mustGap = true;
            System.out.printf("%s %-6s %d/%d%s%n", e.getKey(), priority.get(e.getKey()), got.size(), e.getValue(),
                missing.isEmpty() ? "" : "  missing " + missing);
        }
        System.out.println(testNames.size() + " criterion tests; build " + (mustGap ? "FAILS" : "passes"));
    }
}
Output
US-01 MUST   3/3
US-02 MUST   6/6
US-03 MUST   4/4
US-04 MUST   3/3
US-05 MUST   5/5
US-06 MUST   4/4
US-07 SHOULD 3/3
US-08 SHOULD 3/5  missing [US-08.4, US-08.5]
US-09 SHOULD 2/2
34 criterion tests; build passes

What this code does

The stories table holds each story's priority and number of criteria, and the test names are generated here to stand in for the real ones, which the build reads from the JUnit report. A TreeSet per story collects the criterion numbers that at least one test covers, so a second test for US-03.3 does not count twice. US-08's last two criteria are missing, but because it is a Should story the build passes; the same gap in a Must story would fail it.

Accessibility, automated and by hand

N1 required WCAG 2.2 level AA. The team checked it in three ways. axe-core, an open-source rules engine from Deque Systems, ran inside the Playwright browser tests against every page in several states, such as the booking form with errors shown, and failed the build on any violation. Noemi tested each changed page with the keyboard alone and with a screen reader, NVDA on Windows and VoiceOver on her phone. And once a sprint someone checked every page at the largest text size on a phone, the rule from the last retrospective.

The three were needed because automated tools can decide only some of the success criteria. A tool can tell that an image has no text alternative, or that a form field has no label, or that text has too little contrast with its background. It cannot tell whether the alternative text describes the image, whether the label makes sense, or whether the reading order matches the meaning. Studies by tool makers and accessibility groups agree that automated checks find only a minority of the problems a skilled human tester finds. Noemi wrote a small checker for a few of the rules axe-core applies, to show the team what such a rule looks like from the inside.

Java Example
import java.util.*;
import java.util.regex.*;

public class Main {
    static List<String> check(String html) {
        List<String> problems = new ArrayList<>();
        if (!Pattern.compile("<html[^>]*\\blang=\"[a-z]{2}").matcher(html).find()) problems.add("html element has no lang");
        Matcher img = Pattern.compile("<img(?![^>]*\\balt=)[^>]*>").matcher(html);
        while (img.find()) problems.add("image without alt: " + img.group());
        Set<String> withLabel = new HashSet<>();
        Matcher lab = Pattern.compile("<label[^>]*\\bfor=\"([^\"]+)\"").matcher(html);
        while (lab.find()) withLabel.add(lab.group(1));
        Matcher input = Pattern.compile("<input(?![^>]*type=\"(?:hidden|submit)\")[^>]*\\bid=\"([^\"]+)\"[^>]*>").matcher(html);
        while (input.find()) if (!withLabel.contains(input.group(1))) problems.add("input without label: #" + input.group(1));
        Matcher h = Pattern.compile("<h([1-6])").matcher(html);
        int last = 0;
        while (h.find()) {
            int level = Integer.parseInt(h.group(1));
            if (last > 0 && level > last + 1) problems.add("heading jumps from h" + last + " to h" + level);
            last = level;
        }
        Matcher a = Pattern.compile("<a [^>]*>(click here|here|more)</a>", Pattern.CASE_INSENSITIVE).matcher(html);
        while (a.find()) problems.add("link text says nothing: '" + a.group(1) + "'");
        return problems;
    }

    public static void main(String[] args) {
        String draft = """
            <html><title>Book</title><h1>Book a repair</h1><h3>Your details</h3>
            <img src="logo.png"><label for="name">Name</label><input id="name" name="name">
            <input id="phone" name="phone" placeholder="Phone"><input type="hidden" id="csrf" name="_csrf">
            <p>To cancel, <a href="/my-booking">click here</a>.</p></html>""";
        String fixed = """
            <html lang="en"><title>Book</title><h1>Book a repair</h1><h2>Your details</h2>
            <img src="logo.png" alt="Second Gear Cycles"><label for="name">Name</label><input id="name" name="name">
            <label for="phone">Phone number</label><input id="phone" name="phone" autocomplete="tel">
            <input type="hidden" id="csrf" name="_csrf"><p><a href="/my-booking">Find or cancel your booking</a></p></html>""";
        System.out.println("draft:");
        check(draft).forEach(p -> System.out.println("  " + p));
        System.out.println("fixed: " + (check(fixed).isEmpty() ? "no problems found by these rules" : check(fixed)));
    }
}
Output
draft:
  html element has no lang
  image without alt: <img src="logo.png">
  input without label: #phone
  heading jumps from h1 to h3
  link text says nothing: 'click here'
fixed: no problems found by these rules

What this code does

Each rule is a pattern over the HTML: the page language, images without an alt attribute, inputs whose id no label points to, headings that skip a level and links whose text means nothing out of context. Hidden inputs and submit buttons are excluded from the label rule because they are not fields a person fills in. The fixed page passes these five rules, which proves only that it avoids five mistakes; a real engine such as axe-core works on the rendered page, not the text, and checks many more.

axe-core in the browser tests found two real problems that sprint. The error summary's links were announced but moved focus to the field's wrapper rather than the input, and the grey used for helper text under each field had a contrast ratio of 3.9 to 1 against the white background, below the 4.5 to 1 that WCAG requires for normal text. Noemi darkened the grey and fixed the links in an afternoon. The screen reader check found something no tool could: the problem kinds were read in the order of the source, Flat tire first, while the visual layout put Tune-up first because of a CSS grid, so a sighted helper and a screen reader user saw different lists.

Representative output
./gradlew browserTest --tests '*Accessibility*'
Output
> Task :browserTest

AccessibilityTest > free times page has no violations() PASSED
AccessibilityTest > booking form has no violations() PASSED
AccessibilityTest > booking form with errors has no violations() FAILED
    color-contrast (serious): Elements must meet minimum color contrast ratio thresholds
      .hint: Element has insufficient color contrast of 3.9 (foreground #8a8f94, background #ffffff,
             font size 12.0pt, font weight normal). Expected contrast ratio of 4.5:1
AccessibilityTest > my-booking page has no violations() PASSED
AccessibilityTest > staff day page has no violations() PASSED

5 tests completed, 1 failed

What this code does

The browser tests are a separate Gradle task so that developers can run the fast tests constantly and these before pushing. Each test opens a page in Chromium through Playwright, puts it in a state, runs axe-core with the WCAG 2.2 A and AA rule tags and fails with the list of violations. The output below is representative of a run on July 19, before the contrast fix.

Is it fast enough?

N3 asked for a 95th percentile under 300 ms. The budget in lesson 3 said the design should manage it comfortably, but a budget is a prediction. Thierry wrote a small load generator in Java, using virtual threads from the concurrency course to run many simulated customers at once, and pointed it at staging, which runs on the same kind of server as production, with generated data. The simulated customers followed the mix the team expected on a busy spring morning: mostly loading the free-times page, some booking forms, a few lookups and the occasional booking.

The latency numbers that came back needed summarizing correctly. A mean hides the slow tail that the objective is about, so the program reports percentiles: the p95 is the time that 95 requests out of every 100 beat. The example computes them, using the nearest-rank method, from a seeded sample with the same shape as the staging run, a fast body and a slow tail caused by the occasional garbage collection pause and database checkpoint.

Java Example
import java.util.*;

public class Main {
    static long percentile(long[] sorted, int p) {
        int rank = (int) Math.ceil(p / 100.0 * sorted.length);   // nearest-rank method
        return sorted[Math.max(0, rank - 1)];
    }

    public static void main(String[] args) {
        Random rnd = new Random(290);
        int n = 5000;
        long[] ms = new long[n];
        for (int i = 0; i < n; i++) {
            double base = 18 + 10 * Math.abs(rnd.nextGaussian());   // most requests
            if (rnd.nextInt(100) < 3) base += 80 + rnd.nextInt(220);  // a slow tail: GC pauses, checkpoints
            ms[i] = Math.round(base);
        }
        Arrays.sort(ms);
        double mean = Arrays.stream(ms).average().orElse(0);
        System.out.printf("requests %d, mean %.1f ms%n", n, mean);
        for (int p : new int[] {50, 90, 95, 99})
            System.out.println("p" + p + " " + percentile(ms, p) + " ms");
        System.out.println("max " + ms[n - 1] + " ms");
        long p95 = percentile(ms, 95);
        System.out.println("objective p95 < 300 ms: " + (p95 < 300 ? "met" : "MISSED") + " with " + (300 - p95) + " ms to spare");
    }
}
Output
requests 5000, mean 31.8 ms
p50 25 ms
p90 36 ms
p95 42 ms
p99 259 ms
max 325 ms
objective p95 < 300 ms: met with 258 ms to spare

What this code does

The generator draws most latencies from a narrow distribution around 18 to 40 ms and adds, for about three requests in a hundred, an extra delay of 80 to 300 ms, which is the shape of the real measurements. Sorting the sample and taking the value at the nearest rank gives each percentile, so the p95 is the 4,750th fastest of 5,000. The mean looks excellent and says nothing about the tail, which is why the objective uses the p95 and why the p99 is worth watching too.

Representative output
java -cp build/load LoadRun --base https://staging.book.secondgear.org --users 40 --minutes 10
Output
mix: free-times 70, book form 15, lookups 10, bookings 5 (per 100 requests)
requests 23,914 in 600 s (39.9 per second), errors 0
latency p50 27 ms   p90 41 ms   p95 49 ms   p99 212 ms   max 418 ms
by page p95: free-times 44 ms, book form 38 ms, booking POST 96 ms, my-booking POST 61 ms

What this code does

The real run used 40 simulated customers for ten minutes against staging, several times the busiest hour Odile could remember. Latencies are measured by the client, so they include the network and Caddy, which is what a customer experiences. The output is representative of the run on July 24, recorded in the team's notes rather than from the course machine.

The p95 of 49 ms left a wide margin. The booking POST was the slowest page because it writes the booking and two outbox rows in one transaction and waits for the commit, and that was expected. The p99, over 200 ms, came from garbage collection pauses on the 768 MB container that the deployment course had sized, and Leif said that was fine for now but worth a look if traffic grew. The team stopped there: performance work beyond the objective is time taken from something else.

Quality gates and coverage

The pipeline from the deployment course already blocked a merge on failing tests or a likely secret. Sprint 4 added the website's gates, so that the Definition of Done was checked by machines wherever a machine could check it. The order mattered as before: cheap checks first, so a developer learns about a formatting slip in seconds, not after five minutes of browser tests.

The pipeline's gates for the website.
GateFails the pull request whenAdded
Compile and unit testsAny test failsExisting
Architecture rulesThe dependency rule is broken or main code calls now() without a clockSprint 1, sprint 4
TraceabilityA Must criterion has no testSprint 4
Slice and integration testsAny test fails on PostgreSQL in a containerExisting
Contract testThe counter app's 1.5.0 contract is brokenSprint 3
Browser and accessibility testsA page fails a Playwright test or an axe-core WCAG A or AA ruleSprint 4
Secret scanA likely secret appears in the diffExisting
Coverage reportNever fails; the report is attached for reviewersSprint 4

The last row was a deliberate choice. JaCoCo measured that the tests executed 87 lines in every 100 of the main code, and the team was tempted to fail the build below some number. Ingrid argued against it from experience. A coverage threshold is easy to satisfy with tests that run code without checking anything, and teams under pressure do exactly that. Coverage is useful as a map of what no test touches, which a reviewer can read; it is a poor target. The team attached the report to every pull request and looked at the uncovered lines in review instead.

Bugs found and fixed

Exploratory testing in the last days of the sprint found the sprint's most interesting bug. Hamza booked two starts on the same Saturday with the same phone, joined the waiting list for that Saturday out of curiosity, cancelled one booking, and received an offer for his own freed start. Nothing was technically broken, but it made no sense: someone who already had a booking that day was not really waiting. The team decided, with Odile, that joining the waiting list requires having no active booking that day, and that an offer is skipped for anyone who has booked that day since joining. It became a new acceptance criterion, US-08.6, with its test, which is how a bug found by a person becomes a rule the machine keeps.

At the review on Wednesday, July 26, Odile opened the figures page, US-09, on staging and saw bookings, cancellations and missed appointments for the generated weeks, with the share of bookings made on the web. Kofi asked whether it could show the part warnings that had been issued; it could not yet, and that went on the Later page. The sprint finished 9 points, the waiting list was complete apart from its last two criteria, and the website, as far as tests could tell, was ready for someone to try to break it on purpose.

Exercises

Try it yourself

Nearest-rank percentiles

Complete percentile using the nearest-rank method: sort a copy of the values, take rank = ceiling of p/100 times the count, and return the value at position rank - 1 (or the first value if rank is 0). Print p50, p95 and p99 for the given latencies.

import java.util.*;

public class Main {
    static long percentile(long[] values, int p) {
        return values[0];
    }

    public static void main(String[] args) {
        long[] ms = {31, 22, 18, 240, 25, 19, 27, 33, 21, 26, 24, 30, 29, 20, 23, 28, 35, 41, 37, 120};
        for (int p : new int[] {50, 95, 99}) System.out.println("p" + p + " " + percentile(ms, p) + " ms");
    }
}

Try it yourself

Every field needs a label

Complete withoutLabel so that it returns the ids of <input> elements that have an id but no <label for="..."> pointing to it, ignoring inputs whose type is hidden or submit, in the order they appear.

import java.util.*;
import java.util.regex.*;

public class Main {
    static List<String> withoutLabel(String html) {
        return List.of();
    }

    public static void main(String[] args) {
        String html = "<label for=\"name\">Name</label><input id=\"name\">"
            + "<input id=\"phone\" placeholder=\"Phone\"><input type=\"hidden\" id=\"csrf\">"
            + "<label for=\"note\">Note</label><input id=\"note\"><input id=\"remind\" type=\"checkbox\">"
            + "<input type=\"submit\" id=\"go\">";
        System.out.println("without a label: " + withoutLabel(html));
    }
}

Common mistakes

  • MistakeTesting everything through the browser.

    InsteadKeep most tests small and fast, with fewer integration and browser tests, in a pyramid.

  • MistakeCalling now() in code and tests.

    InsteadInject a Clock everywhere and fix it in tests; enforce the rule with an architecture test.

  • MistakeRerunning a flaky test until it passes.

    InsteadTreat flakiness as a bug and find the cause, which is often time, order or shared state.

  • MistakeTrusting an automated accessibility tool alone.

    InsteadCombine automated rules with keyboard, screen reader and large-text checks by a person.

  • MistakeReporting mean response times.

    InsteadReport percentiles, especially the one in the objective, and watch the tail.

  • MistakeFailing the build below a coverage number.

    InsteadUse coverage as a map of untested code for reviewers, not as a target.

Summary and key terms

  • Sprint 4 made quality part of the goal.
  • The team's tests form a pyramid: many unit tests of domain rules and use cases with builders, fakes and a fixed clock, fewer slice tests with MockMvc and Testcontainers, a few integration tests of the whole application and twelve Playwright browser tests with axe-core, plus contract and architecture checks.
  • A flaky test caused by the system clock on a UTC build server led to a rule, enforced by ArchUnit, that main code never calls now() without a clock.
  • The traceability check now fails the build when a Must criterion has no test.
  • Accessibility is checked by axe-core in the browser, which found a contrast failure, and by people with the keyboard, screen readers and large text, who found a reading-order problem no tool could.
  • A load test against staging measured a p95 of 49 ms against the 300 ms objective, reported as percentiles rather than a mean.
  • New gates were added to the pipeline, while coverage stays a report for reviewers rather than a threshold.
  • Exploratory testing turned a waiting-list oddity into a new criterion.
Test pyramid
A suite with many fast unit tests, fewer integration tests and few end-to-end tests.
Test data builder
A helper that creates valid test objects with defaults, so tests state only what matters.
Fake
A working substitute for a dependency, such as a notifier that records messages instead of sending them.
Flaky test
A test that sometimes fails without any change to the code.
Exploratory testing
Testing by a person who designs and runs tests while learning the system, without a script.
axe-core
An open-source engine that checks web pages against accessibility rules.
Contrast ratio
The ratio of the lightness of text to its background; WCAG AA requires 4.5 to 1 for normal text.
Load test
Sending many simulated requests to measure how a system performs under traffic.
Percentile
The value below which a given share of measurements fall, such as the p95.
Quality gate
An automated check in the pipeline that must pass before a change can merge.

Practice problems

Work each problem on paper first, then reveal the answer to check.

  1. Why are most of the team's tests unit tests?

    Show answer

    They are fast and point at the broken rule, while browser tests are slow and their failures are harder to interpret.

  2. What made the tomorrow test flaky?

    Show answer

    It read the system clock, and on a UTC build server late on a Friday it was already Saturday, so tomorrow was a closed Sunday.

  3. What could the screen reader check find that axe-core could not?

    Show answer

    That the problem kinds were read in a different order from the one shown on screen.

  4. Why does the load test report percentiles?

    Show answer

    The objective is about the slow tail, which a mean hides.

  5. Why does the coverage report never fail the build?

    Show answer

    A threshold invites tests that run code without checking it; the report is more useful as a map for reviewers.

Lesson quiz · 5 questions

Score 60% or more to complete this lesson.

Take the quiz