JS-290 · Lesson 8 of 12 · 27 min

Testing and quality

▶ Video coming soon A video version of this lesson is on the way.

Tuesday, May 1, is the start of Sprint 3, and Tavish arrives with a bug. On Saturday he had been reading the claim rules with the seed data loaded, and he noticed that a driver had been allowed to claim a run from 23:30 to 00:30 and another from 23:45 to 00:15 on the same night. Every acceptance example from lesson 2 had passed. The rule compared times as strings, *HH:MM*, which works perfectly until a window crosses midnight, when 00:30 sorts before 23:30. Nobody had written an example with a window that crosses midnight, because no donor does pickups at midnight. Except that Loaf and Ladle, a café that closes late on Fridays, had asked for exactly that during the April demo.

The bug is the best possible start to a lesson on testing, because it shows what tests are for and where they fall short. Tests did their job: the examples Lucy agreed to all pass and will keep passing. But tests only check what someone thought to check. This lesson covers how the team decides what to test at each level, how acceptance examples become automated tests, how to generate cases nobody thought of, how to test with real people, and what *quality* means beyond passing tests.

A test strategy in layers

The team's strategy follows the shape JS-210 introduced, often drawn as a test pyramid: many small, fast unit tests at the bottom, fewer integration tests in the middle, and a handful of slow end-to-end tests at the top. The shape is a guide to cost. A unit test of the claiming rules runs in a millisecond and points at the exact line that broke. An end-to-end test that drives a browser through posting, claiming and receiving takes ten seconds, can fail for reasons unrelated to the change, and says only that something somewhere is wrong. Both are needed, in different numbers.

A test pyramid with four layers and test counts at the end of Sprint 3. At the bottom, 214 unit tests for rules, conversions and reducers. Above, 63 integration tests against a real PostgreSQL database. Above that, 38 component tests with Testing Library. At the top, 6 end-to-end journeys with Playwright. Beside the pyramid, an accessibility check runs in every end-to-end journey.
The team's tests at the end of Sprint 3
The test strategy
LevelWhat it testsToolsRuns
UnitClaiming rules, state machine, conversions, reducers, validatorsnode:test for the service, Vitest for the web appEvery save on a laptop and every push
IntegrationRepositories and handlers against a real PostgreSQL 18node:test with the PostgreSQL service container in CIEvery push
ComponentCards, forms and pages with fake dataVitest and Testing LibraryEvery push
ContractWeb app requests match the API's schemasShared zod schemas in riverside-dataEvery push
End-to-endSix journeys from posting to receiving, on staging-like dataPlaywright with an accessibility checkEvery pull request and after each deployment
Exploratory and usabilityWhat nobody thought to scriptPeople: Wendy, Copperleaf staff, BeatrixWeekly demos and two sessions

Acceptance examples become tests

The Given, When, Then examples from lesson 2 were written as data on purpose. Tavish moves them into a file that the service's test suite runs with node:test, the test runner built into Node that the studio uses for its services. Each example becomes one test, named after the rule it checks, so that a failure reads as a sentence Lucy could understand.

JavaScript Example · test/claims.test.mjs
import { test } from 'node:test';
import assert from 'node:assert/strict';

// The rule under test (in the service this is imported from src/services/claims.js).
const ms = (iso) => Date.parse(iso);
const overlaps = (a, b) => ms(a.start) < ms(b.end) && ms(b.start) < ms(a.end);
function canClaim(driver, run, mine) {
  if (run.claimedBy) return 'already claimed';
  if (run.chilled && !driver.cooler) return 'needs a cooler';
  const day = run.start.slice(0, 10);
  if (mine.filter((m) => m.start.slice(0, 10) === day).length >= 3) return 'three runs that day already';
  if (mine.some((m) => overlaps(m, run))) return 'overlaps another run';
  return 'ok';
}

const wendy = { cooler: true };
const at = (d, t) => `2029-04-${d}T${t}:00-04:00`;
const mine = [{ start: at(10, '16:00'), end: at(10, '17:00') }];

test('touching windows may both be claimed', () => {
  assert.equal(canClaim(wendy, { start: at(10, '17:00'), end: at(10, '18:00') }, mine), 'ok');
});
test('overlapping windows are refused', () => {
  assert.equal(canClaim(wendy, { start: at(10, '16:30'), end: at(10, '17:30') }, mine), 'overlaps another run');
});
test('a run claimed by someone else cannot be claimed', () => {
  assert.equal(canClaim(wendy, { start: at(10, '19:00'), end: at(10, '19:30'), claimedBy: 'raj' }, mine), 'already claimed');
});
test('chilled food needs a cooler, and that reason comes first', () => {
  assert.equal(canClaim({ cooler: false }, { start: at(10, '16:30'), end: at(10, '17:30'), chilled: true }, mine), 'needs a cooler');
});
test('windows that cross midnight are compared as instants', () => {
  const late = [{ start: at(13, '23:30'), end: at(14, '00:30') }];
  assert.equal(canClaim(wendy, { start: at(13, '23:45'), end: at(14, '00:15') }, late), 'overlaps another run');
});

What this code does

Each Given, When, Then example from the requirements becomes one test whose name states the rule in plain words. The rule now compares windows as instants parsed from full timestamps with offsets, so the last test, the midnight case Tavish found, passes. Running the file with node --test prints each test by name, which is the report a reviewer or Lucy can read.

Running it
$ node --test --test-reporter=spec claims.test.mjs
✔ touching windows may both be claimed (0.4ms)
✔ overlapping windows are refused (0.4ms)
✔ a run claimed by someone else cannot be claimed (0.4ms)
✔ chilled food needs a cooler, and that reason comes first (0.4ms)
✔ windows that cross midnight are compared as instants (0.4ms)
ℹ tests 5
ℹ suites 0
ℹ pass 5
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 0.4

The fix itself is two lines: the rule now compares full timestamps parsed to milliseconds instead of time strings, and the in-memory repository stores windows as timestamps, as the PostgreSQL schema already did. The real database would have caught the double claim with its exclusion constraint, which is defence in depth working as intended; but the service would have shown the driver a database error instead of a clear message, and the unit tests would have kept passing with a wrong rule.

Generating the cases nobody wrote

After the midnight bug, Gareth shows the team property-based testing. Instead of writing examples one at a time, you state a property that must hold for all inputs, then let a generator produce hundreds of random inputs and check the property on each. For the overlap rule, two properties are obvious once stated: overlap is symmetric, so if A overlaps B then B overlaps A; and windows that only touch never overlap. A seeded generator, the same technique as the seed data in lesson 5, makes every run reproducible, so a failing case can be printed and turned into an ordinary example test.

JavaScript Example · Properties catch the midnight bug
// Property tests for the overlap rule, with a seeded generator.
function mulberry32(seed) {
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
const rand = mulberry32(7);
const base = Date.UTC(2029, 3, 13, 20, 0);
const makeWindow = () => {
  const start = base + Math.floor(rand() * 48) * 15 * 60000; // any quarter hour over 12 hours
  return { start, end: start + (1 + Math.floor(rand() * 8)) * 15 * 60000 };
};
const fmt = (t) => new Date(t - 4 * 3600000).toISOString().slice(11, 16); // shown as local HH:MM

const byInstant = (a, b) => a.start < b.end && b.start < a.end;
const byString = (a, b) => fmt(a.start) < fmt(b.end) && fmt(b.start) < fmt(a.end);

for (const [name, rule] of [['instants', byInstant], ['HH:MM strings', byString]]) {
  let failure = null;
  for (let i = 0; i < 500 && !failure; i++) {
    const a = makeWindow(), b = makeWindow();
    const touching = { start: a.end, end: a.end + 3600000 };
    if (rule(a, b) !== rule(b, a)) failure = 'not symmetric';
    else if (rule(a, touching)) failure = 'touching windows overlap';
    else if (rule(a, b) !== byInstant(a, b)) failure = `wrong for ${fmt(a.start)}-${fmt(a.end)} and ${fmt(b.start)}-${fmt(b.end)}`;
  }
  console.log(`${name.padEnd(14)} ${failure ?? 'all 500 cases pass'}`);
}
Output
instants       all 500 cases pass
HH:MM strings  wrong for 23:00-00:45 and 23:15-01:00

What this code does

The generator produces windows that start at any quarter hour over twelve hours from 4:00 p.m. local time, so many of them cross midnight. Each rule is checked for symmetry, for treating touching windows as separate, and against the correct rule on the same pair. The string comparison passes the first two properties but fails the third as soon as a window crosses midnight, which is exactly the bug found by hand.

The team keeps four property tests in the suite: overlap symmetry, touching windows, the unit conversions round-tripping within one gram, and the state machine never reaching a state outside its list from any sequence of events. They run 500 cases each in under a second. When one fails, the printed seed and case go straight into a new example test, so the case is never forgotten.

Test data builders

Most tests need an offer, a run or a driver, and writing every field in every test makes tests long and fragile: add a field to offers and forty tests break. The team uses test data builders: small functions that return a valid object with sensible defaults, and take only the fields a test cares about as overrides. A test that checks the cooler rule then says only what matters, *a chilled offer and a driver without a cooler*, and the builder fills in the rest.

JavaScript Example · Builders keep tests short
// Builders return valid objects; tests override only what they care about.
let seq = 0;
const offer = (o = {}) => ({
  id: ++seq, donor: 'Test Bakery', category: 'bakery', estGrams: 10000, chilled: false,
  start: '2029-04-24T17:00:00-04:00', end: '2029-04-24T18:00:00-04:00', status: 'open', ...o,
});
const driver = (d = {}) => ({ id: `driver-${++seq}`, firstName: 'Test', cooler: true, ...d });

const cases = [
  ['cooler rule', offer({ chilled: true, category: 'dairy' }), driver({ cooler: false })],
  ['plain bakery run', offer(), driver()],
];
for (const [name, o, d] of cases) {
  const blocked = o.chilled && !d.cooler;
  console.log(`${name.padEnd(16)} offer ${o.id} ${o.category} chilled=${o.chilled}, ${d.id} cooler=${d.cooler} -> ${blocked ? 'needs a cooler' : 'ok'}`);
}
Output
cooler rule      offer 1 dairy chilled=true, driver-2 cooler=false -> needs a cooler
plain bakery run offer 3 bakery chilled=false, driver-4 cooler=true -> ok

What this code does

Each builder spreads its overrides over a complete default object, so the result is always valid and a test only states the fields it is about. A counter gives every object a unique id, which avoids accidental clashes between tests. When a new required field is added, only the builder changes, not every test that uses it.

Integration, contract and end-to-end tests

Integration tests run the real repositories and handlers against a real PostgreSQL database, because the most important rules, the unique constraint, the exclusion constraint and the conditional claim update, only exist in the database. CI starts PostgreSQL 18 as a service container, as the partner service's workflow does, runs the migrations from empty, and each test runs inside a transaction that is rolled back at the end, so tests cannot see each other's data. One integration test sends two claims for the same offer at the same moment and checks that exactly one succeeds.

Contract tests protect the line between Odalys's code and Mira's. The zod schemas for request and response bodies live in the shared riverside-data package, and both sides import them: the service uses them to validate, and the web app's tests use them to check that every request its API client builds is valid and that its fake server's responses have the real shape. If Mira renames a field, the web app's tests fail on the same pull request, instead of the screen failing on staging.

Six end-to-end journeys run in Playwright against a full stack: a donor posts an offer; a driver claims it; the driver tries to claim an overlapping run and sees the reason; a pickup is marked offline and synced; a receiver records a chilled delivery that is too warm and sees it refused; Lucy finds an unclaimed run and assigns a driver. Every journey ends with an automated accessibility check of the final page, using the same checker Sana's WEB-230 review used, and fails on any serious violation.

JavaScript Example · e2e/claiming.spec.js
import { test, expect } from '@playwright/test';

test('a driver cannot claim an overlapping run and is told why', async ({ page }) => {
  await page.goto('/rescue/runs?area=north-end');
  await page.getByRole('article', { name: 'Copperleaf Bakery' }).getByRole('button', { name: 'Claim this run' }).click();
  await expect(page.getByRole('status')).toHaveText('Run claimed. It is in My runs.');

  await page.getByRole('article', { name: 'Loaf and Ladle' }).getByRole('button', { name: 'Claim this run' }).click();
  await expect(page.getByRole('alert')).toHaveText('This run overlaps another of your runs.');

  await page.getByRole('link', { name: 'My runs' }).click();
  await expect(page.getByRole('heading', { level: 1 })).toBeFocused();
  await expect(page.getByRole('listitem')).toHaveCount(1);
});

What this code does

The journey finds elements by role and accessible name, the way a screen reader user would, so a card without a proper name or a div pretending to be a button would make the test fail. It claims one run, tries an overlapping one and checks the explanation announced in the alert region. After navigating it also checks that focus moved to the main heading, which was one of Sana's portal fixes.

Testing with people

Automated tests check what the team expects. People find what the team did not expect. In Sprint 3 the team runs two short usability tests. In the first, two staff at Copperleaf and one at Pine Hill each post three offers on their own phones while Odalys watches without helping, timing each attempt and noting where they hesitate. The requirement from lesson 2 was under a minute.

JavaScript Example · Did posting take under a minute?
// Seconds to post an offer in the usability test, three tries per person.
const times = {
  'Copperleaf A': [74, 41, 33],
  'Copperleaf B': [58, 36, 29],
  'Pine Hill': [96, 52, 47],
};
const median = (xs) => {
  const s = [...xs].sort((a, b) => a - b), m = Math.floor(s.length / 2);
  return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2;
};

const first = Object.values(times).map((t) => t[0]);
const last = Object.values(times).map((t) => t[2]);
console.log('first try median:', median(first), 's');
console.log('third try median:', median(last), 's');
const over = Object.entries(times).filter(([, t]) => t[2] > 60).map(([p]) => p);
console.log('still over a minute on the third try:', over.length ? over.join(', ') : 'nobody');
Output
first try median: 74 s
third try median: 33 s
still over a minute on the third try: nobody

What this code does

The median of the first tries shows how the form feels to a newcomer, and the median of the third tries shows how it feels once it is familiar. The median is used rather than the mean because one slow attempt, such as Pine Hill's first try, would pull a mean of three numbers a long way. Everyone is under a minute by the third try, so the requirement is met, but the first-try numbers show where to improve.

Watching mattered more than timing. Pine Hill's staff member spent most of her first 96 seconds looking for where to choose *produce*, because the category list was sorted alphabetically and the most common category for grocers sat fourth. The list is now ordered by how often each donor uses each category. Nobody would have found that from a test report.

The second session is Beatrix, invited by Sana for Wednesday, May 9, as part of the review in lesson 9. And throughout Sprint 3, Wendy uses the staging app on her real Tuesday round, with invented offers that mirror her real ones, and reports anything odd by text. Her best find was that the *Picked up* button sat exactly where her thumb rested while holding the phone in a car mount, so she pressed it by accident twice; it now needs a deliberate press and shows an undo option for ten seconds.

What quality means

At the end of Sprint 3 the suite has 321 automated tests and takes four minutes in CI. The number is not the point, and Gareth warns against treating coverage percentages as goals: a test that executes a line without checking its result adds coverage and no safety. The team's measure of quality is the Definition of Done from lesson 4, applied every time, plus three habits. Every bug fixed gets a test that failed before the fix. Flaky tests, tests that sometimes fail for no reason, are fixed or deleted within a day, because a suite people learn to ignore protects nothing. And every Friday the team spends an hour on exploratory testing: one person tries to break one feature, with no script, and writes down what they find.

Worked problem

Choosing the level for each test

Odalys has five things she wants to test: (a) that 40.5 lb converts to 18,371 g; (b) that two simultaneous claims on one offer let exactly one through; (c) that the offer form shows a field error next to the weight box when the API returns 422; (d) that a receiver can record a delivery from the receive screen; (e) that the API client's request body for posting an offer matches the service's schema. Which level should each use?

Show solution
  1. (a) Pure calculation: unit test.
  2. (b) Depends on the database's conditional update and unique constraint: integration test against PostgreSQL.
  3. (c) A component reacting to a known response: component test with a fake API response.
  4. (d) A whole journey through several screens and the API: end-to-end test, or part of an existing journey.
  5. (e) The boundary between the two codebases: contract test with the shared schema.

Answer: Unit, integration, component, end-to-end and contract. Choosing the lowest level that can see the behaviour keeps the suite fast and makes failures point at the right place.

Common mistakes

  • MistakeTrusting a passing suite to mean the rule is right.

    InsteadTests check only what someone wrote; add property tests and exploratory testing for the rest.

  • MistakeComparing times as HH:MM strings.

    InsteadCompare instants, parsed from full timestamps with offsets.

  • MistakeWriting every field of every object in every test.

    InsteadUse test data builders with defaults and overrides.

  • MistakeTesting database rules with an in-memory fake only.

    InsteadRun integration tests against a real PostgreSQL in CI.

  • MistakeKeeping a flaky test because it usually passes.

    InsteadFix or delete it within a day, so failures are always taken seriously.

Summary and key terms

  • A test strategy puts many fast unit tests under fewer integration, component and contract tests and a handful of end-to-end journeys.
  • Acceptance examples written as data become automated tests named after their rules.
  • Tests check only what someone thought of, so property-based tests with a seeded generator check general properties over hundreds of cases; the midnight bug showed why.
  • Test data builders keep tests short and stable.
  • Integration tests run against a real PostgreSQL because key rules live in the database, and contract tests with shared schemas protect the boundary between the web app and the API.
  • End-to-end journeys find elements by role and include an accessibility check.
  • Usability tests with real people find what no script can.
  • Quality means the Definition of Done every time, a test for every fixed bug, no flaky tests, and regular exploratory testing.
Test pyramid
A guide to having many fast low-level tests and fewer slow high-level ones.
Unit test
A test of a small piece of code, such as one function, in isolation.
Integration test
A test of several parts working together, such as a repository with a real database.
Contract test
A test that two sides of an interface agree on the shape of the data they exchange.
End-to-end test
A test that drives the whole system through a user journey.
Property-based testing
Testing a general property against many generated inputs.
Test data builder
A function that returns a valid test object with defaults that a test can override.
Flaky test
A test that sometimes fails without any change to the code.
Exploratory testing
Unscripted testing in which a person tries to find problems in a feature.
Usability test
Watching real users try real tasks to see where they struggle.

Practice problems

Work each problem on paper first, then reveal the answer to check.

  1. Why did all the acceptance examples pass despite the midnight bug?

    Show answer

    None of them had a window that crossed midnight; tests check only what someone wrote.

  2. What two properties of the overlap rule did the team test?

    Show answer

    Overlap is symmetric, and windows that only touch never overlap.

  3. Why are integration tests run against a real PostgreSQL?

    Show answer

    The unique constraint, the exclusion constraint and the conditional update only exist in the database.

  4. Why use the median in the usability timings?

    Show answer

    With only three tries, one slow attempt would pull the mean a long way.

Try it yourself

A test data builder

Complete makeRun(overrides). It returns a run with defaults { id: 1, status: 'claimed', driver: 'test-driver', chilled: false } and any fields in overrides replacing them.

Calling it with no argument must also work.

function makeRun(overrides) {
  return {};
}

console.log(JSON.stringify(makeRun()));
console.log(JSON.stringify(makeRun({ status: 'picked_up', chilled: true })));
console.log(JSON.stringify(makeRun({ id: 7 })));

Try it yourself

Overlap by instants

Complete overlaps(a, b). Each window has start and end as ISO timestamps with offsets, such as 2029-04-13T23:30:00-04:00.

Return true if the windows overlap. Windows that only touch do not overlap. Compare instants, not strings.

function overlaps(a, b) {
  return a.start.slice(11, 16) < b.end.slice(11, 16) && b.start.slice(11, 16) < a.end.slice(11, 16);
}

const w = (s, e) => ({ start: `2029-04-${s}:00-04:00`, end: `2029-04-${e}:00-04:00` });
console.log(overlaps(w('13T23:30', '14T00:30'), w('13T23:45', '14T00:15')));
console.log(overlaps(w('13T16:00', '13T17:00'), w('13T17:00', '13T18:00')));
console.log(overlaps(w('13T16:00', '13T17:00'), w('13T16:30', '13T17:30')));

Lesson quiz · 5 questions

Score 60% or more to complete this lesson.

Take the quiz