Testing and quality

Quality was not a phase at the end of the Ridgeline Ops project. Tests were written with every story, every change was reviewed by another person, and the pipeline from lesson 4 refused to merge anything that failed. Still, the fourth week of the term, with the main features built, was when the team stepped back and asked harder questions. Are we testing the right things? Would we notice if the SQLite repository and the in-memory fake behaved differently? Are our code reviews catching real problems or just approving? What have the staff found that we have not?

This lesson covers the testing strategy the team settled on, how they kept fakes honest, how they reviewed each other's code, and how a morning of testing with real staff found the bugs that automated tests missed. The exercises are about writing tests that catch real bugs and fixing the problems a good review finds.

A testing strategy for the whole system

PYTH-230 introduced the test pyramid: many fast unit tests at the bottom, fewer integration tests in the middle, and a handful of slow end-to-end tests at the top. The capstone team applied it deliberately. At the bottom, about 220 unit tests covered the domain rules and application services using in-memory fakes; together they ran in under two seconds. In the middle, about 40 integration tests ran the SQLite repositories against a real temporary database, the Flask routes through the test client, and the CLI through its main function. At the top, five end-to-end tests drove the whole system as a user would: create a subscription through the API, run the renewal job for its delivery day, check that the order appears on the packing list and in the daily report.

Each layer answers a different question. Unit tests ask whether a rule is right. Integration tests ask whether the pieces fit: does the SQL match the schema, do the routes return the right status codes, does the CLI parse what the scheduler sends? End-to-end tests ask whether a real user's journey works from start to finish. A team with only unit tests ships code where every piece works and the whole does not. A team with only end-to-end tests has a suite so slow and fragile that nobody runs it.

The team also agreed what not to test. They did not test Flask's routing or SQLite's ability to store an integer; those are other people's code, already well tested. They did not write tests that merely repeated the implementation line by line, which break on every refactor while catching nothing. And they did not chase a coverage number. Coverage tells you what code your tests ran, not whether they checked anything; a test that calls a function without asserting its result raises coverage and proves nothing.

Tests that catch bugs

A useful way to judge a test is to ask what bug it would catch. PYTH-230 introduced mutation testing, which makes small deliberate changes to the code, such as turning less-than into less-than-or-equal, and checks whether any test fails. A test suite that still passes after a mutation has a gap. The team ran a mutation tool over the domain layer once a week and treated each surviving mutant as a missing test.

The surviving mutants were instructive. One changed the skip cut-off from one day to zero days, and no test failed, because every skip test used dates weeks away from the delivery. Another removed the reset of the skip counter after a delivery, and again nothing failed. Both were real rules from the acceptance criteria that the tests named but did not actually check at their boundaries. Boundary values, the day before, the exact day, the day after, catch more bugs than any number of comfortable middle values.

def check(next_ship):
    start = date(2026, 11, 30)
    assert next_ship(start, 2, 1) == date(2026, 12, 14)

Keeping fakes honest

The in-memory fakes made the unit tests fast, but they carried a risk: what if the fake and the real SQLite repository disagreed? A unit test might pass against the fake while the real application failed. The team removed that risk with contract tests: one set of tests describing how any subscription repository must behave, run against both implementations. If the fake ordered due subscriptions differently from the SQL query, or returned the same object instead of a copy, a contract test failed for one of them.

In pytest this is a fixture with two parameters, so every contract test runs twice, once per implementation. The contract is the specification of the port from lesson 3, written as executable tests. It found two real differences in its first run: the fake returned paused subscriptions that SQLite correctly excluded, and the fake let a test change a subscription without calling save, which hid a missing save call in the pause service.

import pytest

from ridgeline_ops.adapters.sqlite_repo import SqliteSubscriptions
from ridgeline_ops.testing.fakes import InMemorySubscriptions
from ridgeline_ops.testing.db import fresh_database

@pytest.fixture(params=["memory", "sqlite"])
def repo(request):
    if request.param == "memory":
        return InMemorySubscriptions()
    return SqliteSubscriptions(fresh_database())

def test_due_on_excludes_paused(repo, make_sub):
    repo.add(make_sub(next_delivery="2026-12-01", status="paused"))
    assert repo.due_on("2026-12-01") == []

def test_changes_need_save(repo, make_sub):
    sub_id = repo.add(make_sub(next_delivery="2026-12-01"))
    sub = repo.get(sub_id)
    sub.status = "paused"           # changed but not saved
    assert repo.get(sub_id).status == "active"

Static analysis

Some problems are easier for a tool to find than a person. The pipeline ran Ruff, a fast linter that flags unused imports, undefined names, risky patterns such as mutable default arguments and bare except clauses, and style problems, and formats the code so that reviews never discuss spacing. It also ran mypy over the domain and application layers with strict settings, which caught several places where a function could return None and the caller did not handle it. The adapters were checked less strictly, because Flask's dynamic request objects produce noise.

Static analysis is cheap and runs on every push, so its findings arrive while the code is fresh in the author's mind. The team treated warnings as errors in the pipeline: a new warning failed the build. Allowing a few warnings to accumulate is how projects end up with hundreds that everyone ignores, including the one that matters.

Code review

Every change to Ridgeline Ops was read by another person before it was merged. Code review catches bugs, but that is not its only value: it spreads knowledge so that no part of the system is understood by only one person, it keeps the design consistent, and it teaches. Sam learned more about SQL from Jordan's review comments in four weeks than from a whole course, and Jordan learned about accessibility from Nia's.

The team's review guide, a page in the repository, gave reviewers an order to work in. First, does the change do what the story needs? Read the acceptance criteria and the tests before the code. Second, is it correct at the edges: empty lists, the boundary dates, short stock, invalid input? Third, does it fit the design: are the layers respected, is the logic in the right place, does it reuse what exists? Fourth, is it clear: names, function size, comments that explain why rather than what? Style comes last and is mostly the formatter's job.

How comments are written matters as much as what they say. The team used a simple convention borrowed from a practice often called conventional comments: each comment starts with a label that says how seriously to take it. Blocking means the change cannot merge until this is resolved. Suggestion means the reviewer would do it differently but the author decides. Question means the reviewer does not understand and wants an explanation, which often reveals a missing comment or test. Praise means something was done well, which is worth saying out loud. Comments are about the code, never about the person: this function does two things, not you always write long functions.

Reviews were expected within four working hours, and pull requests were kept small enough to review in twenty minutes. A reviewer who could not understand a change in that time said so, and the author split it. Large reviews are where bugs hide, because reviewers tire and start skimming.

def add_line(order, sku, bags, price_table, lines=[]):
    try:
        lines.append((sku, bags, float(price_table[sku])))
    except:
        pass
    order["lines"] = lines
    return sum(b * p for _, b, p in lines)

Tracing tests to requirements

Before the feature freeze, Jordan wanted evidence that every Must and Should story's acceptance criteria were covered by at least one automated test. The team had tagged each test with the criteria it checked, using a pytest marker such as criteria("SKIP-2"), so a short script could list the criteria with no tests. The answer was uncomfortable: four criteria had no test, including the rule that cancelled subscriptions never appear on the packing list. Two of the four turned out to be broken.

def untested(criteria, tests):
    checked = set()
    for name, ids in tests.items():
        checked.update(ids)
    return [c for c in criteria if c not in checked]

Testing with real people

Automated tests check what the team thought of. Real users find what nobody thought of. On the Tuesday of the fourth week, the team ran a two-hour testing session in the roastery with Theo, Kai, Mei and Anita, each given a short list of tasks and asked to think aloud while doing them: sign up a regular at the till, pack this morning's orders, check whether next week's roast plan covers subscriptions, reconcile yesterday's report. The developers watched and took notes without helping unless someone was completely stuck.

The session found eleven issues. None was a crash. Mei could not find the sign-up form because it was under a menu called Admin. Theo's tablet went to sleep between orders and logged him out, losing his place. Anita found that subscription orders appeared under the date they were created rather than the date they shipped, which put revenue on the wrong day. Kai wanted the roast plan in kilograms of green coffee, not bags, because that is what he loads into the roaster. Each issue went into the backlog, was triaged by severity and Rosa's priority, and the important ones were fixed in the final week.

Triage kept the list under control. A severity scale described how bad a problem was: critical for data loss or wrong money, major for a blocked task, minor for an annoyance, and cosmetic for appearance only. Severity and priority are different: the report date bug was critical and fixed first, while the Admin menu was minor but cheap and highly visible, so it was fixed the same afternoon.