Testing II: integration, system and advanced techniques

Unit tests check parts in isolation, but software fails just as often where the parts meet: a query that works against a fake repository fails against the real database because of a missing index or a time zone; a service and its client disagree about a field name; the whole system works for one user and falls over for five hundred. This lesson moves up the test pyramid to integration, system and acceptance testing, then covers techniques for judging and strengthening a test suite: code coverage and its limits, property-based testing, mutation testing, and the causes and cures of flaky tests. It ends with non-functional testing for performance, security and accessibility. Lesson 9 covered the fundamentals; lesson 11 puts all of these tests into an automated pipeline.

Integration testing

Integration tests check that units work together, or that a unit works with real infrastructure such as a database, a file system, a message broker or another service. They catch faults that unit tests with doubles cannot: SQL that is valid in SQLite but not in PostgreSQL, a migration that forgets a column, an ORM mapping that loads the wrong relationship, serialization that loses time zones. Integration tests are slower than unit tests, so a suite has fewer of them, but they are essential at every boundary where a fake stands in for something real in the unit tests.

Two practical rules keep integration tests trustworthy. First, test against the same kind of system used in production; tools such as Testcontainers start a real PostgreSQL or Kafka instance in a container for the duration of the tests. Second, give each test a clean, known state, usually by creating the schema in a fixture and running each test inside a transaction that is rolled back afterwards. pytest fixtures make this easy: a fixture function sets up a resource, yields it to the test and cleans up after it.

pytest -q test_repo.py

$ pytest -q test_repo.py
....                                                                     [100%]
4 passed in 0.01s

The last test documents a weakness the integration test exposed: signing up for a shift that does not exist fails with an unhelpful TypeError from deep inside the repository, because the query returns no row. A unit test with a fake repository would never have found this. The right fix is to raise a clear domain exception, such as ShiftNotFound, and change the test to expect it; recording the current behavior first, as here, is the characterization approach from lesson 7.

When services talk to each other over a network, contract tests check that a consumer's expectations and a provider's behavior agree without running both together. In consumer-driven contract testing, popularized by the Pact tool, the consumer's tests record the requests it makes and the responses it expects; the provider's build then replays those interactions against the real provider. Contract tests catch breaking API changes before deployment, at much lower cost than full end-to-end environments.

System, end-to-end and acceptance testing

System tests exercise the whole deployed application through its external interfaces, as a user or client would: through the browser, a public API or a command-line interface. Browser automation tools such as Playwright and Selenium drive a real browser, clicking buttons and checking what appears. End-to-end tests give the most realistic confidence and are the only way to check that every layer works together, but they are slow, need a full environment, and break easily when the user interface changes. Keep them few, focused on the most valuable user journeys such as signing up for a shift and producing the monthly report, and make them robust by selecting elements by stable labels or test identifiers rather than by layout.

Acceptance tests check that the system meets its requirements from the user's point of view; the Given/When/Then scenarios from lesson 3 are acceptance criteria, and BDD tools such as Cucumber, behave and pytest-bdd turn them into executable tests. User acceptance testing (UAT) puts the system in front of real users before release, which validates as well as verifies. Exploratory testing, in which a skilled tester explores the software without a script, guided by charters such as 'explore sign-up on a phone with poor connectivity', finds problems that scripted tests miss because nobody thought to script them.

Regression testing reruns existing tests after a change to check that nothing that used to work has broken. Automated suites make regression testing cheap enough to run on every commit, which is the foundation of continuous integration in lesson 11. Whenever a bug is found, the fix should come with a test that fails without it, so that the bug can never quietly return.

Code coverage

Code coverage measures how much of the code the tests execute. Statement (line) coverage is the percentage of statements executed at least once. Branch coverage is the percentage of branch outcomes taken: for each if, both the true and false outcomes. Branch coverage is stronger, because a test suite can execute every statement of an if with no else without ever testing the case where the condition is false. In Python the standard tool is coverage.py, usually through the pytest-cov plugin; the small tracer below shows how such tools work by recording which lines run and which jumps between lines occur.

import sys, inspect, ast

def late_cancel_fee(hours_before, is_minor=False):
    if hours_before < 0:
        raise ValueError("cannot cancel after the start")
    fee = 0
    if not is_minor:
        if hours_before < 24:
            fee = 15
        elif hours_before < 72:
            fee = 5
    return fee

src, first = inspect.getsourcelines(late_cancel_fee)
code = late_cancel_fee.__code__
lines_run, arcs = set(), set()

def tracer(frame, event, arg):
    if frame.f_code is code:
        if event == "line":
            lines_run.add(frame.f_lineno)
            if tracer.last is not None:
                arcs.add((tracer.last, frame.f_lineno))
            tracer.last = frame.f_lineno
        return tracer
    return tracer
tracer.last = None

def run(*args, **kw):
    tracer.last = None
    sys.settrace(tracer)
    try:
        late_cancel_fee(*args, **kw)
    except ValueError:
        pass
    finally:
        sys.settrace(None)

# Statement lines (excluding the def line) and the two outcomes of every if
tree = ast.parse("".join(src))
offset = first - 1
stmts = {n.lineno + offset for n in ast.walk(tree) if isinstance(n, ast.stmt)} - {first}
branches = []
for node in ast.walk(tree):
    if isinstance(node, ast.If):
        line = node.lineno + offset
        true_line = node.body[0].lineno + offset
        false_line = node.orelse[0].lineno + offset if node.orelse else None
        branches.append((line, true_line, false_line))

def report(label):
    taken = 0
    for line, t, f in branches:
        taken += (line, t) in arcs
        taken += any(a == line and b != t for a, b in arcs) if f is None else (line, f) in arcs
    print(f"{label:28s} statements {len(lines_run & stmts)}/{len(stmts)}  "
          f"branches {taken}/{2 * len(branches)}")

run(48)
report("one test: hours=48")
run(3); run(100)
report("+ hours=3 and hours=100")
run(3, is_minor=True); run(-1)
report("+ minor and negative")

one test: hours=48           statements 7/9  branches 4/8
+ hours=3 and hours=100      statements 8/9  branches 6/8
+ minor and negative         statements 9/9  branches 8/8

A single test executes most statements but leaves several branches untaken. Adding the cases from lesson 9's partitions brings both measures to 100 percent. Coverage is a useful tool for finding untested code: a function with 0 percent coverage is definitely untested, and a branch never taken is a missing test case. But high coverage does not mean good tests. A test can execute every line and assert nothing. Coverage targets also invite gaming, Goodhart's law again. Many teams therefore watch coverage trends and require new code to be covered, rather than chasing a single number; Google's published guidance treats around 60 percent as acceptable, 75 percent as commendable and 90 percent as exemplary, while warning against treating any number as the goal.

Property-based testing

Example-based tests check specific inputs chosen by a person. Property-based testing, pioneered by the QuickCheck library for Haskell in 2000, instead states a property that should hold for all inputs, generates hundreds of random inputs, and checks the property for each. When it finds a failing input, it shrinks it to a minimal failing case that is easy to understand. In Python the standard library is Hypothesis. Good properties include round trips (decode(encode(x)) == x), invariants (sorting does not change the multiset of elements), equivalence to a simple but slow reference implementation, and algebraic laws such as symmetry. The example below hand-rolls the idea to show how it works, testing a function that merges overlapping shifts in a volunteer's calendar.

import random

def merge(intervals):
    """Merge overlapping half-open intervals [start, end). (Contains a bug.)"""
    result = []
    for s, e in sorted(intervals):
        if result and s < result[-1][1]:          # bug: touching intervals are not merged
            result[-1] = (result[-1][0], max(result[-1][1], e))
        else:
            result.append((s, e))
    return result

def covered(intervals):                           # reference: the set of covered hours
    return {h for s, e in intervals for h in range(s, e)}

def prop(intervals):
    m = merge(intervals)
    no_gaps_lost = covered(m) == covered(intervals)
    disjoint = all(a[1] < b[0] for a, b in zip(m, m[1:]))   # merged blocks must not touch
    return no_gaps_lost and disjoint

def shrink(case):                                 # try removing elements, keep if still failing
    changed = True
    while changed:
        changed = False
        for i in range(len(case)):
            smaller = case[:i] + case[i + 1:]
            if smaller and not prop(smaller):
                case, changed = smaller, True
                break
    return case

rng = random.Random(301)
for trial in range(1, 501):
    case = []
    for _ in range(rng.randint(1, 6)):
        s = rng.randint(0, 20)
        case.append((s, s + rng.randint(1, 5)))
    if not prop(case):
        print(f"falsified on trial {trial}: {case}")
        print("shrunk to:", shrink(case), "->", merge(shrink(case)))
        break
else:
    print("property held for 500 random cases")

falsified on trial 5: [(17, 20), (15, 17), (19, 23)]
shrunk to: [(17, 20), (15, 17)] -> [(15, 17), (17, 20)]

The property says the merged calendar must cover exactly the same hours and contain no touching blocks. Random generation finds a failure, and shrinking reduces it to two intervals that meet end to start, which the code fails to merge because it uses < where <= was needed. A human writing examples might never have thought of touching intervals; the generator tries them because it tries everything. Hypothesis does this far more cleverly, with strategies for generating structured data, a database of past failures and very effective shrinking.

Mutation testing

How good is a test suite? Coverage says what it executes, not whether it would notice a fault. Mutation testing answers the second question directly. A tool makes many small changes to the code, called mutants, such as replacing < with <=, + with -, or True with False, and runs the tests against each. If some test fails, the mutant is killed; if all tests still pass, the mutant survived, which reveals either a missing test or, occasionally, an equivalent mutant that behaves identically to the original. The mutation score is the percentage of mutants killed. Tools include mutmut and cosmic-ray for Python, PIT for Java and Stryker for JavaScript. The example below applies the idea to lesson 9's fee function with two test suites.

import ast

SOURCE = """
def late_cancel_fee(hours_before, is_minor=False):
    if hours_before < 0:
        raise ValueError
    if is_minor:
        return 0
    if hours_before < 24:
        return 15
    if hours_before < 72:
        return 5
    return 0
"""
SWAP = {ast.Lt: ast.LtE, ast.LtE: ast.Lt, ast.Gt: ast.GtE, ast.GtE: ast.Gt}

def mutants(source):
    """Yield (description, mutated source) for each comparison and each return constant."""
    tree = ast.parse(source)
    targets = [n for n in ast.walk(tree) if isinstance(n, ast.Compare)
               or (isinstance(n, ast.Return) and isinstance(n.value, ast.Constant))]
    for i in range(len(targets)):
        t = ast.parse(source)
        node = [n for n in ast.walk(t) if isinstance(n, ast.Compare)
                or (isinstance(n, ast.Return) and isinstance(n.value, ast.Constant))][i]
        if isinstance(node, ast.Compare):
            node.ops[0] = SWAP[type(node.ops[0])]()
            desc = f"line {node.lineno}: flip comparison"
        else:
            node.value = ast.Constant(node.value.value + 1)
            desc = f"line {node.lineno}: return {node.value.value}"
        yield desc, ast.unparse(t)

def passes(source, tests):
    env = {}
    exec(source, env)
    f = env["late_cancel_fee"]
    for args, expected in tests:
        try:
            if f(*args) != expected:
                return False
        except ValueError:
            if expected != "error":
                return False
        else:
            if expected == "error":
                return False
    return True

typical = [((100,), 0), ((48,), 5), ((3,), 15), ((3, True), 0), ((-1,), "error")]
boundary = typical + [((0,), 15), ((24,), 5), ((72,), 0)]
for name, tests in (("typical values", typical), ("typical + boundaries", boundary)):
    results = [(d, not passes(src, tests)) for d, src in mutants(SOURCE)]
    killed = sum(k for _, k in results)
    print(f"{name}: killed {killed}/{len(results)} mutants")
    for d, k in results:
        if not k:
            print("   survived:", d)

typical values: killed 4/7 mutants
   survived: line 3: flip comparison
   survived: line 7: flip comparison
   survived: line 9: flip comparison
typical + boundaries: killed 7/7 mutants

With only typical values, the three mutants that flip a comparison at a boundary survive: changing < 0 to <= 0, < 24 to <= 24 or < 72 to <= 72 breaks nothing the tests check. Adding the three boundary cases kills every mutant. Mutation testing is computationally expensive, since each mutant runs the suite, so teams run it on critical modules or on changed code rather than on everything; Google reported running it during code review on changed lines only, surfacing a few surviving mutants as review comments.

Flaky tests

A flaky test passes and fails on the same code without any change. Flakiness destroys trust in a test suite, because developers learn to rerun failures until they pass, and then miss real failures. Studies at Google found that a significant fraction of its tests had shown flaky behavior at some point. Common causes are dependence on timing (sleeping for a fixed time and hoping an asynchronous operation finished), shared state between tests, test order dependence, real network calls, unseeded randomness, time zones and the current date, concurrency races, and resource leaks. Cures follow from the causes: wait for conditions rather than for fixed times, isolate state with fixtures, inject clocks and random seeds, replace network calls with fakes or contract tests, and run tests in random order (the pytest-randomly plugin does this) to expose hidden dependencies. A flaky test should be quarantined and fixed promptly, not ignored.

Non-functional testing

Performance testing checks the quantified performance requirements from lesson 3. Load testing applies the expected load, stress testing pushes beyond it to find the breaking point and how the system fails, soak testing runs a normal load for hours to find memory leaks and slow degradation, and spike testing applies sudden bursts like the first-of-the-month rush in lesson 6's example. Tools such as Locust (Python), k6 and JMeter generate load and report percentile response times. Results are only meaningful in an environment and with data similar to production.

Security testing includes static application security testing (SAST), which scans source code for vulnerable patterns; dependency scanning for libraries with known vulnerabilities; dynamic testing (DAST) with tools such as OWASP ZAP that probe a running application; fuzzing, which feeds random or malformed inputs to find crashes; and penetration testing by specialists. The OWASP Top 10 lists the most critical web application risks, such as broken access control, injection and security misconfiguration. CMPUT-331 and CISSP-117 cover security in depth; here the point is that security tests belong in the same automated pipeline as functional tests.

Accessibility testing combines automated checkers, such as axe-core, Lighthouse and the WAVE tool, which can find perhaps a third to a half of WCAG issues such as missing labels, low contrast and missing alternative text, with manual testing: navigating by keyboard alone, using screen readers such as NVDA, JAWS or VoiceOver, zooming to 200 percent, and, best of all, testing with disabled users. Because accessibility is a legal requirement for many organizations (lesson 3), it belongs in the definition of done.