Code coverage and test strategy

By the second week of December, Ridgeline's test suite had grown to 140 tests, and they all passed. Rosa asked a fair question: is that enough? Are there parts of the code that no test ever touches? Jordan had no idea. A count of tests says nothing about what they cover; 140 tests could all be checking the loyalty tiers while the order import goes untested.

Code coverage measures which parts of the code run while the tests run. It is a useful tool and an easily misused number. This lesson explains line and branch coverage and how to measure them with coverage.py, what coverage can and cannot tell you, how mutation testing checks the tests themselves, how to choose test cases systematically, and how to put these ideas together into a test strategy: deciding what to test, how much and at which level.

Measuring coverage

The standard tool is coverage.py, installed with pip. It runs your tests while watching which lines of your program execute, then reports, for each file, how many statements ran and which ones did not. With pytest, the pytest-cov plugin does the same through an option.

> coverage run -m pytest
================ 140 passed in 1.84s ================
> coverage report -m
Name                        Stmts   Miss  Cover   Missing
---------------------------------------------------------
src/ridgeline/giftcard.py      24      0   100%
src/ridgeline/inventory.py     61      3    95%   88-90
src/ridgeline/loyalty.py       18      0   100%
src/ridgeline/orders.py        74     31    58%   41-60, 77-89, 102
src/ridgeline/storage.py       39      6    85%   55-61
---------------------------------------------------------
TOTAL                         216     40    81%

> pytest --cov=ridgeline --cov-report=term-missing     # the same, through pytest-cov

The Missing column is the useful part. Lines 41 to 60 of orders.py never ran during the tests: that turned out to be the code that handles an order with no shipping address, exactly the kind of case that went wrong on that Saturday in lesson 7. Lines 88 to 90 of inventory.py were an error branch for a corrupted stock file. coverage html produces a web page with every file shown line by line, run lines in green and missed lines in red, which makes gaps easy to see.

Line coverage and branch coverage

Line coverage, also called statement coverage, counts the lines that ran. It has a blind spot: an if statement whose condition was always true in the tests counts as covered, even though the case where it is false was never tried. Branch coverage counts each possible direction of every decision separately, so it reports that missing case. coverage.py measures branches when run with the --branch option, and it is worth turning on.

import unittest

def report(*cases):
    """Run TestCase classes and print one line per test (a small test runner)."""
    loader = unittest.TestLoader()
    tests = [t for case in cases for t in loader.loadTestsFromTestCase(case)]
    result = unittest.TestResult()
    unittest.TestSuite(tests).run(result)
    problems = {}
    for kind, items in (("FAIL", result.failures), ("ERROR", result.errors)):
        for test, tb in items:
            problems.setdefault(test.id().split(" ")[0], []).append((kind, tb.strip().splitlines()[-1]))
    for t in tests:
        found = problems.get(t.id(), [])
        print(f"{t.id().rsplit('.', 1)[-1]:<34} {found[0][0] if found else 'ok'}")
        for kind, line in found:
            print("    " + line)
    print(f"{len(tests)} tests: {len(tests) - len(problems)} passed, {len(problems)} failed")

def fee(subtotal, express):
    cost = 0 if subtotal >= 5000 else 795
    if express:
        cost += 1000
    return cost

class TestFee(unittest.TestCase):
    def test_small_express_order(self):
        self.assertEqual(fee(3000, True), 1795)

report(TestFee)
print("every line ran, yet fee(3000, False) and fee(6000, ...) were never tried")

test_small_express_order           ok
1 tests: 1 passed, 0 failed
every line ran, yet fee(3000, False) and fee(6000, ...) were never tried

This single test executes every line of fee, so line coverage is 100. But it never tries a non-express order, the false branch of if express, and never tries a large order, the other side of the conditional expression. A bug in either case would go unnoticed. Branch coverage would report the gaps.

To see how a coverage tool works underneath, the listing below uses sys.settrace, the hook that debuggers and coverage.py are built on. Python calls the trace function for every line executed, and the function records the line numbers that belong to the function being measured.

import sys

def shipping(subtotal):
    if subtotal <= 0:
        raise ValueError("empty order")
    if subtotal < 2500:
        return 1295
    if subtotal < 5000:
        return 795
    return 0

def lines_run(func, *calls):
    """Relative line numbers of func executed while making the calls."""
    seen = set()
    start = func.__code__.co_firstlineno
    def tracer(frame, event, arg):
        if frame.f_code is func.__code__ and event == "line":
            seen.add(frame.f_lineno - start)
        return tracer
    sys.settrace(tracer)
    try:
        for args in calls:
            func(*args)
    finally:
        sys.settrace(None)
    return sorted(seen)

all_lines = list(range(1, 8))      # the seven lines of the body
ran = lines_run(shipping, (1000,), (3000,))
print("ran:   ", ran)
print("missed:", [n for n in all_lines if n not in ran])

ran:    [1, 3, 4, 5, 6]
missed: [2, 7]

Two test inputs, 1000 and 3000 cents, leave the empty-order error and the free-shipping return unexecuted: relative lines 2 and 7. Adding tests for 0 and 6000 would cover them. Real tools do the same thing much more efficiently and across a whole project.

What coverage cannot tell you

Coverage tells you what the tests did not check: a line that never ran was certainly not tested. It cannot tell you that covered code was tested well. A test that calls a function and asserts nothing at all still produces full coverage of that function. Coverage measures which code was executed, not which behavior was verified.

import unittest

def report(*cases):
    """Run TestCase classes and print one line per test (a small test runner)."""
    loader = unittest.TestLoader()
    tests = [t for case in cases for t in loader.loadTestsFromTestCase(case)]
    result = unittest.TestResult()
    unittest.TestSuite(tests).run(result)
    problems = {}
    for kind, items in (("FAIL", result.failures), ("ERROR", result.errors)):
        for test, tb in items:
            problems.setdefault(test.id().split(" ")[0], []).append((kind, tb.strip().splitlines()[-1]))
    for t in tests:
        found = problems.get(t.id(), [])
        print(f"{t.id().rsplit('.', 1)[-1]:<34} {found[0][0] if found else 'ok'}")
        for kind, line in found:
            print("    " + line)
    print(f"{len(tests)} tests: {len(tests) - len(problems)} passed, {len(problems)} failed")

def points(total_cents):
    if total_cents <= 0:
        return 0
    return total_cents * 100        # a bug: should be // 100

class TestPointsWeak(unittest.TestCase):
    def test_runs(self):
        points(1850)                # no assertion: covers lines, checks nothing
        points(-5)

report(TestPointsWeak)
print("coverage: 100 out of 100 lines; bugs found: none")

test_runs                          ok
1 tests: 1 passed, 0 failed
coverage: 100 out of 100 lines; bugs found: none

This is why coverage targets can backfire. If a team is told that coverage must reach 90, people write tests that execute code without checking it, and the number goes up while the protection does not. The economist Charles Goodhart's observation applies: when a measure becomes a target, it ceases to be a good measure. Use coverage to find untested areas and ask why, not as a score to maximize. Many teams settle on a reasonable floor, such as 80, enforced in continuous integration to stop coverage from slipping, while reviewing the quality of tests by reading them.

Mutation testing: testing the tests

If coverage cannot judge test quality, what can? Mutation testing makes many small deliberate changes to the code, called mutants: changing < to <=, + to -, a constant 100 to 101, deleting a line. For each mutant it runs the tests. If a test fails, the mutant is killed, which is good: the tests noticed the change. If all tests still pass, the mutant survived, which reveals behavior the tests do not actually check. Tools such as mutmut and cosmic-ray automate this for Python.

def tier(points):
    if points >= 300:
        return "Reserve"
    if points >= 100:
        return "Roast"
    return "Bean"

def weak_tests(f):
    return f(0) == "Bean" and f(500) == "Reserve" and f(150) == "Roast"

def strong_tests(f):
    return all(f(p) == t for p, t in [(99, "Bean"), (100, "Roast"), (299, "Roast"), (300, "Reserve")])

mutants = {
    ">= 300 became > 300": lambda p: "Reserve" if p > 300 else "Roast" if p >= 100 else "Bean",
    ">= 100 became > 100": lambda p: "Reserve" if p >= 300 else "Roast" if p > 100 else "Bean",
    "100 became 101":      lambda p: "Reserve" if p >= 300 else "Roast" if p >= 101 else "Bean",
}
for name, mutant in mutants.items():
    print(f"{name:<22} weak: {'survived' if weak_tests(mutant) else 'killed':<9} strong: {'survived' if strong_tests(mutant) else 'killed'}")

>= 300 became > 300    weak: survived  strong: killed
>= 100 became > 100    weak: survived  strong: killed
100 became 101         weak: survived  strong: killed

Every mutant survives the weak tests, even though those tests cover every line of tier, because none of them sits on a boundary. The boundary tests kill them all. Mutation testing is slow, since it runs the whole suite once per mutant, so teams usually run it occasionally or on the most important modules rather than on every commit.

Choosing test cases systematically

Good test cases are not random. Three techniques find most of what matters. Equivalence partitioning divides the possible inputs into groups that the code should treat the same way, such as orders under 2500 cents, from 2500 to 4999, and 5000 and up, and tests one representative from each. Boundary value analysis adds the values at and on either side of every boundary between groups, such as 2499, 2500, 4999 and 5000, because off-by-one mistakes cluster there. Error guessing adds the inputs experience says cause trouble: zero, negative numbers, empty lists, empty strings, None, very large values, and text with accented characters.

A fourth technique, property-based testing, lets a library generate hundreds of inputs automatically and checks properties that must hold for all of them, such as the shares of a split bill always adding up to the total. The Hypothesis library brings this to Python and is remarkably good at finding edge cases people miss.

# tests/test_split.py, using the Hypothesis library
from hypothesis import given, strategies as st
from ridgeline.money import split_bill

@given(total=st.integers(min_value=0, max_value=10_000_000),
       people=st.integers(min_value=1, max_value=50))
def test_shares_add_up_and_differ_by_at_most_one(total, people):
    shares = split_bill(total, people)
    assert sum(shares) == total
    assert max(shares) - min(shares) <= 1
    assert len(shares) == people

A test strategy

A test strategy answers four questions for a project: what to test, at which level, how much, and when the tests run. For Ridgeline, Jordan and Sam wrote theirs on one page.

What to test first is decided by risk: the cost of a failure multiplied by its likelihood. Money calculations, stock counts and anything customers see are high risk; a help message is low. Recently changed code and code with a history of bugs deserve extra attention. At which level follows the pyramid from lesson 1: rules as fast unit tests, the fit between modules as a smaller number of integration tests with real files and a fake shop, and a handful of end-to-end tests that drive the command-line tool through a whole day's script. How much is guided by coverage reports, branch coverage on, used to find gaps, plus boundary tests for every rule and a regression test for every bug. When is: unit tests on every save in the editor, the whole suite before every push, everything plus linting, type checking and coverage in continuous integration on every pull request.

The strategy also covers keeping the suite healthy. Slow tests get run less, so the suite should stay fast; Ridgeline's 140 tests finish in under two seconds, and any test that takes more than a fraction of a second is examined. Flaky tests are fixed or removed the day they are noticed, because a suite people do not trust is a suite people ignore. And tests are refactored like any other code when they become hard to read.

CASES = [
    (3000, False, 795),
    (6000, False, 0),
]

def test_shipping(f):
    return f(1000) == 1295 and f(3000) == 795 and f(9000) == 0

import sys

def lines_run(func, *calls):
    for args in calls:
        func(*args)
    return []