Testing I: fundamentals and unit testing

Testing is the most widely used way to gain confidence that software works. A test runs the software under controlled conditions and compares what happens with what should happen. Testing cannot prove that a program is correct; as Edsger Dijkstra famously put it, program testing can be used to show the presence of bugs, but never to show their absence. What well-designed tests can do is find defects early and cheaply, document expected behavior precisely, make refactoring safe and catch regressions when later changes break something that used to work. This lesson covers the vocabulary and levels of testing, unit testing with pytest, how to choose test cases systematically with equivalence partitioning and boundary value analysis, test doubles such as mocks and fakes, and test-driven development. Lesson 10 continues with integration, system and advanced techniques. CMPUT-174 lesson 11 introduced assert and unittest, and CMPUT-201 lesson 11 tested C programs; here testing becomes a team discipline.

The vocabulary of testing

Precise words help. In the IEEE's standard terminology, a person makes an error (a mistake, such as misreading a requirement), which introduces a fault, also called a defect or bug, into the code; when the faulty code executes under the right conditions, it may cause a failure, an observable difference between actual and expected behavior. Not every fault causes a failure: a fault in code that never runs, or that runs only with rare inputs, may lie dormant for years. Testing looks for failures; debugging then finds and fixes the faults behind them.

Every test needs a test oracle, a way to decide whether an outcome is correct. Often the oracle is simply the expected value written in the test, but sometimes it is harder: what is the correct output of a route planner or a machine learning model? Techniques such as property-based testing and metamorphic testing (lesson 10) help when exact expected values are unknown.

Mike Cohn's test pyramid summarizes how a healthy test suite is balanced: many fast unit tests at the base, fewer integration tests in the middle, and a small number of slow end-to-end tests at the top. End-to-end tests give the most realistic confidence but are slow, brittle and hard to diagnose when they fail; unit tests are fast and precise but cannot catch problems in how parts fit together. Lesson 10 returns to the pyramid and its alternatives.

Unit testing with pytest

Python's standard library includes unittest, modeled on Java's JUnit, but most Python teams use pytest, which runs plain functions whose names start with test_ and uses ordinary assert statements, rewriting them so that failures show the values involved. A good unit test follows the arrange–act–assert pattern: set up the inputs, call the code under test, and check the result. It tests one behavior, has a name that describes that behavior, runs fast, and does not depend on other tests or on the order in which tests run.

Consider a rule from a volunteer organization that also sells seats in paid training workshops: canceling a seat less than 24 hours before costs $15, between 24 and 72 hours costs $5, and minors never pay. Here is the function and a first set of tests.

def late_cancel_fee(hours_before: float, is_minor: bool = False) -> int:
    """Fee in dollars for cancelling a paid workshop seat."""
    if hours_before < 0:
        raise ValueError("cannot cancel after the start")
    if is_minor:
        return 0
    if hours_before < 24:
        return 15
    if hours_before < 72:
        return 5
    return 0

pytest -v --no-header test_fee.py

$ pytest -v --no-header test_fee.py
============================= test session starts ==============================
collecting ... collected 5 items
test_fee.py::test_no_fee_well_in_advance PASSED                          [ 20%]
test_fee.py::test_small_fee_two_days_before PASSED                       [ 40%]
test_fee.py::test_large_fee_on_the_day PASSED                            [ 60%]
test_fee.py::test_minors_never_pay PASSED                                [ 80%]
test_fee.py::test_cancelling_after_start_is_an_error PASSED              [100%]
============================== 5 passed in 0.01s ===============================

Each test name reads as a statement of the rule, so the test file doubles as documentation. The last test checks an error path with pytest.raises, which passes only if the expected exception occurs. Error paths deserve tests as much as the happy path, because that is where defects hide.

Choosing test cases: equivalence partitioning

Exhaustive testing is impossible for nearly any real function: even late_cancel_fee accepts infinitely many floating-point values. Test design techniques choose a small set of cases that is likely to reveal faults. Equivalence partitioning divides the input domain into classes, or partitions, whose members the program should treat the same way, then picks one representative from each. For late_cancel_fee the partitions of hours_before are: negative (error), 0 up to but not including 24 (fee 15), 24 up to but not including 72 (fee 5), and 72 or more (no fee); is_minor adds two more partitions. Invalid partitions, such as negative hours, are as important as valid ones.

Boundary value analysis

Faults cluster at the edges of partitions, because programmers write < when they meant <=, or count from 1 when they should count from 0. Boundary value analysis tests values at and immediately around each boundary: for the 24-hour boundary, test 23.99, 24 and 24.01 (or 23, 24 and 25 for whole numbers). pytest's parametrize decorator makes this compact: one test function, many cases, each reported separately.

pytest -q test_boundaries.py

$ pytest -q test_boundaries.py
.....F...                                                                [100%]
=================================== FAILURES ===================================
_________________________ test_fee_at_boundaries[72-0] _________________________
hours = 72, expected = 0
    @pytest.mark.parametrize("hours, expected", [
        (0, 15), (23.99, 15),           # lowest valid partition and its upper edge
        (24, 5), (24.01, 5),            # the 24-hour boundary
        (71.99, 5), (72, 0), (72.01, 0) # the 72-hour boundary
    ])
    def test_fee_at_boundaries(hours, expected):
>       assert late_cancel_fee(hours) == expected
E       assert 5 == 0
E        +  where 5 = late_cancel_fee(72)
test_boundaries.py:10: AssertionError
=========================== short test summary info ============================
FAILED test_boundaries.py::test_fee_at_boundaries[72-0] - assert 5 == 0
1 failed, 8 passed in 0.01s

This run uses a deliberately faulty version of the function, with <= 72 where the rule says less than 72 hours. All the representative-value tests from the previous section would still pass, because 48 and 100 are far from the boundary. Only the boundary case at exactly 72 hours exposes the fault, and pytest's failure message shows the actual and expected values. That is the whole argument for boundary value analysis in one example.

When several inputs interact, the number of combinations grows quickly. Decision tables list the combinations of conditions and the expected action for each; for example, is_minor combined with each time band. Pairwise (all-pairs) testing chooses a small set of cases that covers every pair of parameter values at least once, on the evidence that most interaction faults involve only two parameters. Tools such as Microsoft's PICT generate pairwise suites automatically.

Test doubles: stubs, fakes and mocks

Unit tests should be fast and isolated, but code often depends on slow or unpredictable things: databases, networks, email servers, the clock, random numbers. A test double stands in for such a dependency. Gerard Meszaros named several kinds. A dummy is passed but never used. A stub returns canned answers, such as a fixed current time. A fake is a working but simplified implementation, such as an in-memory repository instead of a database. A spy records how it was called. A mock is configured with expectations about how it should be called and lets the test verify them. Python's unittest.mock provides Mock and MagicMock objects that can act as stubs, spies or mocks, and patch, which temporarily replaces a name during a test.

from datetime import datetime, timedelta
from unittest.mock import Mock

class ReminderService:
    def __init__(self, shifts, notifier, clock):
        self.shifts, self.notifier, self.clock = shifts, notifier, clock

    def send_due(self):
        """Remind volunteers whose shift starts within the next 24 hours."""
        now = self.clock()
        sent = 0
        for s in self.shifts.upcoming():
            if now <= s["start"] <= now + timedelta(hours=24):
                self.notifier.send(s["email"], f"Reminder: shift at {s['start']:%H:%M}")
                sent += 1
        return sent

# Stub for the clock: a fixed "now" makes the test deterministic
fixed_now = datetime(2026, 11, 6, 9, 0)
# Fake repository: a tiny in-memory implementation
class FakeShifts:
    def upcoming(self):
        return [{"email": "ana@example.org", "start": fixed_now + timedelta(hours=3)},
                {"email": "ben@example.org", "start": fixed_now + timedelta(hours=30)}]
# Mock notifier: records calls so the test can verify them
notifier = Mock()

service = ReminderService(FakeShifts(), notifier, clock=lambda: fixed_now)
assert service.send_due() == 1
notifier.send.assert_called_once_with("ana@example.org", "Reminder: shift at 12:00")
print("reminders sent:", notifier.send.call_count)
print("calls recorded:", notifier.send.call_args_list)

reminders sent: 1
calls recorded: [call('ana@example.org', 'Reminder: shift at 12:00')]

This test runs in microseconds, never sends real email and never depends on the time of day. It works because ReminderService receives its dependencies through its constructor rather than creating them, the dependency injection from CMPUT-274 lesson 5. Code that reads the clock or opens a database connection deep inside a function is hard to test, which is one reason testability is a design quality, not only a testing concern.

Mocks are powerful and easily overused. A test that mocks everything and checks that particular internal methods were called in a particular order is coupled to the implementation: a harmless refactoring breaks it even though behavior is unchanged. Two schools describe the tension. The classicist (or Detroit) school uses real objects wherever practical and doubles only for awkward dependencies, testing state and results. The mockist (or London) school, associated with the book Growing Object-Oriented Software, Guided by Tests, uses mocks to drive the design of object interactions. A good rule of thumb for both: verify behavior that matters to the caller, such as 'one reminder was sent to Ana', and prefer fakes to mocks for things like repositories.

Test-driven development

Test-driven development (TDD), popularized by Kent Beck as part of Extreme Programming, reverses the usual order: write a failing test for the next small piece of behavior, write just enough code to make it pass, then refactor while the tests stay green. The cycle, often summarized as red–green–refactor, takes minutes and repeats continuously. TDD produces a thorough test suite as a by-product, keeps the design testable because every unit is used by a test before it exists, and gives frequent feedback. Its critics point out that it can lead to over-specified tests and that it suits well-understood problems better than exploratory work. Research results are mixed: studies at IBM and Microsoft found substantially fewer defects with TDD at the cost of somewhat more initial development time, while other controlled experiments found smaller effects. Many developers use TDD selectively, for logic-heavy code and bug fixes, where writing the failing test first proves the bug is understood.

Here is one red–green cycle for a new rule: a volunteer may not sign up for two overlapping shifts. First the test, which fails because the function does not exist yet.

pytest -q test_overlap.py 2>&1 | grep -E "^E |error"

$ pytest -q test_overlap.py 2>&1 | grep -E "^E |error"
E   ImportError: cannot import name 'overlaps' from 'overlap' (~/project/overlap.py)
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
1 error in 0.01s

cat overlap.py
pytest -q test_overlap.py

$ cat overlap.py
def overlaps(a, b):
    """True if the half-open intervals [start, end) a and b share any time."""
    return a[0] < b[1] and b[0] < a[1]
$ pytest -q test_overlap.py
...                                                                      [100%]
3 passed in 0.01s

The implementation uses half-open intervals, a choice the first test forced: a shift ending at 12:00 and one starting at 12:00 do not overlap. Writing the test first made that decision explicit before any code existed. The refactor step here might rename the parameters or introduce a Shift type; with the tests green, such changes are safe.

What makes tests good

Good unit tests share a set of properties sometimes summarized as FIRST: Fast, so they can run on every change; Independent, so they can run in any order; Repeatable, giving the same result every time in any environment; Self-validating, passing or failing without a person reading output; and Timely, written with or before the code. Tests are code and deserve the same care: clear names, no duplication that hides intent, and helpers or fixtures for shared setup. A test that fails intermittently, a flaky test, is worse than no test, because people learn to ignore failures; lesson 10 discusses causes and cures.