Testbenches and simulation: self-checking, assertions

In software, a bug found in the field can be patched in an afternoon. An FPGA bitstream can also be updated, but a bug in the NS-5 front end could corrupt radar frames for weeks before anyone notices, and diagnosing it on a pole beside a highway is slow and expensive. Industry surveys of FPGA and chip projects consistently find that verification takes more engineering effort than design, often well over half. This lesson teaches the core of that work: testbenches that check their own results, golden reference models, assertions that watch every cycle, and coverage that tells you when you have tested enough.

Jonah has written the offset block and the capture controller. Ines Carvalho's rule for her team is that no block is merged until it has a self-checking testbench that a continuous-integration server can run without anyone looking at waveforms. The pilot in Kingston will run unattended for months, and she wants every bug that can be found in simulation found there.

What a testbench is

A testbench is a module with no ports that instantiates the design under test (DUT), generates its clocks and resets, drives its inputs, observes its outputs and decides whether they are correct. Because it is never synthesized, it can use the whole language: delays, files, dynamic arrays, queues, classes, random number generation and text output. The figure shows the parts of a structured testbench. Small testbenches combine several of these parts in one block; large verification environments build each as a separate component.

The stimulus generator decides what to send: directed tests for specific cases, random values within constraints, or vectors read from a file. The driver turns each transaction into pin-level activity with correct timing. The monitor watches the outputs and turns pin activity back into transactions. The reference model, also called the golden model, predicts the correct output for each input, and the scoreboard compares predicted and observed transactions. Assertions check protocol and timing rules continuously, and coverage records which interesting cases have actually been exercised.

Self-checking

A testbench that prints values for a person to inspect, or dumps waveforms for someone to look at, is not a test. It works once, on the day it was written, and then decays as the design changes and nobody looks closely enough. A self-checking testbench compares every output against an expected value and ends with a single pass or fail result and an exit status that a script can read. It can then run on every commit, with every random seed, overnight, which is how verification finds the rare bugs.

The simplest reference model for a datapath block is a bit-accurate model in another language, usually Python or C, that computes exactly what the hardware should compute, including widths, rounding and saturation. The model generates test vectors, input and expected output, which the testbench reads with $readmemh. This splits the work cleanly: Python makes it easy to write the algorithm correctly and to generate interesting inputs, while the testbench deals only with timing.

Choosing stimulus

Exhaustive testing is possible for small combinational blocks. The offset block has two 12-bit inputs, so there are 2^24 = 16,777,216 combinations; a simulator can apply them all in under a minute. Most blocks are far beyond that. The capture controller's behaviour depends on its history; the filter has 32 samples of 12-bit state per channel, 2^384 states. For these, verification combines three kinds of stimulus. Directed tests target the corners a designer can name: maximum and minimum values, zero, overflow by one, back-to-back events, reset in every state. Constrained random tests generate legal but unpredictable inputs, biased toward interesting regions. Coverage measurement then shows which cases have been hit and steers the next tests.

The model below generates vectors for the offset block. It mixes directed corner cases with random values, biases half of the random draws toward the saturation boundaries where bugs live, computes the expected result bit-exactly, and tracks coverage bins: whether the output clipped high, clipped low, landed exactly on each limit without clipping, or passed through unchanged.

import random
random.seed(450)
W = 12
LO, HI = -(1 << (W - 1)), (1 << (W - 1)) - 1

def model(x, off):
    d = x - off
    return (HI, 1) if d > HI else (LO, 1) if d < LO else (d, 0)

directed = [(0, 0), (HI, 0), (LO, 0), (HI, -1), (LO, 1), (HI, LO), (LO, HI),
            (-1, -1), (1, 0), (HI, HI), (LO, LO), (0, LO), (0, HI)]
vectors = list(directed)
while len(vectors) < 5000:
    off = random.randint(-200, 200)           # realistic calibration offsets
    if random.random() < 0.5:                 # bias toward the clamp limits
        x = random.choice([HI, LO]) + off + random.randint(-3, 3)
        x = max(LO, min(HI, x))
    else:
        x = random.randint(LO, HI)
    vectors.append((x, off))

bins = {"clip high": 0, "clip low": 0, "exactly HI": 0, "exactly LO": 0, "pass": 0}
lines = []
for x, off in vectors:
    y, c = model(x, off)
    key = ("clip high" if y == HI else "clip low") if c else \
          "exactly HI" if y == HI else "exactly LO" if y == LO else "pass"
    bins[key] += 1
    word = ((x & 0xFFF) << 25) | ((off & 0xFFF) << 13) | ((y & 0xFFF) << 1) | c
    lines.append(format(word, "010x"))       # one 37-bit word per line for $readmemh
    if len(lines) <= 6:
        print(f"  x={x:6d} off={off:5d} -> y={y:6d} clipped={c}   hex line {lines[-1]}")
print(f"{len(lines)} vectors written")
for k, v in bins.items():
    print(f"  coverage bin {k:11s} {v:5d}  {'hit' if v else 'MISSING'}")

  x=     0 off=    0 -> y=     0 clipped=0   hex line 0000000000
  x=  2047 off=    0 -> y=  2047 clipped=0   hex line 0ffe000ffe
  x= -2048 off=    0 -> y= -2048 clipped=0   hex line 1000001000
  x=  2047 off=   -1 -> y=  2047 clipped=1   hex line 0fffffefff
  x= -2048 off=    1 -> y= -2048 clipped=1   hex line 1000003001
  x=  2047 off=-2048 -> y=  2047 clipped=1   hex line 0fff000fff
5000 vectors written
  coverage bin clip high     271  hit
  coverage bin clip low      288  hit
  coverage bin exactly HI     95  hit
  coverage bin exactly LO     85  hit
  coverage bin pass         4261  hit

Without the bias, uniformly random inputs with small offsets would almost never land exactly on a limit, and the exactly-HI and exactly-LO bins, the cases most likely to reveal an off-by-one error in the comparison, might stay empty for thousands of vectors. Biasing the stimulus toward boundaries and measuring which bins were hit is the essence of coverage-driven verification.

A self-checking testbench

The testbench below reads those vectors, drives them into the offset block and compares the outputs. It is written for a clocked version of the block with one cycle of latency, which is how it is used in the front end. Inputs are driven with non-blocking assignments at a rising edge, and outputs are checked 1 ns after the following edge, once the registered result has settled, so the testbench never races the design.

`timescale 1ns/1ps
module tb_offset_sat;
    localparam int W = 12, NV = 5000;
    logic clk = 0, rst_n = 0;
    logic signed [W-1:0] x, offset, y;
    logic clipped;
    logic [3*W:0] vec [NV];             // x, offset, y, clipped packed per line
    int errors = 0;

    always #5 clk = ~clk;               // 100 MHz

    offset_sat_reg #(.W(W)) dut (.*);   // connect ports by name

    initial begin
        $readmemh("offset_vectors.hex", vec);
        repeat (3) @(posedge clk);
        rst_n <= 1;
        for (int i = 0; i < NV; i++) begin
            @(posedge clk);
            x      <= vec[i][3*W:2*W+1];
            offset <= vec[i][2*W:W+1];
            @(posedge clk); #1;          // one cycle later the registered output is ready
            if (y !== vec[i][W:1] || clipped !== vec[i][0]) begin
                errors++;
                if (errors <= 10)
                    $display("mismatch %0d: x=%0d off=%0d got y=%0d c=%b",
                             i, x, offset, y, clipped);
            end
        end
        if (errors == 0) $display("PASS: %0d vectors", NV);
        else             $display("FAIL: %0d mismatches", errors);
        $finish;
    end
endmodule

Three details in the listing are worth copying. The comparison uses !==, the four-state inequality operator, so an output of X or Z counts as a mismatch instead of silently comparing as unknown; with != an undriven output would pass. The number of printed messages is limited so that a systematic failure does not bury the log. And the final message has a fixed form that a regression script can search for, together with an exit status from $finish or $fatal that the continuous-integration server checks.

Timing in testbenches and races

The commonest testbench bug is a race between the testbench driving an input and the design sampling it on the same clock edge. If the testbench assigns an input with a blocking assignment at the rising edge, it is undefined whether the design's always_ff block sees the old or the new value. The cures are to drive inputs with non-blocking assignments at the edge, to drive them at the opposite edge or after a small delay, or to use a SystemVerilog clocking block, which specifies input sampling and output driving skews relative to a clock and removes the problem by construction. Monitors should sample outputs in the same disciplined way.

Assertions

A scoreboard checks results; an assertion checks behaviour. SystemVerilog assertions (SVA) are statements about the design that the simulator evaluates continuously, and they come in two forms. An immediate assertion is a procedural check, like assert in C, evaluated when the statement executes. A concurrent assertion describes a temporal property evaluated on every clock edge, in a compact language of sequences and implications. The implication a |-> b means whenever a is true, b must be true in the same cycle; a |=> b means b must be true in the next cycle; ##n inserts a delay of n cycles. Functions such as $rose, $fell, $stable and $past refer to changes and earlier values.

module capture_checks (input logic clk, rst_n, start, sync, s_valid,
                       wr_en, last, busy, done_irq, input logic [9:0] cnt);
    default clocking cb @(posedge clk); endclocking
    default disable iff (!rst_n);

    // a write happens only while capturing and only with a valid sample
    a_wr_valid:  assert property (wr_en |-> s_valid && busy);

    // the sample counter never wraps inside a frame
    a_cnt_range: assert property (wr_en |-> cnt <= 10'd1023);

    // last is followed by done within 100 cycles (80-cycle flush plus margin)
    a_done_time: assert property (last |-> ##[1:100] done_irq);

    // no output is ever unknown after reset
    a_no_x:      assert property (!$isunknown({wr_en, last, busy, done_irq}));

    // coverage: a start that arrives while a frame is already in progress
    c_restart:   cover property (busy && start);
endmodule

// attached to the controller without editing its source
bind capture_ctrl capture_checks u_chk (.*);

The bind statement attaches the checker module to every instance of the capture controller without changing the design source, so the same assertions run in the block-level testbench, in the full front-end simulation and, as lesson 12 shows, in formal verification. Assertions find bugs close to their cause: a scoreboard might report a corrupt frame thousands of cycles after the controller miscounted, while a_cnt_range fires on the exact cycle. The cover property does the reverse of an assertion: it reports whether a scenario ever happened, so a test plan can require that a restart during a frame was exercised.

Interface protocol assertions

Streaming interfaces between blocks, which lesson 10 develops, follow a valid and ready handshake with rules that are easy to state and easy to break. Once a source asserts valid, it must hold valid and keep its data stable until the sink accepts the transfer with ready. Two assertions capture this rule and belong on every streaming interface in the design.

a_hold_valid: assert property (valid && !ready |=> valid);
a_hold_data:  assert property (valid && !ready |=> $stable(data));

Coverage

Coverage answers the question of when to stop. Code coverage is measured automatically by the simulator: line coverage records which statements executed, branch coverage which arms of each if and case were taken, toggle coverage whether each bit went both from 0 to 1 and from 1 to 0, and FSM coverage which states and transitions were visited. Code coverage shows what was not exercised but cannot show that what was exercised was checked, or that the required features exist at all. Functional coverage fills the gap: the verification engineer writes covergroups or cover properties that describe the scenarios in the test plan, such as every channel overloading, a frame aborted in each state, or a full FIFO, and the simulator counts how often each was hit.

Regressions and the cost of simulation

A regression is the full set of tests, each run with several random seeds, executed automatically on every change. The NS-5 front end's regression grows through the course: block-level tests for each module, integration tests on the complete front end with the Python filter model as the reference, and long random runs overnight. Each failure records its seed so that it can be reproduced exactly. RTL simulation of a design of this size runs at tens of thousands of clock cycles per second, so one second of real time at 100 MHz takes over half an hour of simulation. That is why block-level tests with short frames, a frame of four samples rather than 1,024, are the backbone of verification, and why the full-length runs are reserved for the overnight regression.

Simulation also has a blind spot: it checks the RTL, not the synthesized netlist or the device. Synthesis tools are very reliable, but a design that depends on undefined behaviour, such as an X in simulation that synthesis resolves to 0, can differ. Gate-level simulation, equivalence checking and hardware tests close that gap, as lesson 12 describes.