Firmware architecture, testing and security: a capstone

The previous eleven lessons took the microcontroller apart: its core and memory, its toolchain, its pins, converters, timers, interrupts, buses and power modes. A product needs all of them working together for years, in the field, without anyone watching. This capstone is about the whole. It covers how firmware is structured, from a simple loop to state machines to a real-time operating system; how firmware is tested before it reaches hardware and on hardware; how it is debugged; how it recovers from faults with watchdogs and safe states; and how it is updated and protected against attackers. It ends with the NS-5 design review, where Jonah presents the firmware architecture that draws on every lesson in the course.

Superloops and cooperative scheduling

The simplest firmware architecture is the superloop: after initialization, a main loop calls each part of the application in turn, forever, and interrupt handlers do the urgent work and set flags for the loop. The NS-4 began as one. A superloop is easy to understand, needs no operating system and has no concurrency inside the loop, so tasks never preempt each other. Its weakness is response time. Every task waits for every other task in the loop, so one slow function, such as parsing a long modem response, delays everything; and long jobs must be split by hand into small steps that each return quickly.

A common refinement is a cooperative scheduler: tasks are functions that run to completion at set periods or when events arrive, driven by software timers from a tick or a low-power timer, as in lessons 6 and 11. Each task must be short, and long work is broken into a sequence of steps, usually written as a state machine. Cooperative designs are deterministic and simple to reason about, and they fit naturally with tickless sleep: when no task is ready, the loop sleeps until the next deadline.

State machines

Much embedded behavior is naturally a finite state machine: the device is in one of a small number of states, and events cause transitions with actions. A modem connection, an I²C transaction, a button press with debouncing, a charging cycle and a firmware update all fit this pattern. Writing them as explicit state machines, rather than as nests of flags and conditions, makes the behavior visible, testable and reviewable. Every combination of state and event has a defined outcome, including the ones that 'cannot happen', which in the field always eventually do.

typedef enum { ST_IDLE, ST_POWERING, ST_REGISTERING, ST_CONNECTED, ST_SENDING, ST_BACKOFF } up_state_t;
typedef enum { EV_TIMER, EV_MODEM_READY, EV_ATTACHED, EV_SESSION_OPEN, EV_ACKED, EV_TIMEOUT, EV_ERROR } up_event_t;

static up_state_t st = ST_IDLE;
static uint8_t failures;

void upload_handle(up_event_t ev)
{
    switch (st) {
    case ST_IDLE:
        if (ev == EV_TIMER) { modem_power_on(); timeout_start(15000); st = ST_POWERING; }
        break;
    case ST_POWERING:
        if (ev == EV_MODEM_READY)  { modem_attach(); timeout_start(120000); st = ST_REGISTERING; }
        else if (ev == EV_TIMEOUT) { goto fail; }
        break;
    case ST_REGISTERING:
        if (ev == EV_ATTACHED)     { modem_open(); timeout_start(30000); st = ST_CONNECTED; }
        else if (ev == EV_TIMEOUT || ev == EV_ERROR) { goto fail; }
        break;
    case ST_CONNECTED:
        if (ev == EV_SESSION_OPEN) { send_batch(); timeout_start(60000); st = ST_SENDING; }
        else if (ev == EV_TIMEOUT || ev == EV_ERROR) { goto fail; }
        break;
    case ST_SENDING:
        if (ev == EV_ACKED)        { modem_power_off(); failures = 0; st = ST_IDLE; }
        else if (ev == EV_TIMEOUT || ev == EV_ERROR) { goto fail; }
        break;
    case ST_BACKOFF:
        if (ev == EV_TIMER) { modem_power_on(); timeout_start(15000); st = ST_POWERING; }
        break;
    }
    return;
fail:
    modem_power_off();
    if (++failures >= 5) { failures = 0; st = ST_IDLE; log_event(LOG_UPLOAD_GAVE_UP); }
    else { timer_start_backoff(60000u << failures); st = ST_BACKOFF; }
}

# Test the upload state machine against event sequences, including the ones that "cannot happen"
T = {  # (state, event) -> next state; anything missing is ignored in that state
    ("IDLE", "timer"): "POWERING", ("BACKOFF", "timer"): "POWERING",
    ("POWERING", "ready"): "REGISTERING", ("REGISTERING", "attached"): "CONNECTED",
    ("CONNECTED", "open"): "SENDING", ("SENDING", "acked"): "IDLE",
}
FAILS = {"POWERING", "REGISTERING", "CONNECTED", "SENDING"}
def run(events):
    st, failures, modem_on = "IDLE", 0, False
    for ev in events:
        if st in FAILS and ev in ("timeout", "error"):
            modem_on, failures = False, failures + 1
            st = "IDLE" if failures >= 5 else "BACKOFF"
            if failures >= 5: failures = 0
            continue
        nxt = T.get((st, ev), st)
        if nxt == "POWERING": modem_on = True
        if nxt == "IDLE" and ev == "acked": modem_on, failures = False, 0
        st = nxt
    return st, modem_on
tests = {
    "normal upload": ["timer", "ready", "attached", "open", "acked"],
    "network never attaches": ["timer", "ready", "timeout"],
    "five failures then give up": ["timer", "timeout"] + ["timer", "timeout"] * 4,
    "stray ack while idle": ["acked", "timer", "ready"],
    "error while sending": ["timer", "ready", "attached", "open", "error"],
}
for name, evs in tests.items():
    st, on = run(evs)
    print(f"{name:28s} -> state {st:12s} modem {'ON' if on else 'off'}")

normal upload                -> state IDLE         modem off
network never attaches       -> state BACKOFF      modem off
five failures then give up   -> state IDLE         modem off
stray ack while idle         -> state REGISTERING  modem ON
error while sending          -> state BACKOFF      modem off

Real-time operating systems

As firmware grows, a superloop becomes hard to keep responsive. A real-time operating system provides preemptive tasks, each with its own stack and priority, plus queues, semaphores, mutexes, event flags and timers. A high-priority task waiting for a radar frame runs as soon as the DMA interrupt posts to its queue, preempting a lower-priority task that is formatting a log record. That makes response times independent of the length of unrelated code, and makes it natural to structure a complex system as cooperating tasks with clear interfaces.

The cost is concurrency. Tasks share data, so every shared structure needs a mutex or a queue, and lesson 7's race conditions now occur between tasks as well as between tasks and interrupts. Mutexes can cause priority inversion, where a high-priority task waits for a mutex held by a low-priority task that is itself preempted by a medium-priority one; priority inheritance, offered by most real-time kernels, fixes this. Each task needs a stack sized for its worst case, and stack overflow is a classic source of memory corruption. Scheduling analysis can show whether a set of periodic tasks will meet their deadlines; under rate-monotonic priorities, a sufficient test is that total utilization stays under a bound that falls towards 69 percent as tasks are added, and exact response-time analysis gives tighter answers. CMP-440 Real-Time Systems covers these methods in depth.

# Rate-monotonic utilization check for the NS-5 tasks (sufficient test) plus exact response times
import math
tasks = [  # name, period ms, worst-case execution ms; shorter period = higher priority
    ("radar frame",   6.25, 1.6),
    ("classify",     50.0,  9.0),
    ("CAN service",  10.0,  0.8),
    ("log and store", 1000.0, 40.0),
]
tasks.sort(key=lambda t: t[1])
U = sum(c / p for _, p, c in tasks)
n = len(tasks)
bound = n * (2 ** (1 / n) - 1)
print(f"utilization {U:.3f}, bound for {n} tasks {bound:.3f} -> {'passes' if U <= bound else 'inconclusive'}")
for i, (name, p, c) in enumerate(tasks):
    r = c
    while True:
        r_new = c + sum(math.ceil(r / pj) * cj for _, pj, cj in tasks[:i])
        if r_new == r or r_new > p: break
        r = r_new
    print(f"  {name:14s} period {p:7.2f} ms, response {r_new:6.2f} ms, {'meets' if r_new <= p else 'MISSES'} deadline")

utilization 0.556, bound for 4 tasks 0.757 -> passes
  radar frame    period    6.25 ms, response   1.60 ms, meets deadline
  CAN service    period   10.00 ms, response   2.40 ms, meets deadline
  classify       period   50.00 ms, response  15.40 ms, meets deadline
  log and store  period 1000.00 ms, response  89.20 ms, meets deadline

The NS-5's utilization of about 0.56 passes the sufficient test, and the exact analysis confirms every task meets its deadline with room to spare. Jonah recommends a small real-time kernel for the NS-5 cabinet controller, where CAN, USB, classification and logging all compete, and keeps the pole-top node's cooperative scheduler, which is simpler, already proven and easier to put to sleep.

Testing embedded software

Testing firmware only on hardware is slow, and many faults never appear on a bench. Good embedded teams therefore test in layers. Unit tests run the application logic on a development computer, compiled for the host, with hardware access replaced by small fake drivers: the debouncer of lesson 4, the ring buffer of lesson 7, the CRC and message parser of lesson 8 and the state machine above can all be tested this way in milliseconds. Keeping register access in thin driver layers, as lesson 3 recommended, is what makes this possible. Integration tests run on the target, often hardware-in-the-loop, where a test computer drives the device's inputs and checks its outputs: a signal generator imitates the radar, a programmable supply imitates a dying battery, and a CAN interface plays the cabinet controller.

Some tests target the embedded environment specifically. Timing tests measure interrupt latency and task response times with a logic analyzer and spare pins. Power tests, as in lesson 11, run scripted days and check average current. Environmental tests run units in a chamber from −40 °C to +70 °C. Fault-injection tests cut power during flash writes, disconnect sensors, corrupt messages and hold buses low, and check that the firmware recovers. Static analysis tools catch undefined behavior, uninitialized variables and buffer overruns before any test runs, and coding standards such as MISRA C, used widely in automotive and industrial work, restrict the language to a safer subset: no dynamic memory after startup, no unchecked conversions, every switch with a default, and many more rules aimed at making C predictable.

Debugging

Most microcontrollers have an on-chip debug port, usually a two-wire serial wire interface or JTAG, through which a debug probe can halt the core, set breakpoints, read and write memory and registers, and program flash. Breakpoints are powerful but stop time, which a running motor or radio cannot tolerate, so embedded debugging also relies on non-intrusive methods: watchpoints that trigger on a memory access, trace output that streams program-counter samples or instrumented events through a dedicated pin, real-time variable monitoring through the debug port while the core runs, logging to a buffer drained in idle time, and toggling spare pins for a logic analyzer. When a field unit fails, the fault records of lesson 7, the error counters of lessons 8 to 10 and the reset-cause register are often the only evidence; designing them in from the start is part of the architecture.

Watchdogs and safe states

A watchdog timer resets the microcontroller unless firmware refreshes it periodically. It is the last line of defense against hangs, runaway code and corrupted state. A watchdog only helps if it is refreshed in the right place. Kicking it from a timer interrupt is useless, because the interrupt keeps running while the main loop is stuck. A better design refreshes it from one place in the main loop or a supervisor task, and only when every monitored task has checked in since the last refresh. Windowed watchdogs also reset if the refresh comes too early, which catches code that is looping too fast through a broken path. An independent watchdog with its own oscillator keeps working even if the main clock fails.

/* Supervisor: refresh the watchdog only when every task has reported progress. */
#define TASK_RADAR   (1u << 0)
#define TASK_CLASSIFY (1u << 1)
#define TASK_UPLOAD  (1u << 2)
#define TASK_ALL     (TASK_RADAR | TASK_CLASSIFY | TASK_UPLOAD)

static volatile uint32_t alive;

void task_checkin(uint32_t bit) { __disable_irq(); alive |= bit; __enable_irq(); }

void supervisor_poll(void)              /* called once per main-loop pass */
{
    if ((alive & TASK_ALL) == TASK_ALL) {
        alive = 0;
        IWDG->KR = 0xAAAA;              /* refresh the independent watchdog */
    }
}

After a reset, firmware should know why it reset. The reset-cause register distinguishes power-on, brownout, watchdog, software and pin resets, and firmware should log the cause and count watchdog resets. A device that resets repeatedly should fall back to a safe, minimal mode rather than looping forever: for the NS-5, that means keep counting vehicles with the simplest detection algorithm, stop classification, and report the problem at the next upload. Safety-critical systems go much further, with redundant processors, memory and register self-tests and formal safety standards, but the principle of failing to a defined safe state applies to every product.

Firmware updates

Field devices need updates to fix bugs, close security holes and add features, and an update is the riskiest operation a device performs. If power fails halfway through writing new firmware, the device must still boot. The standard design has a small, rarely changed bootloader at the start of flash and two application slots. The new image is downloaded into the inactive slot, checked, marked as pending, and the device reboots into it. The new firmware must confirm that it works, for example by completing a successful upload, within a set time; if it does not, or if the watchdog fires repeatedly, the bootloader reverts to the previous image. This A/B scheme with rollback is how the NS-5 will update over the cellular link, and Jonah's flash budget from lesson 3 reserves room for it.

Security

Connected embedded devices are targets. Attackers may want to read data, use the device as a foothold into a network, disrupt the service it provides, or simply add it to a botnet. Field devices are physically accessible too: anyone with a ladder can reach the NS-4. Security has to be designed in, and most of it is ordinary engineering discipline rather than exotic cryptography.

Secure boot is the foundation. The bootloader, stored in flash that cannot be rewritten in the field, holds a public key and checks the signature of the application before running it; only images signed with the company's private key, kept offline, will run. Hashing and signature verification are available as hardware accelerators on many microcontrollers. A version counter in one-time-programmable memory prevents rolling back to signed but vulnerable versions. Per-device keys mean that extracting a key from one unit compromises only that unit. Production units lock their debug ports. And because most attacks exploit ordinary bugs, the testing and coding practices above are security practices too. Regulations increasingly require these measures for connected products, including vulnerability handling and update support over a product's life, so they are now part of the specification rather than an optional extra.

The NS-5 design review

At the end of his term, Jonah presents the NS-5 firmware architecture to Maren, Ines and Dana Okoye's server team. The pole-top node runs a cooperative scheduler on a 1 ms low-power timer with tickless sleep. Radar frames arrive by SPI with DMA at 16 MHz; classification runs in a task between frames; detections are timestamped by input capture; the RS-485 port, the I²C sensor bus with recovery at boot, and the modem's upload state machine run as tasks; and a supervisor feeds the watchdog. The interrupt priorities follow the deadline table of lesson 7. The power budget shows 2.1 mA average at the battery in summer and about 126 days on a full charge in January. Updates use a signed A/B scheme with rollback. Unit tests run on every commit, and a hardware-in-the-loop rig with a radar simulator runs nightly.

Maren's only question is the one every firmware lead asks: what happens when it fails? Jonah's answer, the fault records, error counters, reset-cause log, safe mode and rollback, is the last piece of the design, and the reason the review passes. Embedded systems are judged not by how they behave on the bench but by how they behave on a pole in January, years later, when no one is watching.