Debugging methodology

Every circuit you build will, at some point, not work. The difference between an engineer who fixes it in twenty minutes and one who is still poking at it after two days is rarely knowledge of the circuit; it is method. Debugging is a form of experiment: you have a hypothesis about what is wrong, you make a measurement that could prove it wrong, and you update. Doing that systematically, writing down what you find, and choosing measurements that cut the problem in half each time turns a frustrating hunt into a short, predictable search. This lesson sets out that method and the common faults it tends to find.

Leo's skills are about to be tested. Three prototype controller boards come back from the assembly house. Board 1 works. Board 2 powers up but none of the signal heads light. Board 3 works on the bench but resets randomly when the LED heads switch on outdoors. Amara gives him boards 2 and 3 and a notebook.

Start with the symptom, not the fix

The first step is to describe precisely what is wrong, because a vague symptom produces random guesses. 'Board 2 doesn't work' leads nowhere. 'Board 2 draws 85 mA at 24 V, the 5 V rail is correct, the microcontroller's status LED blinks normally, but none of the six signal-head outputs ever goes high' points straight at a small part of the design. Compare against a known-good reference wherever possible: board 1 is a working copy, and measuring the same points on both boards is often the fastest debugging technique of all. Write the symptom down, including what does work, because what works rules out large parts of the circuit.

Check the obvious first

A large share of faults are not subtle at all, and checking the boring things first saves hours. Is the board powered, at the right voltage, measured at the chip pins and not just at the connector? Is the ground connected? Is the oscilloscope probe compensated and set to the right attenuation? Is the generator's output turned on? Is the firmware the version you think it is? Is the cable plugged into the right connector? Are there visible problems: a part missing, rotated, scorched or bridged with solder? Experienced engineers develop a checklist of these quick checks and run it before forming any clever theory.

Divide and conquer

When the obvious checks pass, the most powerful technique is half-splitting. Think of the circuit as a chain of stages from input to output: for the signal heads, the microcontroller's shift-register outputs, then the shift registers, then the MOSFET gate drivers, then the MOSFETs, then the LED heads. Measure in the middle of the chain. If the signal is correct there, the fault is downstream; if not, it is upstream. Measure in the middle of the remaining half, and repeat. A chain of N stages needs only about log₂N measurements to locate the faulty stage, instead of up to N if you start at one end and work along.

Make one change at a time, and write it down

When you suspect a cause, test it with a change that can only affect that cause, and then measure again. Changing three things at once, swapping a chip, adding a capacitor and reflashing the firmware, may fix the symptom without telling you which change mattered, and the next board will fail the same way. Keep a running log: the time, what you measured, what you saw, what you changed and the result. It feels slow, but it prevents the most common debugging trap, which is going in circles, re-measuring things you already checked an hour ago because you no longer remember the result. The log also becomes the start of the fault report that goes to the designers and the assembly house.

A good hypothesis must be testable, and a good test is one that could prove it wrong. 'Maybe it is noise' is not testable until you say what noise, from where, and what you would expect to see if it were the cause. 'The reset is caused by the 24 V rail dipping below the regulator's dropout when all six heads switch on' is testable: put a scope on the rail and the reset pin, trigger on the reset, and see whether the rail dips first.

Intermittent faults

Faults that come and go are the hardest to find, because the circuit may behave perfectly while you measure it. They usually come from one of a few causes: a marginal supply that dips under load, a poor solder joint or connector that opens with temperature or vibration, noise or ground bounce that crosses a threshold occasionally, a timing margin that fails at a particular temperature, or firmware that mishandles an unusual sequence of events. The strategy is to capture the event rather than wait for it: set the scope to Single with a trigger on the symptom, such as a reset line going low, and include the suspect signals on other channels. Stressing the circuit, by heating it with a hot-air tool, cooling it with freeze spray, tapping it or varying the supply voltage, often turns an intermittent fault into a repeatable one.

# Half-splitting: number of measurements to isolate one faulty stage among N
import math

def half_split(stages, faulty):
    lo, hi, tests = 0, len(stages) - 1, []
    # Invariant: signal is good at the input of stage lo and bad at the output of stage hi
    while lo < hi:
        mid = (lo + hi) // 2
        ok = mid < faulty               # output of stage mid is correct if the fault is later
        tests.append((stages[mid], "good" if ok else "bad"))
        if ok: lo = mid + 1
        else: hi = mid
    return stages[lo], tests

chain = ["MCU pin", "level shift", "shift reg 1", "shift reg 2", "gate driver",
         "MOSFET", "connector", "LED head"]
for f in (2, 6):
    found, tests = half_split(chain, f)
    print(f"fault in '{chain[f]}': found '{found}' after {len(tests)} tests: {tests}")
for n in (8, 16, 64, 1000):
    print(f"{n:5d} stages: at most {math.ceil(math.log2(n))} half-split tests vs up to {n} sequential")

fault in 'shift reg 1': found 'shift reg 1' after 3 tests: [('shift reg 2', 'bad'), ('level shift', 'good'), ('shift reg 1', 'bad')]
fault in 'connector': found 'connector' after 3 tests: [('shift reg 2', 'good'), ('MOSFET', 'good'), ('connector', 'bad')]
    8 stages: at most 3 half-split tests vs up to 8 sequential
   16 stages: at most 4 half-split tests vs up to 16 sequential
   64 stages: at most 6 half-split tests vs up to 64 sequential
 1000 stages: at most 10 half-split tests vs up to 1000 sequential

The program finds a fault in either half of the eight-stage chain in three tests, and shows why the method matters more as systems grow: a thousand-stage chain, such as a long list of firmware commits or a set of wire connections, needs only ten tests. The same idea drives the bisect tools that software engineers use to find which change in a long history introduced a bug.

Common hardware faults