Integration and the integration plan

Integration is where a robot becomes a robot, and where most capstone schedules break. Subsystems that each passed their bench tests start interfering with each other: the motors inject electrical noise into the IMU, the arm's swing shifts the centre of mass and the wheels slip, the perception node's processing delay makes the robot overshoot items, and the battery voltage sags when the arm and motors draw current together. This lesson covers how to plan integration so that problems appear one at a time, how to manage time and latency across a ROS 2 system, how to debug an integrated robot systematically, and how to keep a disciplined record of what was integrated and what was found.

Integration strategies

The worst strategy, and a common one, is big-bang integration: assembling everything at once in week 11 and switching it on. When it fails, as it will, there are too many possible causes to diagnose. Incremental integration adds one subsystem or capability at a time and tests after each step, so each new problem has a small number of candidate causes. Bottom-up integration starts with low-level components, such as motors, drivers and firmware, and builds upward. Top-down integration starts with the high-level software running against stubs or simulation and replaces stubs with real hardware one by one. Thread-based integration builds one end-to-end capability, such as "drive to a waypoint", across all layers, and then adds further threads.

Most robotics teams combine these approaches. Team Tern's plan used threads, each building on the last: first, teleoperated driving with the emergency stop and safety lidar; second, autonomous waypoint following with localization; third, litter detection while driving, with detections logged but not acted on; fourth, approaching a detected item and stopping in a grasp pose; fifth, grasping and depositing in the bin; and sixth, a full patrol with logging and reporting. Each thread has entry criteria, such as the subsystems it needs having passed bench tests, and exit criteria, such as a demonstration repeated five times in the test area. This sequence also means that if the term runs short, the robot is still useful at the last completed thread, which is the staged-concept idea from lesson 3 applied to integration.

Preparing for integration

Integration goes faster when the groundwork is done. Before each thread, confirm that the subsystems have passed their bench tests, that the ICD is current and both sides agree on it, that software versions are tagged and recorded, that the integration test procedure is written, and that the test area, safety equipment and people are booked. An integration readiness checklist, reviewed in five minutes at the start of each session, prevents the familiar experience of spending the first two hours of a lab booking discovering that someone's branch was never merged or the battery was not charged.

Mechanical and electrical integration also need planning. Cable routing, connector access, service loops for moving parts, and access to fuses and batteries should be designed in, not improvised with cable ties at the end. Team Tern made a rule that nothing goes on the robot that is not in the master CAD assembly and the wiring diagram. When they added a status light late in the term, the change went through both documents first, and they discovered that the light's planned position blocked part of the lidar's field of view.

Time, synchronization and latency

An integrated robot is a distributed real-time system, and timing problems are among the most common integration surprises. Each sensor message carries a timestamp, and fusing data from several sensors only works if those timestamps share a clock. On a single computer, ROS 2 uses the system clock; across several computers, the clocks must be synchronized, typically with chrony using the Network Time Protocol, or with the Precision Time Protocol for tighter accuracy. Sensors with their own clocks, such as some lidars and GNSS receivers, need their timestamps converted or synchronized as well. A camera image timestamped 80 ms late will place every detection in the wrong position while the robot is moving.

Latency, the delay from an event in the world to the robot's response, determines how fast the robot can safely move and how accurately it can act. A latency budget, like the resource budgets in lesson 4, allocates the end-to-end delay across the pipeline: sensor exposure and transfer, perception, planning, control and actuation. Because each stage's delay varies, the budget should consider not only the average but the high percentiles; a pipeline that is fast on average but occasionally takes 400 ms can cause collisions or missed grasps. ROB-320 lesson 1 introduced latency budgets for embedded systems; here they are applied across the ROS 2 graph.

# End-to-end latency budget across Tern's ROS 2 pipeline (Monte Carlo, illustrative distributions)
import numpy as np
rng = np.random.default_rng(7)
N = 100_000
stages = {   # stage: samples in ms
    "camera exposure + USB": rng.normal(28, 3, N),
    "detector (GPU)":        rng.gamma(9.0, 6.0, N),            # mean 54 ms, long tail
    "3D projection + tf":    rng.normal(6, 1, N),
    "planner (10 Hz)":       rng.uniform(0, 100, N),            # 10 Hz planner: wait for next cycle
    "controller + serial":   rng.normal(12, 2, N),
    "motor response":        rng.normal(40, 5, N),
}
total = sum(stages.values())
for k, v in stages.items():
    print(f"{k:22s} mean {v.mean():6.1f} ms   p99 {np.percentile(v, 99):6.1f} ms")
p50, p95, p99 = np.percentile(total, [50, 95, 99])
print(f"end to end: median {p50:.0f} ms, p95 {p95:.0f} ms, p99 {p99:.0f} ms")
v = 0.5   # approach speed, m/s
print(f"distance moved during p99 latency at {v} m/s: {v * p99 / 1000 * 100:.1f} cm")
# effect of running the planner at 20 Hz instead of 10 Hz
total20 = total - stages["planner (10 Hz)"] + rng.uniform(0, 50, N)
print(f"with a 20 Hz planner: p99 {np.percentile(total20, 99):.0f} ms")

camera exposure + USB  mean   28.0 ms   p99   34.9 ms
detector (GPU)         mean   54.0 ms   p99  104.8 ms
3D projection + tf     mean    6.0 ms   p99    8.3 ms
planner (10 Hz)        mean   50.0 ms   p99   99.0 ms
controller + serial    mean   12.0 ms   p99   16.7 ms
motor response         mean   40.0 ms   p99   51.6 ms
end to end: median 190 ms, p95 246 ms, p99 267 ms
distance moved during p99 latency at 0.5 m/s: 13.4 cm
with a 20 Hz planner: p99 225 ms

The simulation shows two things. The largest contributors are the detector, whose processing time has a long tail, and the planner's waiting time: a 10 Hz planner may receive a detection just after it has run and wait up to 100 ms for its next cycle. At an approach speed of 0.5 m/s, the high-percentile latency means the robot moves roughly 13 to 14 cm between seeing an item and responding, which is larger than the 5 cm grasp-pose tolerance in thread 4's exit criteria. Running the planner at 20 Hz cuts the tail; slowing the final approach to 0.1 m/s cuts the distance travelled further; and predicting the item's position relative to the moving robot, using odometry and the image timestamp, removes most of the remaining error. Team Tern used all three. Without a latency budget, the symptom, the robot stopping a little past each item, would likely have been blamed on the detector or the arm.

Debugging an integrated robot

When something fails in integration, resist the urge to change several things at once. A systematic approach starts by reproducing the problem reliably, then narrowing it down by halving the system: does the problem occur in simulation? With recorded data? With the arm disconnected? With motors off? Each answer eliminates half the candidates. Change one thing at a time, keep notes, and when the cause is found, write a test that would have caught it.

ROS 2 provides good tools for this. ros2 topic hz and ros2 topic delay check rates and latencies against the ICD; ros2 node info and rqt_graph show the actual graph, which often reveals a misspelled topic name or a quality-of-service mismatch, for example a sensor publishing with best-effort reliability to a subscriber that requires reliable delivery, in which case no messages arrive. ros2 run tf2_tools view_frames and tf2_echo check the coordinate-frame tree, where a missing or inverted transform is a classic cause of detections appearing behind the robot. Recording everything with ros2 bag during integration sessions allows a problem seen once to be replayed and studied offline.

The integration log

Integration sessions should be logged like lab experiments. Each entry records the date, people present, configuration (hardware revision, firmware and software tags), what was attempted, what happened, any data recorded with file names, issues found and actions assigned. Issues go into the team's tracker with steps to reproduce. This record serves three purposes: it prevents the team from rediscovering the same problem, it provides evidence for the test report and the final report, and it gives examiners insight into the team's engineering process. A short integration log entry written at the end of each session takes ten minutes and repays itself many times over.

Integration also changes the plan. Problems found may require design changes, which go through the change log, updated budgets and, if they are large, a brief delta review with the advisor. The project buffer from lesson 5 exists precisely for this. Team Tern's integration of thread 5 took four days longer than planned because of the brown-out and a gripper alignment problem, and the buffer absorbed it. A team without buffer would have had to cut thread 6, the full patrol, which was what the partner most wanted to see.

If your robot connects to the partner's network or transmits data from a public place, check privacy expectations early. In Canada, municipal partners are generally covered by provincial public-sector privacy legislation, such as Ontario's Municipal Freedom of Information and Protection of Privacy Act, so images of members of the public collected by Tern may be subject to the partner's privacy obligations; lesson 9 discusses how to minimize and protect such data.