Real-time communication, middleware and time synchronization

A robot is a distributed computer. Its joint drives, force sensors, IMUs, cameras, safety controller and main computer are separate processors joined by buses and networks, and a control loop often spans several of them: the encoder reading travels to the controller, the command travels to the drive. The network is therefore part of every latency budget, and its worst case matters as much as any task's. This lesson covers the timing of the networks robots use, from CAN to EtherCAT and time-sensitive Ethernet, the quality-of-service settings that make DDS and ROS 2 middleware behave predictably, and how clocks on separate devices are synchronized so that measurements taken in different places can be combined. ROB-200 described the physical and electrical side of UART, I2C, SPI and CAN; this lesson treats networks as schedulable resources.

What makes a network real-time

A real-time network bounds the time from when a message is ready to send until it is delivered, for every message, even when the network is busy. Ordinary Ethernet with TCP/IP does not: switches queue frames, collisions and congestion cause variable delays, and TCP retransmits lost data after timeouts measured in hundreds of milliseconds. Real-time networks achieve bounded delay in one of three ways: by priority arbitration (CAN), by a master that schedules all traffic in fixed cycles (EtherCAT, PROFINET IRT, many fieldbuses), or by time-aware scheduling of switch queues (time-sensitive networking, TSN).

CAN timing

CAN arbitrates by message identifier: when several nodes start sending at once, the lowest identifier wins without being delayed and the others back off and retry. CAN is therefore a fixed-priority, non-preemptive scheduler for messages, and the response-time analysis of lesson 5 applies to it, with one twist: a frame already being transmitted cannot be interrupted, so a high-priority message can be blocked for up to one maximum-length frame. The worst-case transmission time of a standard-identifier frame with s data bytes, allowing for the stuff bits CAN inserts, is

where \( \tau_{bit} \) is the bit time. An 8-byte frame at 1 Mbit/s therefore takes up to 135 µs, and bus load is each message's worst-case time divided by its period, summed over messages. Classic CAN tops out at 1 Mbit/s and 8 data bytes per frame; CAN FD raises the data phase to several Mbit/s and 64 bytes, and CAN XL, now emerging, goes further. Many mobile robots and drones use CAN or CAN FD between the main controller and motor drives, with protocols such as CANopen and DroneCAN defining the message layout.

import math
tbit = 1e-6                               # 1 Mbit/s classic CAN
msgs = [  # name, CAN id (lower = higher priority), data bytes, period (s)
    ("estop_heartbeat", 0x010, 1, 0.010),
    ("joint_cmd_1_3",   0x100, 8, 0.001),
    ("joint_cmd_4_6",   0x101, 8, 0.001),
    ("joint_fb_1_3",    0x180, 8, 0.001),
    ("joint_fb_4_6",    0x181, 8, 0.001),
    ("imu",             0x200, 8, 0.002),
    ("battery",         0x400, 6, 0.100),
]
msgs.sort(key=lambda m: m[1])
C = {m[0]: (55 + 10 * m[2]) * tbit for m in msgs}
load = sum(C[m[0]] / m[3] for m in msgs)
print("bus load: %.1f %%" % (100 * load))
Bmax = max(C.values())                    # non-preemptive blocking by one frame
for i, (name, cid, s, T) in enumerate(msgs):
    w = Bmax                              # queuing delay, iterate
    while True:
        nxt = Bmax + sum(math.ceil((w + tbit) / Tj) * C[nj] for nj, _, _, Tj in msgs[:i])
        if abs(nxt - w) < 1e-12:
            break
        w = nxt
    print("%-16s worst-case latency %6.0f us" % (name, 1e6 * (w + C[name])))

bus load: 61.5 %
estop_heartbeat  worst-case latency    200 us
joint_cmd_1_3    worst-case latency    335 us
joint_cmd_4_6    worst-case latency    470 us
joint_fb_1_3     worst-case latency    605 us
joint_fb_4_6     worst-case latency    740 us
imu              worst-case latency    875 us
battery          worst-case latency    990 us

At about 60 % load the lowest-priority battery message still arrives within a millisecond, but the analysis shows the cost of a busy bus: every message can wait for one maximum-length frame already on the wire, and lower identifiers stack interference on higher ones. A common design rule keeps CAN load below 50 to 70 % so that bursts and retransmissions after errors still fit.

EtherCAT and cyclic fieldbuses

EtherCAT, developed by Beckhoff and now an IEC standard maintained by the EtherCAT Technology Group, is the most common real-time network in robot arms and industrial machines. The master sends one Ethernet frame that passes through every device in a line or ring; each device reads its commands and inserts its feedback on the fly, in hardware, as the frame passes, adding well under a microsecond of delay per device. A single frame can update dozens of drives, so cycle times of 100 µs to 1 ms are routine, and EtherCAT's distributed clocks synchronize every device to well under a microsecond so that all axes sample and actuate at the same instant. Because the master controls all traffic, timing is deterministic by construction; the master itself, often a PREEMPT_RT Linux computer running an open-source stack such as IgH EtherCAT Master or SOEM, must send its frame on time every cycle, which is one of the most demanding real-time jobs on a robot computer.

Time-sensitive networking

Time-sensitive networking is a family of IEEE 802.1 standards that adds real-time guarantees to standard Ethernet, so that control traffic and bulk data such as camera streams can share one network. Key parts are IEEE 802.1AS, a profile of the precision time protocol that synchronizes every switch and end device; IEEE 802.1Qbv, the time-aware shaper, which opens and closes switch queues on a repeating schedule so that control frames always find an empty path; and frame preemption (IEEE 802.1Qbu and 802.3br), which lets an urgent frame interrupt a long one. TSN is spreading into industrial automation and vehicles, and OPC UA over TSN is one of several initiatives to use it for interoperable machine control.

Middleware: DDS and ROS 2 quality of service

ROS 2 communicates through DDS (Data Distribution Service) implementations such as eProsima Fast DDS, Eclipse Cyclone DDS and RTI Connext, or through the newer Zenoh middleware. Their timing behaviour depends heavily on quality-of-service (QoS) settings, which every real-time designer should set deliberately rather than leave at defaults.

• Reliability: RELIABLE retransmits lost samples and can make a publisher wait or buffer; BEST_EFFORT drops them. For high-rate sensor and command streams, where the next sample replaces the last, best effort avoids head-of-line delays.
• History and depth: KEEP_LAST with a small depth keeps only the newest samples, bounding memory and latency; KEEP_ALL can grow without bound.
• Deadline: declares the maximum expected period between samples; the middleware notifies the application when it is missed, a useful watchdog for a stale command stream.
• Lifespan: discards samples older than a set age, so a late command is never acted on.
• Durability and liveliness: whether late-joining subscribers receive past data, and how failures of publishers are detected.

A typical choice for a 500 Hz joint command topic is best effort, keep last with depth 1, a deadline of 4 ms and a lifespan of 4 ms: the subscriber always uses the newest command, never an old one, and is told within two cycles if commands stop arriving. Inside one process, ROS 2's intra-process communication and loaned messages can pass data without copying or serialization, cutting both latency and jitter.

Clock synchronization

When a lidar on one processor and an IMU on another timestamp their data, those timestamps can only be compared if the two clocks agree. Crystal oscillators drift by tens of parts per million, so two free-running clocks can disagree by several milliseconds after a few minutes. Network Time Protocol keeps computers within about a millisecond over the internet and better on a local network, which is enough for logging but not for sensor fusion at speed. The Precision Time Protocol (PTP, IEEE 1588) synchronizes devices on a local network to well under a microsecond when hardware timestamping is used. A master clock and each slave exchange timestamped messages, and the slave computes its offset and the path delay:

Here \( t_1 \) is when the master sent Sync, \( t_2 \) when the slave received it, \( t_3 \) when the slave sent Delay_Req and \( t_4 \) when the master received it. The calculation assumes the path delay is the same in both directions; asymmetry shows up directly as offset error. Hardware timestamping in the network interface removes the variable software delays that would otherwise dominate. On Linux, the linuxptp tools (ptp4l and phc2sys) implement PTP and IEEE 802.1AS, and many cameras, lidars and industrial devices support PTP natively.

# PTP offset and delay from four timestamps (nanoseconds)
exchanges = [  # t1 master send, t2 slave receive, t3 slave send, t4 master receive
    (1_000_000_000, 1_000_012_450, 1_000_500_000, 1_000_487_710),
    (1_125_000_000, 1_125_012_430, 1_125_500_000, 1_125_487_730),
    (1_250_000_000, 1_250_012_470, 1_250_500_000, 1_250_487_690),
]
for t1, t2, t3, t4 in exchanges:
    offset = ((t2 - t1) - (t4 - t3)) / 2
    delay = ((t2 - t1) + (t4 - t3)) / 2
    print("offset %+7.0f ns   path delay %6.0f ns" % (offset, delay))
# a 400 ns asymmetry (forward path longer) appears as a 200 ns offset error
fwd, back = 1200, 800
print("asymmetry error: %.0f ns" % ((fwd - back) / 2))

offset  +12370 ns   path delay     80 ns
offset  +12350 ns   path delay     80 ns
offset  +12390 ns   path delay     80 ns
asymmetry error: 200 ns

The slave's clock is about 12 µs ahead of the master's, with a path delay of about 80 ns, consistent with a direct link of roughly 15 m of cable; the servo in ptp4l would now steer the slave clock to remove the offset, typically converging to tens of nanoseconds with hardware timestamps. For a robot moving at 2 m/s, a 1 ms timestamp error between camera and odometry is a 2 mm position error, and at 1 rad/s of rotation a 1 ms error misplaces a landmark 10 m away by 1 cm, which is why tightly coupled visual-inertial estimators synchronize sensors in hardware, with trigger lines or PTP, rather than relying on software timestamps.