Communication: speech, gesture, gaze, light and sound

People coordinate with each other constantly and mostly without thinking: a glance shows where someone is about to reach, a hand held out signals readiness to take an object, a pause invites the other person to speak. When a robot shares a task or a space with people, the same coordination has to happen, but nothing about it is automatic. The robot must be able to read human signals and to send signals that people read correctly. This lesson surveys the channels available, speech, gesture, gaze, motion, light and sound, then looks closely at one of the most influential ideas in HRI communication: the difference between motion that is predictable and motion that is legible.

Channels and their trade-offs

Each communication channel has strengths and weaknesses. Speech carries rich, precise content but is slow, fails in noise and excludes people who do not share the language or cannot hear. Gestures are fast and work at a distance but are ambiguous and culturally variable. Gaze, whether from human eyes or a robot's head or screen eyes, signals attention and intent with very little effort but needs the observer to be looking. Light signals are visible across a noisy factory but carry only a few distinct meanings. Sound alerts reach people who are not looking but can be masked by machinery and become irritating if overused. The robot's own motion is always communicating something, whether the designer intended it or not.

Because no single channel is reliable in all conditions, good designs are multimodal and redundant: a mobile robot about to turn might slow down, rotate its head or sensor mast toward the new direction, project an arrow on the floor and flash an indicator light. Redundancy also supports accessibility, because a person who cannot see the light may hear the tone or notice the motion. Lesson 10 develops this point.

Speech

Spoken interaction requires a chain of components: voice activity detection, automatic speech recognition (ASR), natural language understanding to extract the intent, a dialogue manager that tracks context, response generation and text-to-speech. Large language models have made the understanding and generation steps far more flexible, but they also introduce new risks: a robot can now produce fluent replies that sound confident while being wrong, and can be talked into actions outside its intended scope. For a robot that acts in the physical world, language output should be grounded in what the robot can actually perceive and do, and physical commands should be mapped onto a limited, validated set of skills rather than executed freely.

Recognition accuracy is measured by word error rate, WER, the minimum number of substitutions S, deletions D and insertions I needed to turn the recognized text into the reference, divided by the number of reference words N:

WER treats all words equally, but in commanding a robot some words matter far more than others. Good dialogue design confirms critical parameters ("Moving the red bin to station 4, correct?"), offers a quick way to cancel, and uses a constrained vocabulary in noisy environments. Factory floors often exceed 80 dB(A), where recognition degrades sharply without close-talking microphones. Speech also creates accessibility barriers for people with speech impairments, strong accents not represented in training data, or limited proficiency in the robot's language, so speech should rarely be the only way to command a robot.

Gesture

Gestures are commonly grouped into deictic gestures that point at things, iconic gestures that depict shapes or motions, emblems with a conventional meaning such as a thumbs-up, and beats that accompany speech rhythm. Robots recognize gestures using cameras with body-pose estimation, depth sensors or wearable sensors such as inertial bands. Deictic pointing is the most useful in practice, often combined with speech: "put that there" requires the robot to resolve "that" and "there" from where the person points. Pointing is imprecise, so the robot must handle uncertainty, for example by highlighting its best guess and asking for confirmation when two objects are close together.

Emblems vary across cultures. A thumbs-up, an OK sign or a beckoning motion with the palm up can be friendly in one place and rude in another, and the gesture for stop is not universal either. Industrial settings often define a small vocabulary of standardized hand signals, similar to those used by crane operators, and teach them explicitly. Robots can also gesture back. A humanoid or a mobile manipulator can point at an object it is about to fetch, and even a simple arm can perform a short wave or a nod-like motion to acknowledge a command.

Gaze

Gaze is one of the richest nonverbal signals. People look at what they are about to reach for, look at the person whose turn it is to speak, and follow each other's gaze to establish joint attention on an object. Research by Bilge Mutlu and colleagues showed that a robot's gaze shifts can shape who takes the next turn in a group conversation and improve how well listeners recall a robot's story. In object handovers, studies by AJung Moon and colleagues found that when a robot looked toward the handover location as it extended the object, people reached for it sooner, making handovers smoother.

Robots can show gaze with a movable head, with eyes drawn on a screen, or simply by orienting a camera. The robot can also read human gaze using eye trackers or head-pose estimation to predict what a person is about to grab, letting a collaborative arm avoid reaching for the same part. Gaze must be used with care: a robot that stares continuously can feel intimidating, while one that never looks at people seems inattentive. Cultural norms about eye contact also differ.

Light and sound

Light is the cheapest and most common robot signal. Industrial cells use stack lights, typically with red for a fault or stop, amber for a warning or attention needed and green for normal running; indicator colors for machinery are standardized in IEC 60204-1 and IEC 60073, adopted in Canada and the United States through national electrical codes and standards. Collaborative and mobile robots often use LED rings or strips around joints or the base. Projected light can show a robot's planned path on the floor or the region an arm is about to sweep, which studies have found helps bystanders anticipate motion. The main limitation is that people must learn what each color or pattern means, and different manufacturers use different conventions.

Sound alerts include beeps before motion, reversing alarms on mobile robots and spoken announcements. Audible danger signals should be clearly louder than background noise: ISO 7731 sets criteria for danger signals in workplaces, and a common guideline is that a warning should exceed the ambient A-weighted level by about 15 dB, while staying below levels that would themselves harm hearing. Broadband "white noise" reversing alarms are easier to localize than pure tones and less irritating to neighbors. Overuse is the main design risk: alarms that sound constantly are tuned out.

Predictable and legible motion

The way a robot moves tells observers what it intends. Anca Dragan, Kenton Lee and Siddhartha Srinivasa formalized this in 2013 by distinguishing two properties of motion. Predictable motion is what an observer expects when they already know the robot's goal: typically the most efficient path. Legible motion lets an observer infer the robot's goal quickly and confidently when they do not yet know it. The two can conflict. If a robot arm can reach for one of two cups close together, the straight path toward the left cup is predictable, but for most of its length it is ambiguous; an observer cannot tell which cup is the target until the very end. A path that first swings slightly away from the right cup is less efficient but makes the goal obvious early.

Dragan's model assumes the observer reasons as if the robot were efficient. If the robot has moved from start \( S \) along a partial trajectory to point \( Q \), the observer's belief that the goal is \( G \) is proportional to how efficient that partial path would be as part of a path to \( G \):

With path length as cost, the cost so far is the distance traveled and \( V_G \) is the straight-line distance to the goal. Legibility rewards trajectories that drive this probability toward the true goal early. User studies found that legible motions let people predict the goal sooner and more accurately, and that collaborators were more comfortable and coordinated better, although overly exaggerated motion can look strange and reduce trust.

# Goal inference along predictable (straight) and legible (curved) reaches
import numpy as np
S = np.array([0.0, 0.0]); GA = np.array([-0.1, 1.0]); GB = np.array([0.1, 1.0])
def belief(path, beta=20.0):
    out = []
    for k in range(1, len(path)):
        Q = path[k]; c = np.sum(np.linalg.norm(np.diff(path[:k+1], axis=0), axis=1))
        sc = [np.exp(-beta * (c + np.linalg.norm(G - Q))) / np.exp(-beta * np.linalg.norm(G - S)) for G in (GA, GB)]
        out.append(sc[0] / sum(sc))
    return np.array(out)
t = np.linspace(0, 1, 21)
straight = np.outer(t, GA)
legible = np.column_stack([GA[0] * t - 0.3 * np.sin(np.pi * t), t])
for name, p in (("straight", straight), ("legible", legible)):
    b = belief(p)
    first = t[1:][np.argmax(b > 0.8)]
    print(f"{name:8s} P(goal A) at 25%, 50%, 75% of motion: {b[4]:.2f} {b[9]:.2f} {b[14]:.2f};"
          f" passes 0.8 at {first:.0%}")

straight P(goal A) at 25%, 50%, 75% of motion: 0.53 0.60 0.75; passes 0.8 at 85%
legible  P(goal A) at 25%, 50%, 75% of motion: 0.77 0.91 0.95; passes 0.8 at 30%

Legibility and predictability also apply beyond arms. A mobile robot in a corridor that drifts to one side early communicates which way it will pass; one that drives straight at a person until the last moment forces them to guess. Social navigation research borrows from how pedestrians negotiate passing, and lesson 5 returns to social norms of space.

Designing communication

A practical design process for robot communication is to list the messages the robot must send (I am about to move, I need help, I have finished, I did not understand) and the messages it must receive (stop, go, take this, put it there), then choose channels for each message with redundancy for anything safety-related. Each signal should be tested with people who have not been briefed on its meaning, because designers always find their own signals obvious. Consistency across a fleet and across manufacturers helps, and standardization efforts for robot signals are under way, but at the time of writing there is no single universal convention for robot lights or sounds, so on-site training and clear signage remain necessary.

# Mapping robot messages to redundant signals (configuration listing)
signals:
  about_to_move:   {light: "amber pulse", sound: "two short tones", motion: "head turns to path"}
  moving:          {light: "green steady", projection: "path arrow on floor"}
  needs_help:      {light: "blue slow flash", screen: "message with reason", app: "push alert"}
  safety_stop:     {light: "red steady", sound: "single long tone", screen: "stop reason"}
  command_heard:   {screen: "echo of command", speech: "confirm critical slots"}