Foundations of human-robot interaction

Human-robot interaction, usually shortened to HRI, is the study and design of the ways people and robots perceive, communicate with, work alongside and affect one another. Earlier courses in this program treated the robot as a machine to be modeled, controlled and programmed. ROB-430 adds the other half of every real deployment: the people who command the robot, share space with it, repair it, are cared for by it, or simply walk past it in a corridor. A robot arm that meets every accuracy specification can still fail in practice if workers do not trust it, cannot tell what it is about to do, or are hurt by it. A social robot with an impressive dialogue system can still fail if the people it is meant to help find it confusing or patronizing. HRI is where engineering meets psychology, design, ethics and law, and this course draws on all of them.

This first lesson builds the vocabulary used for the rest of the course. We look at where the field came from, the ways HRI researchers classify interactions, the levels of automation that describe how work is divided between person and machine, and the roles people play around robots. Later lessons go deeper into human factors, communication, collaborative workspaces, social and assistive robots, teleoperation, trust, user studies, data analysis, accessibility and ethics, ending with a capstone design and evaluation project.

A short history

Industrial robots entered factories in 1961, when the first Unimate arm began unloading die-cast parts at a General Motors plant in New Jersey. For decades the dominant safety idea was separation: the robot worked inside a fenced cell and people stayed out while it moved. Interaction was mostly programming through a teach pendant and maintenance with the robot stopped. In parallel, researchers building remotely operated manipulators for nuclear, undersea and space work studied how a human operator could control a machine at a distance. Thomas Sheridan and colleagues at MIT developed the idea of supervisory control, in which the human sets goals and monitors while the machine carries out low-level control on its own. That work produced the levels of automation that are still taught today.

From the 1990s onward, robots began to leave the cage. Museum tour-guide robots, research platforms for social interaction such as Kismet at MIT, household vacuum robots, surgical systems and assistive devices all put robots near untrained members of the public. The annual ACM/IEEE International Conference on Human-Robot Interaction began in 2006 and gave the field its own venue. In industry, collaborative robot arms designed to work near people without a fence became commercially common in the 2010s, and safety standards were revised to describe how such applications can be made acceptably safe. Today HRI covers warehouses with mobile robots weaving among pickers, hospital delivery robots riding elevators, robots that help children with autism practice social skills, exoskeletons, telepresence systems and much more.

What counts as an interaction

Every HRI situation has at least three ingredients: one or more people, one or more robots, and a task or setting that brings them together. Researchers have proposed taxonomies to describe the space. Yanco and Drury's taxonomy, published in 2004, lists dimensions such as the robot's level of autonomy, the ratio of people to robots, the amount of intervention required, the time and space relationship between person and robot, the interaction roles people take, and the type of task. Two of these dimensions are especially useful early on: whether the interaction is remote or proximate, and what role each person plays.

In remote interaction, the person and robot are separated in space and often in time. A planetary rover operator on Earth, a drone pilot, or a surgeon at a console a few meters from the patient all interact remotely to some degree. Issues of latency, limited sensor views and situation awareness dominate. In proximate interaction, the person and robot share physical space: a worker beside a collaborative arm, a nurse walking past a delivery robot, an older adult talking with a companion robot. Physical safety, body language, personal space and social expectations dominate. Many real systems mix both, for example a warehouse supervisor who monitors a fleet remotely and occasionally walks onto the floor to clear a fault.

Roles matter because each brings different needs. A supervisor needs good overviews and alerts. An operator needs responsive controls and clear feedback. A mechanic needs safe ways to reach the robot and clear diagnostic information. A teammate needs to predict the robot's next move and coordinate timing. A bystander has had no training at all, so the robot's appearance, motion and signals must make sense to anyone. Designers who think only about the operator often forget the bystander, who is frequently the person most at risk. One person can also switch roles in a single shift, which is a source of confusion that designers must anticipate.

Levels of automation

A central question in HRI is how to divide work between human and machine. In 1978 Thomas Sheridan and William Verplank proposed a scale of ten levels describing how much a computer decides and acts on its own. At level 1 the computer offers no help and the human does everything. At level 10 the computer decides everything and acts autonomously, ignoring the human. The intermediate levels describe a computer that suggests alternatives, recommends one, executes it if the human approves, executes it unless the human vetoes within a time limit, or executes it and then informs the human.

In 2000, Raja Parasuraman, Thomas Sheridan and Christopher Wickens refined this idea by observing that automation can be applied separately to four stages of information processing: acquiring information, analyzing it, selecting a decision or action, and implementing the action. A system can be highly automated in one stage and barely automated in another. A modern collaborative welding cell, for example, might automatically detect the seam (high automation of acquisition), suggest weld parameters (moderate automation of analysis and decision) and then wait for the operator to press a button before welding (lower automation of action). Describing automation stage by stage avoids the trap of calling a whole system simply manual or autonomous.

Higher automation is not automatically better. Research on human performance with automation shows that as automation increases, the person's workload during routine operation falls, but so can their situation awareness and manual skill. When the automation fails or meets a situation it was not designed for, the human must suddenly re-enter the loop, often with poor awareness and degraded skill. Lisanne Bainbridge called this the irony of automation in a 1983 paper: the more reliable the automation, the less practice the human gets at the very tasks they are kept around to perform. Lessons 2 and 7 return to this problem when we discuss workload, situation awareness and over-reliance.

Autonomy, adjustable autonomy and shared control

Autonomy is often used loosely. In HRI it is helpful to treat autonomy as a property of a robot doing a particular task in a particular environment, not as a fixed property of the robot. A warehouse robot may be fully autonomous on a clear aisle and need human help when a pallet blocks the route. Adjustable or sliding autonomy refers to systems in which the level can change during operation, either by the human's choice or by the robot asking for help. Mixed-initiative systems let either party take the lead. Shared control blends human and robot commands continuously, for instance a wheelchair that follows the user's joystick but gently steers around doorframes. Lesson 6 covers shared control in detail.

A useful measure for systems in which one person supervises several robots is fan-out, proposed by Dan Olsen and Michael Goodrich. Neglect time is how long a robot can be left alone before its performance drops unacceptably. Interaction time is how long the operator needs to spend with it to bring it back to good performance. If a person can serve one robot during another's neglect time, the number of robots one person can handle is roughly

# Fan-out with variable neglect and interaction times (simulated shift)
import numpy as np
rng = np.random.default_rng(430)
nt = rng.gamma(shape=4.0, scale=90.0, size=500)   # neglect times, s (mean about 360 s)
it = rng.gamma(shape=3.0, scale=30.0, size=500)   # interaction times, s (mean about 90 s)
fo_mean = nt.mean() / it.mean() + 1
fo_each = nt / it + 1
print(f"mean NT = {nt.mean():.0f} s, mean IT = {it.mean():.0f} s")
print(f"fan-out from means: {fo_mean:.1f}")
print(f"median of per-episode fan-out: {np.median(fo_each):.1f}")
print(f"share of episodes with fan-out under 3: {np.mean(fo_each < 3):.0%}")

mean NT = 355 s, mean IT = 90 s
fan-out from means: 5.0
median of per-episode fan-out: 5.2
share of episodes with fan-out under 3: 18%

The simulation shows why averages can mislead. Even when the mean fan-out looks comfortable, a sizeable share of episodes combine a short neglect time with a long interaction, and several robots needing help at once create queues. Operators then fall behind, robots wait idle, and stress rises. Designers who staff fleets based on average numbers alone discover the problem during the first busy shift.

Why HRI is interdisciplinary

HRI draws on several disciplines. Robotics and control engineering supply the machine. Human factors and ergonomics supply models of perception, attention, workload and error. Cognitive and social psychology explain how people form mental models of machines, attribute intentions to them and respond to social cues. Industrial and interaction design shape the robot's appearance and interface. Statistics and experimental methods support user studies. Law, ethics and occupational health and safety set boundaries on what is acceptable. A good HRI team includes people who can speak to each of these, and a good HRI engineer can at least ask the right questions in each area.

A systems view of interaction

It helps to picture interaction as a loop. The robot senses the world and the person, decides, and acts. The person perceives the robot and the world, interprets what they see, decides and acts. Each side forms a model of the other. Breakdowns can happen anywhere in the loop: the robot misreads a gesture, the person misreads a light signal, the robot's plan conflicts with the person's expectation, or one side simply does not notice the other. Designing for interaction means designing every link in that loop and testing it with real people, not only testing the robot by itself.

This loop also explains why HRI has to be evaluated with people. A perception module can be tested on a dataset and a controller in simulation, but whether a worker understands a warning light, or whether an older adult feels comfortable with a companion robot, can only be found out by observing people. Lessons 8 and 9 teach how to run and analyze such studies so that their conclusions can be trusted.

# A minimal mode manager for adjustable autonomy (listing, ROS 2 style pseudocode)
MODES = ["manual", "assisted", "supervised_auto", "full_auto"]

def next_mode(current, operator_request, robot_confidence, person_nearby):
    if person_nearby and current == "full_auto":
        return "supervised_auto"          # drop a level when someone enters the area
    if robot_confidence < 0.4 and current in ("supervised_auto", "full_auto"):
        return "assisted"                 # ask for help when the robot is unsure
    if operator_request in MODES:
        return operator_request           # the human can always choose explicitly
    return current

The listing captures two design principles that recur throughout the course. First, the human can always explicitly choose the mode, so authority is clear. Second, the robot lowers its own autonomy when conditions leave its design envelope, rather than carrying on and hoping for the best. Real systems add hysteresis so that modes do not flicker back and forth, and they announce every mode change, because unannounced mode changes are a classic cause of mode confusion, in which the person believes the system is doing one thing when it is doing another.

Ethics from the start

HRI raises ethical questions in every lesson, not only in the ethics lesson. Who bears the risk when a robot shares space with workers? Is it acceptable for a companion robot to encourage an older adult to believe it cares about them? Whose data does a home robot's camera collect? Who is left out when a robot interface assumes the user can see a screen or speak clearly? Thinking about these questions early is cheaper than retrofitting answers late, and in many jurisdictions research ethics boards, privacy law and occupational safety rules require it. Lessons 8, 10 and 11 cover these frameworks in Canada and the United States.