Foundation models for robotics

Large language and vision-language models trained on internet-scale data know a great deal about the world: what objects are called, what they are used for, how tasks are usually done, and how to break a goal into steps. Robotics has long lacked exactly this kind of broad common sense, and since about 2022 researchers have been finding ways to bring foundation models into robots. The approaches range from using a language model as a high-level planner that calls hand-built skills, to training vision-language-action models that output robot actions directly from images and instructions, to building generalist policies trained on data from many different robots. These efforts have produced robots that follow open-ended instructions, recognize unfamiliar objects and transfer skills across tasks, but they also raise hard questions about grounding, reliability, latency, data and safety. This lesson surveys language models as planners, grounding with affordances, vision-language models for perception and reasoning, vision-language-action models, cross-embodiment data and generalist policies, world models, humanoid robots, and the limitations that remain.

Language models as task planners

A language model can turn an instruction such as "I spilled my drink, can you help?" into a plausible plan: find a sponge, bring it to the spill, wipe, throw away the cup. Early systems used language models to choose among a library of pre-trained robot skills, each described in words. Other systems prompted language models to write code that calls robot perception and control functions ("code as policies"), so the model can compose loops, conditions and spatial reasoning in a program the robot executes. Language models can also translate instructions into formal task specifications for classical planners, combining their flexibility with the guarantees of symbolic planning.

The main problem is grounding: a language model does not know what the robot can actually do in its current situation. It might suggest picking up a sponge that is not there, or a skill the robot cannot perform reliably.

Grounding with affordances

One influential approach, SayCan, combined the language model's judgment of how useful each skill would be for the instruction with learned value functions estimating how likely each skill is to succeed from the current state (its affordance). The robot chooses the skill with the highest product of the two scores, executes it, and repeats with the history of completed steps. The toy example below shows the idea with simple stand-ins for both models.

import numpy as np
# Grounding language in what the robot can actually do (a SayCan-style toy)
skills = ["pick up the sponge", "go to the sink", "wipe the table", "open the drawer",
          "pick up the cup", "put the cup in the sink", "go to the table", "done"]
instruction = "the table is dirty, clean it with the sponge"

stop = {"the", "it", "is", "with", "to", "in", "up"}
def bag(text):
    words = [w for w in text.replace(",", " ").split() if w not in stop]
    vocab = sorted(set(" ".join(skills + [instruction]).replace(",", " ").split()))
    return np.array([words.count(w) for w in vocab], dtype=float)

history = []
def relevance(skill):
    # stand-in for a language model: similarity between instruction and skill words
    a, b = bag(instruction), bag(skill)
    sim = float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-9)) + 0.05
    return sim * (0.3 if skill in history else 1.0)   # the model sees what has already been done

def affordance(skill, state):
    # stand-in for learned value functions: probability the skill succeeds from this state
    table = {
        "pick up the sponge": 0.9 if state["at"] == "sink" and not state["holding"] else 0.05,
        "go to the sink": 0.95 if state["at"] != "sink" else 0.05,
        "go to the table": 0.95 if state["at"] != "table" else 0.05,
        "wipe the table": 0.9 if state["at"] == "table" and state["holding"] == "sponge" and not state["clean"] else 0.02,
        "open the drawer": 0.5, "pick up the cup": 0.1, "put the cup in the sink": 0.05,
        "done": 0.95 if state["clean"] else 0.01,
    }
    return table[skill]

state = {"at": "door", "holding": None, "clean": False}
for step in range(1, 8):
    scores = {s: relevance(s) * affordance(s, state) for s in skills}
    best_lang = max(skills, key=relevance)
    choice = max(scores, key=scores.get)
    history.append(choice)
    print(f"step {step}: language alone picks [{best_lang}]; language x affordance picks [{choice}]")
    if choice == "go to the sink": state["at"] = "sink"
    elif choice == "go to the table": state["at"] = "table"
    elif choice == "pick up the sponge": state["holding"] = "sponge"
    elif choice == "wipe the table": state["clean"] = True
    elif choice == "done": break

step 1: language alone picks [pick up the sponge]; language x affordance picks [go to the table]
step 2: language alone picks [pick up the sponge]; language x affordance picks [go to the sink]
step 3: language alone picks [pick up the sponge]; language x affordance picks [pick up the sponge]
step 4: language alone picks [wipe the table]; language x affordance picks [go to the table]
step 5: language alone picks [wipe the table]; language x affordance picks [wipe the table]
step 6: language alone picks [pick up the sponge]; language x affordance picks [done]

Language relevance alone keeps choosing "pick up the sponge" or "wipe the table" because those words match the instruction, even when the robot is at the door with nothing in its hand. Multiplying by affordance makes the robot choose feasible steps: it goes to the table, finds that wiping is not yet possible, fetches the sponge from the sink, returns, wipes and finishes. It even wastes one step going to the table first, as real systems sometimes do. Modern systems often use vision-language models that look at the scene directly, reducing such errors, but the principle of checking language-model suggestions against what is feasible remains essential.

Vision-language models in the robot

Vision-language models (VLMs) answer questions about images, describe scenes, and locate objects described in words. Robots use them to identify task-relevant objects ("which of these is the recycling bin?"), check whether a step succeeded ("is the drawer open?"), detect failures and ask for help, read labels and instructions, and reason about spatial relationships. Combined with open-vocabulary detection and segmentation (Lesson 4) and 3D scene graphs (Lesson 6), they let robots follow instructions about objects they were never explicitly trained on. Their limitations include imprecise spatial and metric reasoning, latency of hundreds of milliseconds to seconds, and occasional confident errors.

Vision-language-action models

Vision-language-action models (VLAs) go further: they take camera images and a language instruction and output robot actions directly. A typical VLA starts from a pre-trained vision-language model and is fine-tuned on large robot demonstration datasets, with actions represented as discrete tokens or generated by a diffusion or flow-matching action head. Well-known examples include RT-2 (2023), which showed that web knowledge could transfer to robot control, enabling instructions involving concepts never seen in robot data; OpenVLA (2024), an open-source VLA trained on cross-embodiment data; π0 (2024), which combined a VLM backbone with a flow-matching action expert for dexterous tasks such as folding laundry; and robotics models from major AI developers in 2025, including vision-language-action models for humanoid robots.

VLAs often use a dual-system design: a slower VLM "reasoning" component running at a few hertz interprets the scene and instruction, and a faster action component generates smooth motor commands at tens to hundreds of hertz, reflecting the loop hierarchy of Lesson 1. They are trained in stages: pre-training on large, diverse robot and web data, then fine-tuning on high-quality demonstrations for specific tasks or robots.

Cross-embodiment data and generalist policies

In language and vision, foundation models improved dramatically with more data. Robot data is scarce by comparison: each demonstration costs human time and robot hardware. To pool data, researchers combined datasets from many labs and robot types; the Open X-Embodiment dataset (2023) assembled demonstrations from over twenty robot embodiments, and models trained on it (such as the RT-X models and Octo) performed better on many robots than models trained on each robot's data alone. Generalist policies trained this way can be fine-tuned to new robots and tasks with relatively few demonstrations. Challenges include different action spaces, camera placements and control rates across robots, and inconsistent data quality. Scaling robot data, through fleets of robots collecting data in real deployments, low-cost collection devices and simulation, is now a major industry effort.

World models and video generation

Another line of work trains models to predict how the world will change in response to actions, learned from video and robot data. Such world models can be used to evaluate candidate plans by imagining their outcomes, to generate synthetic training data, or to pre-train representations that capture physics and object behaviour. Video generation models trained on internet video show some grasp of how objects move, and researchers are exploring their use as simulators for policy training and evaluation, though physical accuracy and controllability remain limited.

Humanoid robots

Humanoid robots, designed to work in spaces built for people and to learn from human demonstrations and video, have attracted large investment, with several companies developing general-purpose humanoids for factories, warehouses and eventually homes. Recent progress in learned locomotion (Lesson 9), whole-body teleoperation and VLAs has made humanoids far more capable than a few years ago. Significant challenges remain: dexterous hands that are robust and affordable, battery life, safety around people (humanoids can fall, and their size and power create hazards), cost, reliability over thousands of hours, and proving useful productivity compared with simpler specialized machines.

Limitations and open problems

• Reliability: success rates of 80 % or 90 % are impressive in research but inadequate for most deployments, where errors cause damage or require costly human intervention.
• Latency and compute: large models need powerful onboard or remote computers; network delays to cloud models are unacceptable for fast control.
• Precision and contact: language and vision knowledge does not directly provide the force control and precision needed for tight assembly.
• Evaluation: generalist claims need large, standardized, real-world evaluations that are hard to run and compare.
• Safety and security: models can misinterpret instructions, act on hallucinated perceptions, or be manipulated by malicious instructions or images (prompt injection), so robots need independent safety layers, permissions and confirmation for risky actions (Lesson 12).
• Data rights and privacy: data collected in homes and workplaces contains personal information and must be handled accordingly.