Foundation models and vision-language-action policies

In language and vision, a single large model pretrained on vast data and then adapted to many tasks, a foundation model, has replaced most narrow task-specific models. Robotics is attempting the same shift. Since 2022 a series of systems has trained large policies on demonstrations from many robots and tasks, often starting from vision-language models pretrained on internet data, so that one network can follow natural-language instructions such as 'put the apple in the drawer' across many scenes. These are called vision-language-action models, or VLAs. This lesson explains how they are built, surveys the main systems at the time of writing (October 2026), and, just as importantly, teaches how to evaluate their claims critically. The field moves very quickly; treat specific model names and numbers as a snapshot and the principles as the lasting content.

From task policies to generalist policies

The policies of lessons 8 to 10 are specialists: one task, one robot, one lab, typically tens to hundreds of demonstrations. A generalist policy aims to handle many tasks, objects and scenes, conditioned on an instruction, and ideally more than one robot body. Three ideas make this plausible. Scale: pooling data from many tasks lets common skills, such as reaching and grasping, be learned once. Transfer from the internet: vision-language models already know what an apple or a drawer looks like and what 'put in' means. Shared action interfaces: if actions from different robots can be expressed in a common form, data from one robot can help another.

Anatomy of a vision-language-action model

Most VLAs share a common structure. A vision encoder, often a pretrained model such as SigLIP or DINOv2, turns camera images into a sequence of visual tokens. A language model backbone, pretrained on internet text and image-text pairs, processes the visual tokens together with the instruction's text tokens. An action output component turns the backbone's final representations into robot actions. The output component is where designs differ most.

The first approach, used by RT-2 and OpenVLA, treats actions as language. Each continuous action dimension is discretized into a fixed number of bins, often 256, and each bin is mapped to a token in the language model's vocabulary. The model then predicts actions exactly as it predicts words, with the same next-token training objective. This reuses the language model unchanged but limits precision to the bin width and produces one action at a time slowly.

# Action tokenization: map continuous actions to 256 bins and back
import numpy as np
rng = np.random.default_rng(11)
low, high, bins = -1.0, 1.0, 256
edges = np.linspace(low, high, bins + 1)
centers = 0.5 * (edges[:-1] + edges[1:])
def tokenize(a):
    return np.clip(np.digitize(a, edges) - 1, 0, bins - 1)
def detokenize(tok):
    return centers[tok]
actions = rng.uniform(-1, 1, size=(1000, 7))         # 7-D arm actions, normalized
tokens = tokenize(actions)
recon = detokenize(tokens)
err = np.abs(recon - actions)
print("example action :", np.round(actions[0], 3))
print("tokens         :", tokens[0])
print("reconstructed  :", np.round(recon[0], 3))
print(f"bin width {(high - low) / bins:.4f}, max error {err.max():.4f}, mean error {err.mean():.4f}")

example action : [-0.743 -0.001  0.203 -0.943 -0.704  0.856 -0.859]
tokens         : [ 32 127 153   7  37 237  18]
reconstructed  : [-0.746 -0.004  0.199 -0.941 -0.707  0.855 -0.855]
bin width 0.0078, max error 0.0039, mean error 0.0020

The second approach attaches a continuous action head. Octo uses a small diffusion head; the π0 family adds a separate action expert network trained with flow matching, a close relative of the diffusion objective in lesson 9, which outputs chunks of continuous actions at high rates suitable for dexterous tasks such as folding laundry. Continuous heads give finer precision and smoother motion, and they combine the semantic knowledge of the backbone with the multimodal action modelling of generative policies.

Data: Open X-Embodiment and cross-embodiment learning

Foundation models need foundation-scale data, and robot data is scarce compared with internet text. The Open X-Embodiment project, announced in 2023, pooled existing datasets from many labs into a common format, reporting over a million episodes from 22 robot embodiments. Models trained on the mixture, called RT-X, performed better on several robots than models trained only on each robot's own data, which was early evidence that cross-embodiment pooling helps. Other large collections followed, such as DROID, gathered with a common platform across many institutions and scenes, and companies building generalist policies now collect large proprietary datasets, often thousands of hours of teleoperation.

Cross-embodiment learning is hard because robots differ in kinematics, grippers, camera placement, control rates and action spaces. Common strategies include normalizing actions per dataset, expressing actions in the end-effector frame, padding to a maximum action dimension with masks, using separate output heads per embodiment, and weighting datasets to prevent the largest from dominating. Data quality varies widely across pooled datasets, and careful curation often matters as much as volume.

Adapting foundation models

Even the strongest generalist policies, at the time of writing, rarely work out of the box on a new robot or task at the reliability a deployment needs. The usual workflow is to fine-tune: collect tens to hundreds of demonstrations on the target robot and task, and continue training the pretrained model on them, sometimes alongside some of the pretraining mixture to avoid forgetting. Parameter-efficient methods such as low-rank adaptation, LoRA, train small added matrices rather than all weights, which cuts memory and lets one base model support many adapted versions. The promise of foundation models in practice is often that they need fewer task demonstrations and generalize better to new objects and scenes than a policy trained from scratch, rather than that they need no task data at all.

# Fine-tuning an open VLA with LoRA (listing only; library APIs change often)
from transformers import AutoModelForVision2Seq, AutoProcessor
from peft import LoraConfig, get_peft_model
model = AutoModelForVision2Seq.from_pretrained("openvla/openvla-7b", torch_dtype="bfloat16")
processor = AutoProcessor.from_pretrained("openvla/openvla-7b")
lora = LoraConfig(r=32, lora_alpha=16, target_modules="all-linear", lora_dropout=0.0)
model = get_peft_model(model, lora)
model.print_trainable_parameters()       # a small fraction of the 7B weights
for batch in loader:                      # images, instruction text, tokenized actions
    out = model(**batch)                 # next-token loss over action tokens
    out.loss.backward(); optimizer.step(); optimizer.zero_grad()

Evaluating claims critically

Robot foundation models are announced with impressive videos and headline success rates. A careful engineer reads them the way a scientist reads any experimental claim. Several questions help.

First, how many trials, and what are the confidence intervals? A success rate of 90 % from 20 trials has a 95 % interval running from about 70 % to 97 %, so differences of a few points between models are often meaningless; lesson 12 shows how to compute this. Second, what was held out? Performance on objects, scenes and instructions seen in training says little about generalization; look for explicit unseen-object, unseen-scene and unseen-instruction splits. Third, who ran the evaluation, and were baselines given equal effort in tuning and data? Fourth, are the videos representative, or selected best runs? Fifth, what are the failure modes, and how does the policy behave when it fails: does it stop, retry or do something dangerous? Sixth, what are the compute and latency requirements, and can the model run on the target robot? Finally, can the result be reproduced: are weights, code and evaluation protocols available?

Standardized evaluation is an active research topic. Simulated benchmarks such as LIBERO and SIMPLER give reproducible comparisons but do not fully predict real performance. Real-world evaluation across labs, with shared protocols and blind comparisons, is more convincing and much more expensive. At the time of writing, there is no agreed benchmark that settles which generalist policy is best, and claims of general-purpose capability should be read as claims about specific tested conditions.

Limits and open problems

Despite rapid progress, robot foundation models at the time of writing face real limits. Data is still orders of magnitude smaller than for language models, and collecting it is slow and expensive. Precision and reliability lag behind what industrial deployment requires: a success rate of 90 % is impressive in research but unacceptable for a production line that needs 99.9 %. Long-horizon tasks, tasks needing memory and contact-rich assembly remain difficult. Large models are slow and power-hungry for on-robot use. Language understanding can create a false impression of general competence. And safety assurance for a large learned policy, whose behaviour cannot be exhaustively specified or verified, is unsolved; lesson 12 discusses practical layers of protection. Treat foundation models as a powerful new tool for building robot behaviour, especially for perception and flexible manipulation, combined with classical control and safety engineering rather than replacing them.