Deformable and articulated objects and learning-based manipulation

Most of this course has assumed rigid objects: a box stays a box, and a grasp that holds it at one moment holds it the next. Many of the objects people handle every day are not rigid. Cloth folds, cables bend, bags sag, dough stretches, fruit bruises. Others are rigid parts joined by constraints: doors swing on hinges, drawers slide on rails, laptop lids rotate. This lesson covers how robots manipulate deformable and articulated objects, and then previews the learning-based methods that have become the leading approach for tasks that are hard to model, which ROB-410 develops in depth.

The two halves are connected. Deformable objects have so many possible shapes that hand-written models and planners struggle, which is exactly where learning from data shines. Articulated objects sit in between: their constraints are simple once identified, so classical estimation works well, and learning helps mostly with recognizing the handle and the joint type.

Why deformable objects are hard

A rigid object's state is six numbers. A towel's state is the position of every point on its surface: effectively infinite, approximated by hundreds or thousands of mesh points. Its shape changes with every contact, it folds over itself so that parts hide other parts, and its response to a pull depends on material properties such as bending stiffness that vary from item to item. Grasping a towel does not fix its pose; it fixes one point, and the rest hangs according to gravity. Planning, perception and control all have to cope with this.

Engineers handle deformable objects in four broad ways. They reduce the state to a few task-relevant features, such as the positions of a towel's corners or a cable's endpoints and a handful of points along it. They use the environment, laying cloth flat on a table or routing a cable through fixtures, so that gravity and friction do much of the work. They use simple, repeatable motion primitives, such as flinging a garment to spread it or tracing a cable with sliding fingers. And they learn policies from demonstrations or experience instead of modelling the physics explicitly.

Cloth, cables and bags

Cloth manipulation research has focused on smoothing, folding and dressing. A common pipeline finds two corners, grasps one, lifts it so the cloth hangs, finds a second corner on the hanging cloth, grasps it, and lays the cloth flat; then folding becomes a sequence of pick-and-place actions on known points. Dynamic actions, such as a fling that throws air under the cloth, can flatten it in a few moves instead of many quasi-static pulls.

Cables and wires appear in assembly, where they must be routed through clips and connectors inserted at the end. Robots trace a cable by sliding a lightly closed gripper along it, use the cable's known length and stiffness to predict its shape, and rely on fixtures and clips to constrain it. Connector insertion combines everything from lesson 4 onward: a precise grasp near the connector, force control for insertion, and verification by force signature or vision.

Bags and sacks, from polybags in e-commerce to grain sacks in agriculture, are deformable containers whose contents shift. Suction on thin film works if the cup is large and compliant and the vacuum moderate, as lesson 7 noted; heavy sacks need forks or wide grippers that support them from below.

Food and soft products

Food handling adds hygiene and fragility to deformability. Grippers must be cleanable, made of food-safe materials and often washable; soft pneumatic fingers and needle grippers are common for meat and baked goods, and suction for packaged items. Variation is large: no two chicken breasts or loaves are the same shape. Successful systems combine vision to measure each item, forgiving grippers and conservative grip forces based on the item's measured size. Regulations shape design: in Canada the Safe Food for Canadians Regulations and in the United States the FDA Food Safety Modernization Act require hazard controls in food facilities, and equipment hygiene standards such as NSF/ANSI and 3-A guide materials and cleanability in both countries.

Articulated objects: doors, drawers and lids

An articulated object has parts connected by joints, most often a revolute joint (a hinge) or a prismatic joint (a slide). To open it, the robot must grasp the handle and move along the constraint. If the hinge axis or slide direction is known exactly, that is a simple trajectory. Usually it is not, and a stiff arm moving along a slightly wrong path builds up large forces. Two complementary approaches work well. Compliant control, with low stiffness in the directions where the constraint should guide motion, lets the robot follow the true path while pulling. Estimation fits a joint model to the observed handle motion during the first part of the opening and then plans the rest along the fitted path.

For a revolute joint seen from above, the handle moves on a circle centred on the hinge. Fitting a circle to the measured handle positions gives the hinge location and the radius. A linear least-squares fit, known as the Kåsa method, rewrites the circle equation so that the centre and radius come from one matrix solve. If the fitted radius is very large, the motion is nearly straight, which suggests a prismatic joint such as a drawer; model selection compares how well each model explains the data.

# Hinge estimation from noisy handle positions: linear least-squares (Kasa) circle fit
import numpy as np
rng = np.random.default_rng(11)
hinge, R = np.array([0.10, 0.85]), 0.78              # true hinge position and handle radius, m
ang = np.radians(np.linspace(0, 25, 12))              # first 25 degrees of opening
pts = hinge + R * np.stack([np.cos(-np.pi/2 + ang), np.sin(-np.pi/2 + ang)], axis=1)
pts += rng.normal(0, 0.002, pts.shape)                # 2 mm measurement noise
# x^2 + y^2 = 2 a x + 2 b y + c  ->  linear in (a, b, c); radius^2 = c + a^2 + b^2
A = np.column_stack([2 * pts[:, 0], 2 * pts[:, 1], np.ones(len(pts))])
y = (pts**2).sum(axis=1)
(a, b, c), *_ = np.linalg.lstsq(A, y, rcond=None)
r_fit = np.sqrt(c + a**2 + b**2)
err = np.linalg.norm(np.array([a, b]) - hinge)
print(f"fitted hinge ({a:.3f}, {b:.3f}) m, radius {r_fit:.3f} m; hinge error {1000*err:.1f} mm, radius error {1000*(r_fit-R):+.1f} mm")
# where would the fitted model put the handle at 90 degrees, compared with the truth?
true90 = hinge + R * np.array([np.cos(0.0), np.sin(0.0)])
fit90 = np.array([a, b]) + r_fit * np.array([1.0, 0.0])
print(f"predicted handle position at 90 deg off by {1000*np.linalg.norm(fit90 - true90):.1f} mm")

fitted hinge (0.119, 0.773) m, radius 0.702 m; hinge error 79.6 mm, radius error -78.2 mm
predicted handle position at 90 deg off by 97.1 mm

With only the first 25 degrees of swing observed and 2 mm of noise, the fit places the hinge about 8 cm from its true position and underestimates the radius by a similar amount, so the predicted handle position at 90 degrees is about 10 cm off. A short arc simply does not pin down its centre well: many circles pass close to a nearly straight set of points. The estimate improves quickly as more of the arc is seen. That is why, in practice, the robot refits continuously as it pulls and keeps compliance in the radial direction, so small model errors are absorbed by the controller rather than turning into large forces.

Learning-based manipulation

For tasks such as folding laundry, tidying a cluttered table or assembling flexible parts, writing explicit models and planners is slow and brittle. Learning-based methods instead train a policy, a function from observations (images, depth, joint states, sometimes tactile signals) to actions (joint or gripper motions), from data. Three families dominate at the time of writing.

Imitation learning trains a policy to reproduce human demonstrations. People teleoperate the robot with a leader arm, a virtual reality headset or a handheld data-collection gripper, and the system records observations and actions. Modern approaches such as action chunking and diffusion policies predict short sequences of future actions, which makes behaviour smoother and more robust than predicting one step at a time. A few dozen to a few hundred demonstrations can teach a single task well.

Reinforcement learning improves a policy by trial and error against a reward. It excels where simulation is accurate, such as rigid-body dexterity in lesson 9, and is harder to use directly on real robots because each trial takes time and risks damage. Hybrid methods start from demonstrations and refine with reinforcement learning.

Vision-language-action models are large networks, built on models pre-trained on internet-scale images and text, then trained on robot demonstrations from many robots and tasks. They take a camera image and an instruction such as "put the red cup in the sink" and output actions. They generalize to new objects and instructions better than single-task policies, and several research and commercial models of this kind have appeared since 2023. Their reliability on long, precise or safety-critical tasks remains well below that of engineered industrial systems, so they are usually deployed with guard rails: limited workspaces, force limits and human oversight.

Evaluating and deploying learned policies

Learned manipulation policies need careful evaluation. Success rates should be measured over many trials, on objects and conditions not seen in training, with a clear definition of success and with failures categorized so they can be fixed. Simulation benchmarks help compare methods but do not replace physical testing. In deployment, monitoring matters as much as in classical systems: log every attempt, detect out-of-distribution inputs, fall back to safe behaviours, and keep humans able to intervene. ROB-410 covers these methods, their data requirements and their safety evaluation in more depth.