Perception for manipulation

A manipulation system can only grasp what it can see, and it can only grasp well what it can see accurately. ROB-330 covered computer vision in general: camera models, calibration, depth sensing, detection and pose estimation. This lesson focuses on what manipulation needs from perception and how perception systems are built for grasping, especially in the hardest common setting, picking from a cluttered bin. The theme throughout is that perception for manipulation is judged by grasp success, not by image benchmarks.

Manipulation perception has a different emphasis from perception for navigation. Distances are short, often less than a meter, so millimeter accuracy matters. Objects touch and occlude each other. The robot's own arm and gripper enter the field of view. And the scene changes every time the robot picks something, so perception must run again and again within the cycle time budget.

What manipulation needs to know

Depending on the method, a grasp planner needs some or all of the following: which pixels or points belong to the target object (segmentation), the object's identity or category, its 6-D pose relative to a known model, local surface geometry such as normals and curvature at candidate contacts, free space around the object for the fingers, and physical properties such as whether the surface will seal a suction cup. Model-based pipelines need identity and pose; model-free pipelines work directly from geometry. Each requirement adds sensing, computation and failure modes, so the first design question is which of these the chosen grasping method truly needs.

Sensors and where to put them

Most manipulation cells use RGB-D sensing: a color image registered to a depth image. Structured-light and active-stereo cameras give dense depth at short range with sub-millimeter to few-millimeter precision, depending on distance and model. Time-of-flight cameras are compact but suffer from multipath reflections inside bins. Laser line scanners give very precise profiles of parts moving on conveyors. ROB-220 compares these sensors; here the question is placement.

A camera fixed above the work area, sometimes called eye-to-hand, sees the whole scene and can image while the arm is busy elsewhere, saving cycle time. A camera on the wrist, eye-in-hand, can move close for detail and look from several angles, but the arm must pause to take images. Many cells use both: an overhead camera for coarse localization and a wrist camera for final alignment before inserting or for reading labels. Either way, the camera-to-robot transform from hand-eye calibration (ROB-330) is part of the error budget, and it must be rechecked whenever a camera is bumped.

Segmentation in clutter

In a bin of mixed items, the first job is to separate the target from its neighbors. Classical methods cluster the point cloud using distance and surface-normal discontinuities, or find planes and boxes with RANSAC. These work well for regular shapes on uncluttered backgrounds. Learned instance segmentation networks, trained on large image datasets and fine-tuned on the actual products, produce a mask per object and cope far better with clutter and varied packaging. Foundation segmentation models, which can segment almost anything from a prompt, have made it much cheaper to label training data and to handle new product types, at the time of writing (October 2026).

Segmentation errors propagate directly into grasp failures. A mask that merges two touching boxes can lead the gripper to grasp across both, lifting one and dragging the other. A mask that splits one object can lead to a grasp on a thin edge. Engineers therefore evaluate segmentation by the grasp success it enables, and they often add simple geometric sanity checks, such as expected size ranges, before trusting a mask.

Model-based pose estimation

When the objects are known parts with CAD models, perception estimates each part's 6-D pose and grasps are taken from a database, as in lesson 5. Pose estimation typically finds an initial guess by matching features or by a learned network, then refines it with an iterative closest point alignment of the model to the observed points. Pose accuracy requirements come straight from the grasp: a parallel-jaw grasp on a 30 mm feature with 5 mm of clearance on each side tolerates a few millimeters of position error and a few degrees of rotation error, but an insertion task may need sub-millimeter accuracy, which usually means adding compliance or a final sensing step rather than relying on the pose alone.

Symmetric parts deserve special care. A cylinder looks the same after any rotation about its axis, so its pose about that axis is undetermined, and a pose estimator may report any value. That is harmless if the grasp is also symmetric, but if a feature such as a keyway matters for placement, the system needs another way to find it, such as a second camera view or rotating the part in the gripper while watching it.

Model-free grasp detection

For mixed-item picking, with thousands of products and constant new arrivals, model-based perception is impractical. Model-free methods go directly from sensor data to grasps. Classic approaches score candidate suction points by local flatness and surface normal, or parallel-jaw grasps by fitting the gripper geometry into the depth image. Learned methods, from early networks that rated grasps in depth images to families such as GraspNet, Contact-GraspNet and AnyGrasp at the time of writing, predict dense grasp proposals across the whole point cloud with success scores, often in tens of milliseconds on a GPU.

A simple model-free suction heuristic illustrates the idea. Suction needs a patch that is flat enough for the cup to seal, large enough for the cup diameter, roughly facing the approach direction, and preferably near the top of the pile and near the object's center of mass. Fitting a plane to the points within the cup radius around each candidate gives the normal and a flatness measure; candidates are then ranked.

# Suction point scoring on a synthetic depth patch: plane fit gives normal and flatness
import numpy as np
rng = np.random.default_rng(3)
# synthetic object top: a gently tilted plane with a 4 mm ridge on one side, plus sensor noise
xs, ys = np.meshgrid(np.linspace(-0.03, 0.05, 41), np.linspace(-0.03, 0.03, 31))
z = 0.002 * xs / 0.03 + np.where(xs > 0.015, 0.004, 0.0) + rng.normal(0, 0.0003, xs.shape)
pts = np.stack([xs.ravel(), ys.ravel(), z.ravel()], axis=1)
cup_r = 0.010
def score(cx, cy):
    m = (pts[:, 0] - cx)**2 + (pts[:, 1] - cy)**2 < cup_r**2
    P = pts[m]; c = P.mean(axis=0)
    _, _, vt = np.linalg.svd(P - c); n = vt[2] if vt[2][2] > 0 else -vt[2]
    resid = np.abs((P - c) @ n)
    tilt = np.degrees(np.arccos(n[2]))
    return resid.max() * 1000, tilt
for cx in (-0.015, 0.0, 0.015, 0.035):
    flat, tilt = score(cx, 0.0)
    ok = flat < 1.5 and tilt < 15
    print(f"cup at x={1000*cx:+5.1f} mm: worst deviation {flat:.2f} mm, tilt {tilt:4.1f} deg -> {'good seal' if ok else 'reject'}")

cup at x=-15.0 mm: worst deviation 0.74 mm, tilt  3.4 deg -> good seal
cup at x= +0.0 mm: worst deviation 0.86 mm, tilt  4.3 deg -> good seal
cup at x=+15.0 mm: worst deviation 1.87 mm, tilt 22.0 deg -> reject
cup at x=+35.0 mm: worst deviation 0.91 mm, tilt  4.1 deg -> good seal

The candidate that straddles the 4 mm ridge fails, because the cup lip would leave a gap over the step. The candidates on either flat area pass, despite the gentle tilt and sensor noise. Real systems use the same logic with the cup's measured compliance: a bellows cup tolerates more deviation than a flat cup, which is one reason bellows cups dominate mixed-item picking.

Difficult materials

Transparent and highly reflective objects defeat most depth sensors. Light passes through clear plastic or bounces off chrome, leaving holes or wildly wrong depth values exactly where the robot needs to grasp. Practical responses include polarized or diffuse lighting, multiple camera angles, learned depth completion that fills in plausible surfaces, and grasping strategies that tolerate uncertainty, such as compliant suction or approaching from a direction where the object's outline is visible in the color image. Dark, light-absorbing materials and very thin objects such as polybags and envelopes cause similar trouble.

Closing the loop

Perception does not stop when the grasp is chosen. During approach, a wrist camera can refine the target pose. After closing, the system checks that it holds what it intended to: vacuum level for suction, finger opening for jaws, a weight change on a wrist force-torque sensor, or a quick image of the gripper. After the pick, the bin is re-imaged because removing one item may have shifted others. This sense, act and verify loop is what turns a 90 % grasp planner into a 99 % system, because most failures are detected and retried quickly rather than causing a jam or a dropped item further down the line.