Dexterous and in-hand manipulation

People rarely pick something up and leave it exactly as grasped. We roll a pen into writing position, spin a screw between thumb and finger, shift a phone from a pinch to a palm grip and flip a coin across our knuckles. This ability to reposition an object within the hand, without putting it down, is called in-hand manipulation, and it is the core of what roboticists mean by dexterity. Industrial robots mostly avoid it, using fixtures and regrasping instead, but dexterous hands are central to research on general-purpose robots and are slowly reaching applications such as assembly, laboratory automation and prosthetics.

This lesson explains what multi-finger hands can do that grippers cannot, the contact motions they use, the role of tactile sensing, how dexterous behaviors are controlled and learned, and why dexterity remains so hard. It is more of a survey than the previous lessons, because the field is moving quickly; specific systems are mentioned at the time of writing (October 2026).

Hands with many degrees of freedom

A dexterous hand has several fingers, each with two to four joints, often with an opposable thumb. The human hand has more than 20 degrees of freedom; research hands range from about 9 to 24. More joints let the fingers place contacts almost anywhere on an object and move them while maintaining the grasp. They also bring costs: many actuators, tendons or motors packed into a small volume, complex sensing, high price, fragility and control problems with many dimensions. Underactuated hands, with fewer motors than joints and mechanical linkages or springs coupling the joints, adapt passively to object shapes and are much cheaper and more robust, at the cost of less precise control over each finger.

A useful way to compare hands is by their manipulability around an object: how many independent ways can the object be moved by the fingers while the grasp is held? A parallel-jaw gripper has essentially one internal motion, opening and closing, and cannot reorient the object at all without the arm. A three-finger hand with three joints per finger, holding an object with three fingertips, has nine finger joints but the three contacts constrain them; the leftover freedom lets the fingers translate and rotate the object within a limited workspace.

Primitives of in-hand manipulation

In-hand manipulation is usually decomposed into a small set of contact motions, or primitives, that can be sequenced.

Finger gaiting moves the object by repeatedly relocating fingers, one at a time, while the remaining fingers keep a stable grasp, much as a walking animal moves one leg at a time. It needs enough fingers that the grasp stays stable with one removed, typically four or more for 3-D objects.

Rolling changes the contact point by rolling a fingertip over the object surface without slipping. Rolling a pen between thumb and index finger rotates it about its axis; the distance rolled equals the arc length along both surfaces. Rolling contacts are predictable and need no relocation, but their range is limited by finger travel.

Sliding deliberately lets a contact slip, for example letting a held object slide down between the fingers to grasp it lower. Controlled sliding uses the friction cone boundary from lesson 3: the grip force is reduced until the tangential load just exceeds \( \mu f_n \), then increased again to stop.

Pivoting lets an object rotate about a pair of fingertips under gravity or an external push, like a pendulum, then re-tightens the grip at the new angle. Industrial grippers sometimes pivot parts deliberately, using gravity and a light grip, to reorient them without extra fingers.

Non-prehensile and extrinsic dexterity use the environment instead of extra fingers: pushing an object against a wall to rotate it in the grasp, tapping it against a table to slide it in the fingers, or flipping it with a quick motion. Research on extrinsic dexterity shows that a simple parallel-jaw gripper plus clever use of the surroundings can achieve many reorientations that seem to need a dexterous hand.

# Pivoting under gravity: will a bar held between two fingertips rotate, and at what grip force does it stop?
import numpy as np
m, g = 0.20, 9.81          # bar mass kg
d = 0.06                   # distance from the grasp point to the bar's COM, m
mu, a = 0.8, 0.004         # fingertip friction and contact patch radius, m
k = 2/3                    # uniform-pressure torsional friction constant (lesson 3)
torque = m * g * d          # gravity moment about the finger axis when the bar is horizontal
for Fg in (2.0, 5.0, 10.0, 30.0):
    capacity = 2 * k * mu * a * Fg          # two soft-finger contacts
    print(f"grip {Fg:4.1f} N: torsional capacity {1000*capacity:5.1f} mN·m vs gravity {1000*torque:5.1f} mN·m -> {'holds' if capacity >= torque else 'pivots'}")
print(f"grip needed to stop pivoting: {torque / (2*k*mu*a):.1f} N")

grip  2.0 N: torsional capacity   8.5 mN·m vs gravity 117.7 mN·m -> pivots
grip  5.0 N: torsional capacity  21.3 mN·m vs gravity 117.7 mN·m -> pivots
grip 10.0 N: torsional capacity  42.7 mN·m vs gravity 117.7 mN·m -> pivots
grip 30.0 N: torsional capacity 128.0 mN·m vs gravity 117.7 mN·m -> holds
grip needed to stop pivoting: 27.6 N

The output shows why pivoting is both a failure mode and a tool. Holding the bar horizontal requires a grip of about 28 N on these small soft fingertips, far more than lifting the bar straight up would. A gripper squeezing 5 N will let the bar swing down until it hangs vertically under the fingers; a robot can exploit that deliberately to reorient long parts, then increase grip once the bar reaches the desired angle.

Tactile sensing and slip control

Vision struggles inside the hand, where fingers hide the object. Tactile sensing fills the gap. Fingertip sensors, covered in ROB-220, range from arrays of pressure-sensitive elements to camera-based sensors that image a soft gel surface from inside the finger, recovering contact shape, shear and incipient slip with high resolution. Tactile signals tell the controller where the contacts actually are, how hard each finger is pressing and whether the object is starting to slip.

A classic use is grip force control by slip detection. The hand holds the object lightly; when the tactile signal shows the edge of the contact patch beginning to slip, the controller raises normal force by a fixed ratio. People do the same unconsciously, holding a glass with only about 10 to 40 % more force than needed to prevent slip. Robots using this approach can hold eggs, fruit and soft packages that a fixed high grip force would damage.

Controlling and learning dexterity

Model-based control of dexterous manipulation plans contact sequences using models of the object, the hand and friction, then tracks them with force and impedance control. It works well for known objects and well-defined primitives but is brittle when contacts deviate from the model, which they constantly do.

Learning-based methods have made striking progress. Reinforcement learning in simulation, with heavy domain randomization of friction, masses and sensor noise, produced policies that reorient a cube in a physical five-finger hand, first shown around 2018–2019, and later work has extended this to many objects and to hands with tactile sensing. Imitation learning from human demonstrations, collected with teleoperation gloves or by tracking human hands in video, teaches skills that are hard to specify with rewards. At the time of writing, general-purpose dexterous manipulation remains a research frontier, and most deployed robots use grippers plus fixtures, as earlier lessons described. ROB-410 covers the learning methods in depth.

Why dexterity is still hard

Several factors make dexterous manipulation difficult. Contacts make and break, which makes the dynamics discontinuous and hard to optimize. Friction and object properties are uncertain. Fingers and objects occlude each other, so state estimation is poor. Hands with many actuators are expensive and fragile, and tactile sensors wear. Simulation of contact-rich behavior is imperfect, creating a gap between simulated and real performance. And useful tasks need not just one primitive but sequences, with recovery when one fails.

Engineers deciding whether a dexterous hand is justified usually ask whether the task variation truly demands it. If a small set of parts must be reoriented, a fixture or a second simple gripper is cheaper and more reliable. If the robot must handle open-ended variety, such as assisting people at home or working with tools designed for human hands, dexterity becomes valuable.