Deployment, evaluation and capstone: a learning pipeline for pick-and-place

The previous lessons built the tools: decision processes, reinforcement learning, simulation, imitation, representations and foundation models. This final lesson is about using them responsibly on a real robot. It covers how to measure whether a learned policy is good enough, with honest statistics; how to wrap learned policies in safety layers and build an argument that the system is acceptably safe; what Canadian and American law says about recorded demonstration data and AI governance at the time of writing (October 2026); and how to monitor a policy once deployed. It ends with a capstone that ties the course together: designing a complete learning pipeline for a pick-and-place cell.

Measuring success honestly

The basic metric for most robot tasks is the success rate: the fraction of trials in which the task is completed according to a precise, written criterion. Defining success precisely is the first discipline. 'Placed the part' must say where, within what tolerance, within what time, and whether a drop and re-grasp counts. Ambiguous criteria let evaluators, consciously or not, favour the method they hope will win. Where possible, success should be judged automatically or by someone who does not know which policy ran.

A success rate from a finite number of trials is an estimate with uncertainty. The naive interval \( \hat{p} \pm 1.96 \sqrt{\hat{p}(1-\hat{p})/n} \) behaves badly for the small trial counts and high success rates typical of robotics: at 20 successes in 20 trials it gives an interval of zero width. The Wilson score interval is a better default.

Here \( \hat{p} = k/n \) is the observed success rate from \( k \) successes in \( n \) trials and \( z = 1.96 \) for 95 % confidence.

# Wilson score intervals for robot success rates
import numpy as np
def wilson(k, n, z=1.96):
    p = k / n
    denom = 1 + z**2 / n
    mid = (p + z**2 / (2 * n)) / denom
    half = z / denom * np.sqrt(p * (1 - p) / n + z**2 / (4 * n**2))
    return mid - half, mid + half
for k, n in [(18, 20), (20, 20), (45, 50), (90, 100), (196, 200)]:
    lo, hi = wilson(k, n)
    print(f"{k:3d}/{n:3d} = {k/n:6.1%}   95% interval {lo:6.1%} to {hi:6.1%}")
# Do two policies differ? Compare intervals (a rough check) for 41/50 vs 47/50
for k in (41, 47):
    lo, hi = wilson(k, 50)
    print(f"policy with {k}/50: {lo:.1%} to {hi:.1%}")

 18/ 20 =  90.0%   95% interval  69.9% to  97.2%
 20/ 20 = 100.0%   95% interval  83.9% to 100.0%
 45/ 50 =  90.0%   95% interval  78.6% to  95.7%
 90/100 =  90.0%   95% interval  82.6% to  94.5%
196/200 =  98.0%   95% interval  95.0% to  99.2%
policy with 41/50: 69.2% to 90.2%
policy with 47/50: 83.8% to 97.9%

Two lessons stand out. Twenty trials cannot distinguish 90 % from 75 %, and even 20 successes out of 20 is consistent with a true rate as low as about 84 %. And overlapping intervals are a warning sign that a comparison may be inconclusive; to decide, use a proper two-proportion test or, for paired comparisons on the same initial conditions, a paired test.

Success rate is not the only metric. Report cycle time, the number and kinds of failures, human interventions per hour, damage incidents, near misses, and performance broken down by condition: object type, lighting, clutter and position. A policy that averages 95 % but fails on every transparent object has a specific, fixable problem that the average hides. Evaluation should include conditions outside the training data and should be repeated after every change to the policy, data or hardware.

Layers of safety around a learned policy

A neural policy cannot be exhaustively verified, so safety is built in layers around it, as lesson 6 introduced. At the bottom, hardware and functional safety measures, such as emergency stops, safety-rated monitored speed and separation and force limits for collaborative robots, are designed to standards such as ISO 10218 and ISO/TS 15066 and must not depend on the learned policy at all. Above that, a classical low-level controller enforces joint, velocity and torque limits. A safety filter, such as a control barrier function or a workspace monitor, checks each commanded action and replaces unsafe ones. Runtime monitors watch for out-of-distribution inputs, such as unusual images or unfamiliar states, and slow or stop the robot or ask a person for help. Only then does the learned policy propose actions.

A safety case is a structured argument, supported by evidence, that a system is acceptably safe for a specific use in a specific environment. For a learned system it typically argues that hazards were identified through risk assessment; that each hazard is mitigated by layers that do not rely on the learned policy behaving well; that the policy's operating domain is defined, such as object types, lighting and speeds; that monitors detect departures from that domain; and that testing provides statistically meaningful evidence of performance within it. Standards for machine-learning components in safety-related systems are still developing, at the time of writing, so a conservative architecture, in which the learned policy affects productivity but not safety, is the norm in industry.

Data and AI governance in Canada and the United States

Learned robots raise legal questions that hand-coded robots rarely did. Demonstration recordings capture video and audio of operators, coworkers and sometimes customers or homes, and policies may be trained on that data indefinitely.

In Canada, the Personal Information Protection and Electronic Documents Act, PIPEDA, governs personal information collected in commercial activity in most provinces; it requires meaningful consent, limits on collection and use to identified purposes, safeguards and access rights. Quebec's Law 25, phased in from 2022 to 2024, modernized that province's private-sector privacy law with stronger consent rules, privacy impact assessments for certain projects, and transparency about automated decisions. Alberta and British Columbia have their own private-sector privacy acts. Biometric information such as face images is treated as sensitive, generally requiring express consent.

In the United States there is no comprehensive federal privacy law at the time of writing. Instead, a growing number of states have consumer privacy laws, led by California's Consumer Privacy Act as amended by the California Privacy Rights Act, which give rights to know, delete and limit the use of sensitive data. Illinois's Biometric Information Privacy Act, BIPA, requires written notice and consent before collecting biometric identifiers such as face geometry and allows private lawsuits, which has produced substantial litigation; Texas and Washington have biometric laws too. Sector rules, such as health privacy rules, may also apply.

AI-specific governance also differs. Canada's proposed Artificial Intelligence and Data Act, AIDA, part of Bill C-27, died on the order paper when Parliament was prorogued in January 2025, so Canada has no federal AI statute at the time of writing; the government's Voluntary Code of Conduct on the Responsible Development and Management of Advanced Generative AI Systems, launched in September 2023, remains a non-binding reference. In the United States, Executive Order 14110 on safe, secure and trustworthy AI was revoked in January 2025, and federal policy shifted towards reducing barriers to AI development; in December 2025 Executive Order 14365, Ensuring a National Policy Framework for Artificial Intelligence, sought a single national approach and challenged conflicting state AI laws. Some states, such as Colorado, have passed AI laws of their own. Teams selling into Europe must also consider the EU AI Act, adopted in 2024 with obligations phasing in over several years, which treats AI that serves as a safety component of machinery as high risk.

Practical habits satisfy most regimes. Collect only the data the task needs, for example by pointing cameras at the workspace rather than at operators; tell people what is recorded and why, and obtain consent; blur or crop faces where they are not needed; set retention periods; secure and access-control datasets; record the provenance and licence terms of any external data or pretrained model; and document the policy's intended use and limits.

Deployment and monitoring

Deployment should proceed in stages: shadow mode, in which the policy proposes actions that are logged but not executed; supervised operation at reduced speed with an operator ready to intervene; limited autonomous operation on a subset of items or shifts; and full operation with monitoring. At each stage, define in advance the metrics and thresholds that permit moving to the next. After deployment, monitor success rates, cycle times, interventions and out-of-distribution alarms continuously, because performance drifts as products, lighting, wear and seasons change. Logged failures and interventions are the most valuable new training data, closing the loop with the DAgger idea from lesson 8. Every policy update should be versioned, evaluated against the previous version on a fixed test suite, and deployed with a rollback plan.

Capstone: a learning pipeline for pick-and-place

Consider a cell where a robot arm picks mixed consumer products from a tote and places them into shipping boxes. The product catalogue changes weekly, so hand-coded grasping for each item is impractical. Pull the course together by designing the pipeline.

# Capstone evaluation harness (listing only): log every trial for statistics and retraining
import json, time
def run_trial(policy, cell, item_id):
    obs = cell.observe()
    ood = monitor.score(obs) > OOD_THRESHOLD
    if ood:
        cell.divert(item_id)                 # unfamiliar item goes to the manual station
        return {"item": item_id, "result": "diverted"}
    grasp = policy.select_grasp(obs)
    ok = cell.execute_pick_and_place(grasp)  # motion planning and limits inside
    return {"item": item_id, "result": "success" if ok else "failure",
            "cycle_s": cell.last_cycle_time(), "t": time.time()}
with open("trials.jsonl", "a") as log:
    for item_id in test_suite:
        log.write(json.dumps(run_trial(policy, cell, item_id)) + "\n")