Capstone: a safety case and validation plan for a campus shuttle or delivery drone

This capstone brings the course together. You will develop a safety case and validation plan for one of two systems: a low-speed automated shuttle carrying students and staff around a university campus, or a delivery drone carrying small parcels between campus buildings. Either system involves the full chain covered in the course: an operational design domain, perception, localization, planning and control or flight control, validation, regulation, human factors and ethics. The deliverable is not a working vehicle but the engineering argument and evidence plan that would justify deploying one. This lesson walks through a worked version for the shuttle and highlights where the drone version differs, then sets out the assignment and how it will be assessed.

Step 1: define the system and its ODD

Every safety case starts with a precise description of what is being argued about. For the shuttle: a battery-electric vehicle for up to 10 passengers, with a maximum speed of 25 km/h, running a fixed 3.2 km loop on campus roads with six stops, sharing the road with cars, cyclists and many pedestrians. The ODD specifies the route and its features, speed limits, times of operation, such as 7:00 to 22:00 on weekdays, and environmental limits: daylight and street-lit night, rain up to moderate intensity, temperatures from minus 25 to plus 35 degrees Celsius, and no operation in active snowfall or with snow-covered lane markings unless validated. It specifies a remote supervisor who monitors up to three shuttles and can assist, but not drive, and an on-board attendant during the first phase.

For the drone: a quadrotor under 25 kg carrying parcels up to 2 kg between six rooftop pads within a 2 km radius, cruising at 60 m above ground at 15 m/s, in winds below 10 m/s, visibility above 3 km, no icing conditions, and temperatures above minus 15 degrees Celsius, operating under the applicable BVLOS rules with a remote pilot supervising up to three aircraft.

Step 2: hazard analysis

Next, identify what could go wrong. Use structured methods: a hazard and operability study across operating modes, failure mode and effects analysis for components, and a SOTIF analysis of triggering conditions under which correct components might produce unsafe behavior. For each hazard, estimate severity, exposure and controllability, and assign an integrity target, as in ISO 26262, or a risk class. Record everything in a hazard log that will be updated throughout the project.

For the drone, top hazards include loss of control over people due to motor or battery failure, mid-air collision with a crewed aircraft such as a medical helicopter landing at a campus hospital, flyaway after GNSS or link loss, and dropping a payload. Mitigations include redundant propulsion or a parachute, routing over roofs and less-used areas, DAA sensing and coordination with the helipad, geofences and failsafes, and secure payload latches with pre-flight checks.

Step 3: safety requirements and architecture

Turn mitigations into requirements that can be tested. For example: the shuttle shall not exceed 15 km/h within 5 m of an occluding object adjacent to its path; it shall detect a pedestrian-sized object at 40 m with 99.9 % probability in the ODD; it shall reach a minimal risk condition within 10 s of a primary compute failure; it shall maintain lateral position within 0.3 m with a protection level reported at 10 Hz. Each requirement traces to a hazard and to the evidence that will show it is met. The architecture should support these requirements with redundancy where needed, an independent safety monitor and a minimal risk maneuver available at every point on the route.

# Check a shuttle speed policy: can it stop for a pedestrian appearing from an occlusion?
import numpy as np
rng = np.random.default_rng(42)
t_lat, a_brake = 0.4, 3.5                # latency (s) and comfortable-to-firm braking (m/s^2)
def stop_dist(v):
    return v * t_lat + v**2 / (2 * a_brake)
for v_kmh in (25, 20, 15, 10):
    v = v_kmh / 3.6
    # pedestrian emerges at a random distance 4 to 12 m ahead from behind a parked van
    d_emerge = rng.uniform(4, 12, 100_000)
    p_collide = np.mean(stop_dist(v) > d_emerge)
    print(f"{v_kmh:2d} km/h: stop {stop_dist(v):5.2f} m, fraction of emergences that cannot be avoided {p_collide:6.3f}")

25 km/h: stop  9.67 m, fraction of emergences that cannot be avoided  0.708
20 km/h: stop  6.63 m, fraction of emergences that cannot be avoided  0.330
15 km/h: stop  4.15 m, fraction of emergences that cannot be avoided  0.018
10 km/h: stop  2.21 m, fraction of emergences that cannot be avoided  0.000

Step 4: the validation plan

The validation plan explains how evidence for each requirement and for overall safety will be gathered. It combines the layers from Lesson 8. Simulation covers a scenario library built from the hazard analysis, campus video surveys and public scenario databases, with parameter sweeps and falsification to find failures; hardware-in-the-loop tests check timing and fault handling; a closed test area on campus, such as a parking lot after hours, confirms critical scenarios with soft pedestrian targets; and staged on-road testing proceeds from an attendant with hands near controls, to an attendant without controls, to supervised operation without an attendant. Each stage has entry and exit criteria based on metrics such as minimum time to collision distributions, interventions per 1,000 km, rule violations and passenger comfort, and a safety review board decides on progression.

# Track intervention data during staged testing with a 95 % upper bound
import numpy as np
rng = np.random.default_rng(5)
true_rate = 1 / 25_000                           # assumed true rate per km
km_per_day, days = 360, 120
km = np.cumsum(np.full(days, km_per_day))
events = np.cumsum(rng.poisson(true_rate * km_per_day, days))
for d in (30, 60, 90, 120):
    k, n = events[d - 1], km[d - 1]
    # 95 % one-sided upper bound on the Poisson mean (Wilson-Hilferty approximation)
    mu_up = (k + 1) * (1 - 1 / (9 * (k + 1)) + 1.645 / (3 * np.sqrt(k + 1)))**3
    upper = mu_up / n
    print(f"day {d:3d}: {n:6,.0f} km, {k} events, upper bound 1 per {1 / upper:7,.0f} km")

day  30: 10,800 km, 1 events, upper bound 1 per   2,284 km
day  60: 21,600 km, 1 events, upper bound 1 per   4,568 km
day  90: 32,400 km, 2 events, upper bound 1 per   5,155 km
day 120: 43,200 km, 3 events, upper bound 1 per   5,578 km

Step 5: regulatory pathway and stakeholders

In the United States, the shuttle team would review whether the vehicle meets FMVSS or needs an exemption under Part 555 or 591, set up crash reporting under the Standing General Order, and obtain state authorization where required, for example a California DMV permit. Campus roads may be private property, but public roads crossing campus bring state rules into play. The drone team would operate under Part 107 with a waiver for BVLOS, or under Part 108 if it is in force by then, and coordinate with any nearby airport or hospital helipad. Both teams should consult the campus safety office, security, accessibility services, student associations and local emergency services early.

Human factors and ethics belong in the plan too: how will passengers with disabilities board and get help; how will the shuttle communicate with pedestrians; what data will be recorded and how will privacy be protected; how will the remote supervisor's workload be kept manageable; and how will incidents be communicated openly to the campus community?

Step 6: operations, monitoring and updates

The safety case continues after launch. Define safety performance indicators and thresholds that trigger review, an incident response procedure, and a change management process: every software or map update passes regression testing in simulation and a short on-campus check before release, and the safety case is updated accordingly. Seasonal changes matter on Canadian and northern US campuses; the first winter is effectively a new ODD and should be approached as a new validation stage.

The assignment

Choose the shuttle or the drone, or propose another campus autonomous system of similar scope with instructor approval. Produce: a system and ODD description; a hazard log with at least 15 hazards, causes, severity and mitigations; a top-level safety argument in goal structuring notation with at least four sub-goals and their evidence; at least 10 testable safety requirements traced to hazards; a validation plan with simulation, closed-course and on-road stages, entry and exit criteria and a statistical justification of test quantities; a regulatory and stakeholder plan for your jurisdiction; and a one-page discussion of human factors, equity and privacy. Include at least one calculation from the course, such as stopping distance, protection level budget, hover endurance or the rule of three, with its assumptions.