Accuracy and calibration

When the airborne contractor delivered the corridor data, the cover letter said it met the specification. Leah's job was to find out. Accuracy assessment answers three questions: how well do the points agree with independent ground truth (absolute accuracy), how well do overlapping strips and scans agree with each other (relative accuracy), and are the systematic errors that calibration should remove actually gone? This lesson covers the standards used to answer them, the statistics, the checkpoint survey, and the calibration of airborne, mobile and terrestrial systems.

Sources of error

Every lidar point inherits errors from the whole measurement chain. For airborne and mobile systems the main sources are GNSS positioning (a few centimetres in good conditions, more with long baselines or poor geometry), IMU attitude errors (multiplied by range), boresight and lever-arm calibration errors, ranging errors (timing, refraction, the range offset and the scanner's own calibration), scan angle encoder errors, and timing synchronization. To these the terrain adds its own: the footprint on slopes, penetration into vegetation, and surface properties. For terrestrial scanning the main sources are the scanner's calibration, range noise and surface effects, and registration and control (lesson 5). Some errors are random and average out over many points; others are systematic and shift whole strips or scans, which averaging does not remove.

Error budgets combine these sources. For the corridor flight at 1,100 m, a 3 cm GNSS error, a 0.003° roll and pitch error (about 6 cm at the swath edge), a 2 cm ranging error and residual boresight errors combine to a predicted vertical error of 5 to 8 cm on flat ground at the swath centre, more at the edges and on slopes. The flying contractor's software computes such predictions for each point, the total propagated uncertainty, which some specifications now ask to be delivered with the data.

Accuracy standards

The ASPRS Positional Accuracy Standards for Digital Geospatial Data, whose second edition was published in 2023 with later minor revisions, are the reference standard for lidar and other mapping products in the United States and are widely used in Canada. The second edition made several changes that matter for lidar. It reports accuracy as root mean square error (RMSE) only, dropping the 95 percent confidence values that earlier standards derived by multiplying RMSE by 1.96 for vertical and 1.7308 for horizontal accuracy, which assumed errors were normal and unbiased. It requires at least 30 checkpoints for a product, up to 120 for large projects. It includes the checkpoints' own survey error in the reported product accuracy, so that the stated RMSE describes the product, not just its difference from imperfect checkpoints. And it treats vegetated vertical accuracy as a value to be reported rather than a pass-or-fail test, recognizing that vegetation errors are not normally distributed.

The standard distinguishes non-vegetated vertical accuracy (NVA), tested on open, hard or bare ground where lidar should perform best, from vegetated vertical accuracy (VVA), tested in grass, crops, brush and forest. NVA is reported as RMSE in the vertical (RMSEv); VVA is commonly summarized as RMSE and as the 95th percentile of absolute errors. Horizontal accuracy is reported as RMSE in each axis and combined as RMSEH, the square root of the sum of the squares of RMSEx and RMSEy; testing it requires well-defined features visible in the point cloud, such as painted chevrons or building corners, which are harder to find in lidar than in imagery.

The federal airborne guideline used for the district's survey sets its own accuracy requirements in its quality levels, written in the older style with RMSE and 95 percent values: for the baseline level, 10 cm RMSE (19.6 cm at 95 percent) on non-vegetated ground, 30 cm at the 95th percentile in vegetation, and limits on the agreement within and between swaths. Leah reports both ways: RMSE with checkpoint error included, as the ASPRS edition asks, and the guideline's values, as the contract requires.

The checkpoint survey

Checkpoints must be independent: surveyed separately from any control used by the contractor and never given to them. They must be more accurate than the product they test; the second edition asks that checkpoints be at least twice as accurate as the product's required accuracy, so for a 5 cm product the checkpoints need about 2.5 cm or better, achievable by careful RTK with repeated occupations or by total station from control. They must be on surfaces suited to the test: NVA points on flat or uniformly sloping open ground, away from breaklines, at least several metres from any change in slope; VVA points in each vegetation class in proportion to its extent. And they must be spread through the project, not clustered near roads where the crew can reach them easily, which would hide errors in strips the crew did not visit.

Grant's crew surveys 30 NVA checkpoints and 10 VVA checkpoints through the corridor by RTK with two occupations separated by at least an hour, and 30 horizontal checkpoints on painted chevrons on the road. Each NVA checkpoint is compared with the elevation of a surface interpolated from the ground points, usually a triangulated surface, at the checkpoint's position.

import math, random
random.seed(10)
# 30 non-vegetated vertical checkpoints (RTK, sigma 1.2 cm) and 10 vegetated ones; lidar minus survey, m
nva = [random.gauss(0.012, 0.045) for _ in range(30)]
vva = [random.gauss(0.04, 0.10) for _ in range(10)]
# 30 horizontal checkpoints on painted chevrons: lidar minus survey dx, dy
hz = [(random.gauss(0.02, 0.10), random.gauss(-0.01, 0.09)) for _ in range(30)]
rms = lambda v: math.sqrt(sum(x * x for x in v) / len(v))
mean = lambda v: sum(v) / len(v)
def sd(v):
    m = mean(v); return math.sqrt(sum((x - m) ** 2 for x in v) / (len(v) - 1))
cp_v, cp_h = 0.012, 0.010            # checkpoint survey accuracy (RMSE)
rv = rms(nva); rv_tot = math.sqrt(rv ** 2 + cp_v ** 2)
print(f"NVA: mean {mean(nva)*100:.1f} cm, sd {sd(nva)*100:.1f} cm, RMSEv {rv*100:.1f} cm, with checkpoint error {rv_tot*100:.1f} cm")
v95 = sorted(abs(x) for x in vva)[math.ceil(0.95 * len(vva)) - 1]
print(f"VVA: RMSEv {rms(vva)*100:.1f} cm, 95th percentile {v95*100:.1f} cm (reported, not pass/fail)")
rx, ry = rms([d[0] for d in hz]), rms([d[1] for d in hz]); rh = math.hypot(rx, ry)
print(f"horizontal: RMSEx {rx*100:.1f} cm, RMSEy {ry*100:.1f} cm, RMSEH {rh*100:.1f} cm, with checkpoint error {math.hypot(rh, cp_h)*100:.1f} cm")
# Is the mean error significant? (t test, 29 degrees of freedom, 5 %)
t = mean(nva) / (sd(nva) / math.sqrt(len(nva)))
print(f"bias test t = {t:.2f} against 2.045: {'significant' if abs(t) > 2.045 else 'not significant'}")

# Relative accuracy between overlapping swaths: differences of gridded surfaces on flat open areas
d = [random.gauss(0.03, 0.025) for _ in range(500)] + [random.gauss(-0.05, 0.02) for _ in range(120)]
print(f"interswath: RMSDz {rms(d)*100:.1f} cm, maximum |diff| {max(abs(x) for x in d)*100:.1f} cm")
# A boresight roll error shows as a tilt between opposing swaths: dz grows with distance from nadir
off = [-350, -200, -50, 50, 200, 350]
dz = [0.002 + 2 * x * math.tan(math.radians(0.006)) for x in off]            # opposing flight directions double it
b = sum(x * z for x, z in zip(off, dz)) / sum(x * x for x in off)
print("swath difference across track (cm):", " ".join(f"{z*100:5.1f}" for z in dz))
print(f"fitted tilt {b*1000:.3f} mm/m -> boresight roll correction {math.degrees(math.atan(b))/2:.4f} deg")

NVA: mean 1.0 cm, sd 4.5 cm, RMSEv 4.5 cm, with checkpoint error 4.7 cm
VVA: RMSEv 10.5 cm, 95th percentile 19.9 cm (reported, not pass/fail)
horizontal: RMSEx 8.9 cm, RMSEy 6.8 cm, RMSEH 11.2 cm, with checkpoint error 11.3 cm
bias test t = 1.26 against 2.045: not significant
interswath: RMSDz 4.3 cm, maximum |diff| 11.6 cm
swath difference across track (cm):  -7.1  -4.0  -0.8   1.2   4.4   7.5
fitted tilt 0.209 mm/m -> boresight roll correction 0.0060 deg

The 30 NVA checkpoints show a mean difference of 1.0 cm and an RMSE of 4.5 cm; including the checkpoints' own 1.2 cm accuracy, the product's RMSEv is 4.7 cm, inside the 5 cm project requirement and well inside the 10 cm of the base specification. The bias test gives t = 1.26 against a critical value of 2.045 for 29 degrees of freedom, so the 1 cm mean is not statistically significant. The VVA RMSE is 10.5 cm and its 95th percentile 19.9 cm, reported as the standard asks. Horizontally, RMSEH is 11.2 cm, or 11.3 cm with checkpoint error, which reflects the difficulty of locating chevrons in a point cloud with 21 cm spacing. Leah's report states each number, the method, the checkpoint locations and the surfaces used.

Relative accuracy and swath checks

Absolute accuracy at checkpoints tests the data at a few dozen places. Relative accuracy tests it everywhere that strips overlap. Within a swath, the smooth precision is the scatter of points on flat hard surfaces about a fitted plane; between swaths, the interswath difference is computed by gridding each swath separately on flat, open areas and differencing the grids. Specifications set limits such as an RMSD (root mean square difference) of 6 to 8 cm and a maximum difference of 16 cm on open ground. The code's 620 sample differences give an interswath RMSD of 4.3 cm and a maximum of 11.6 cm, but the differences come in two groups, one of about +3 cm and one of about −5 cm, which suggests a systematic problem in one strip rather than random noise.

Swath separation images, raster maps of the difference between overlapping swaths coloured by size, show such problems at a glance. A uniform offset suggests a GNSS or ranging bias in one strip; a tilt across the overlap that reverses on opposite flight directions suggests a boresight roll error; differences that change along the line suggest trajectory errors. The USGS specification now asks for swath separation images in its deliverables and metadata.

Calibration of airborne and mobile systems

Boresight calibration estimates the small angles between the scanner and the IMU. In the classical method, the contractor flies a calibration site with flat ground, pitched roofs and well-defined edges in opposite directions and at different heights, and adjusts roll, pitch and heading until the strips agree: roll from the tilt between opposing strips on flat ground, pitch from the along-track shift of roof edges and other sharp features on opposite passes, and heading from features near the swath edges. Modern software does this automatically by matching planar patches or tie features across all overlapping strips in a least-squares strip adjustment, which can also estimate range and scan-angle scale errors and correct trajectory errors along each line. The code reproduces the first step: differences between opposing swaths that grow linearly across the overlap, from −7.1 cm to +7.5 cm over 700 m, give a fitted tilt of 0.209 mm per metre and a roll correction of 0.006°.

Calibration must be repeated whenever the sensor is reinstalled in an aircraft or vehicle, and checked on every project, because mounts shift, temperatures change and sensors age. Contractors document their calibration with before and after swath statistics, which belong in the project report.

Calibration of terrestrial scanners

Terrestrial scanners are calibrated by their manufacturers in laboratories that measure the instrument's systematic errors, such as the collimation and trunnion axis errors familiar from total stations, the vertical index error, eccentricities of the mirror and range offset, and store correction parameters in the instrument. Certificates state the date and the parameters checked. Between calibrations, users can check performance in the field by scanning a set of targets at known distances, by two-face measurements on instruments that support them, or by comparing scans with a reference such as a calibrated baseline. Range checks on a surveyed baseline, like the baselines used for electronic distance measurement, test the range offset and scale. Research has also developed self-calibration, adjusting the scanner's error parameters in a network of overlapping scans, but routine practice relies on manufacturer calibration and field checks.

Writing and reading an accuracy statement

A complete statement says what was tested, how and against what. For the corridor it reads, in part: 'This data set was tested to meet ASPRS Positional Accuracy Standards for Digital Geospatial Data, Edition 2, for a 5 cm RMSEv vertical accuracy class. Actual NVA accuracy was found to be RMSEv = 4.7 cm, including checkpoint survey error, from 30 checkpoints on open terrain. VVA accuracy was found to be RMSEv = 10.5 cm, 95th percentile 19.9 cm, from 10 checkpoints.' Readers of other people's data should look for each element; a bare 'accurate to 10 cm' says nothing about how or where.