Computer Vision for Robotics
Turning camera images into metric answers a robot can act on: images, noise and the vision pipeline, the pinhole model and distortion, calibration and hand-eye calibration, filtering, edges and features, matching and RANSAC, epipolar geometry and stereo, visual odometry and bundle adjustment, depth images and point clouds, convolutional networks, detection and segmentation, fiducials, PnP and tracking, with PIPEDA and Law 25 privacy duties and a capstone pick-and-place perception design.
ROB-330 Year 3 3 credits 12 lessons · 6 h 10 min
Offered by HSM University · Robotics
This is the Canadian version of this course. See the American version.
What you'll learn
- Describe digital images as arrays of noisy light measurements, choose colour spaces and thresholds, and explain the stages of a robot vision pipeline and the privacy duties that come with cameras
- Apply the pinhole camera model with intrinsics, extrinsics and lens distortion to project points into images and back-project pixels into rays
- Plan and evaluate a camera calibration, including intrinsics, distortion, uncertainty and hand-eye extrinsics, and judge whether the result can be trusted
- Filter images, compute gradients and edges, detect corners at several scales and describe them with descriptors such as SIFT and ORB
- Match features with ratio and consistency tests and estimate homographies and other models robustly with RANSAC, choosing the number of iterations
- Use epipolar geometry and the essential matrix, and compute and analyse depth from rectified stereo disparity, including how error grows with range
- Explain and assemble a visual odometry pipeline with two-view motion, triangulation, scale handling, drift and bundle adjustment
- Process depth images and point clouds by back-projection, filtering, voxel downsampling, normal estimation, RANSAC plane fitting, clustering and ICP
- Explain how convolutional networks are built, trained and evaluated, and deploy learned classifiers with attention to metrics, calibration and distribution shift
- Use and evaluate object detectors and segmentation models with IoU, non-maximum suppression, precision-recall curves and AP, and lift detections into 3-D
- Estimate 6-D object poses from fiducial markers, PnP and model-based methods, and track multiple objects with motion models and optimal assignment
- Design, budget and plan the verification of a complete perception pipeline for a robotic pick-and-place task
Course content
12 lessons · 6 h 10 min total · final exam
- 1 Images as data and the robot vision pipeline 29 min · quiz
- 2 Image formation and the pinhole camera model 31 min · quiz
- 3 Camera calibration 30 min · quiz
- 4 Image filtering, edges and features 30 min · quiz
- 5 Feature matching and robust estimation 31 min · quiz
- 6 Epipolar geometry and stereo vision 29 min · quiz
- 7 Visual odometry and structure from motion 31 min · quiz
- 8 Depth sensing and point clouds 34 min · quiz
- 9 Learning-based perception I: classification and convolutional networks 31 min · quiz
- 10 Learning-based perception II: detection and segmentation 28 min · quiz
- 11 Tracking and 6-D pose estimation 32 min · quiz
- 12 Capstone: a perception pipeline for a pick-and-place robot 34 min · quiz
- E Final exam 20 questions
Requirements
Recommended background. You can start any course at any time.
Description
Turning camera images into metric answers a robot can act on: images, noise and the vision pipeline, the pinhole model and distortion, calibration and hand-eye calibration, filtering, edges and features, matching and RANSAC, epipolar geometry and stereo, visual odometry and bundle adjustment, depth images and point clouds, convolutional networks, detection and segmentation, fiducials, PnP and tracking, with PIPEDA and Law 25 privacy duties and a capstone pick-and-place perception design.
Part of the Science stream, typically taken in year 3.