sensor fusion and tracking for autonomous systems€¦ · signal and image processing control ......
TRANSCRIPT
![Page 1: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/1.jpg)
1© 2015 The MathWorks, Inc.
Sensor Fusion and Tracking for
Autonomous Systems
Marc Willerton
Senior Application Engineer
MathWorks
![Page 2: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/2.jpg)
2
Abstract
▪ There is an exponential growth in the development of increasingly autonomous systems. These
systems range from road vehicles that meet the various NHTSA levels of autonomy, through
consumer quadcopters capable of autonomous flight and remote piloting, package delivery drones,
flying taxis, and robots for disaster relief and space exploration. Work on autonomous systems
spans industries and includes academia as well as government agencies.
▪ In this talk, you will learn to design, simulate, and analyze systems that fuse data from multiple
sensors to maintain position, orientation, and situational awareness. By fusing multiple sensors
data, you ensure a better result than would otherwise be possible by looking at the output of
individual sensors.
▪ Several autonomous system examples are explored to show you how to:
– Define trajectories and create multiplatform scenarios
– Simulate measurements from inertial and GPS sensors
– Generate object detections with radar, EO/IR, sonar, and RWR sensor models
– Design multi-object trackers as well as fusion and localization algorithms
– Evaluate system accuracy and performance on real and synthetic data
![Page 3: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/3.jpg)
3
Capabilities of an Autonomous System
Sense
![Page 4: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/4.jpg)
4
Capabilities of an Autonomous System
Sense
Perceive
![Page 5: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/5.jpg)
5
Capabilities of an Autonomous System
Learning Algorithms
Optimization
Sense
Perceive
Decide
& Plan
![Page 6: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/6.jpg)
6
Capabilities of an Autonomous System
Control Algorithms
Sense
Perceive
Decide
& Plan
Act
![Page 7: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/7.jpg)
7
Agenda
▪ Introduction
▪ Technology overview of perception
▪ Sensor models for sensor fusion and tracking
▪ Building simulation scenarios
▪ Developing a Multi-Object Tracker
▪ Tracking from multiple platforms
▪ Connecting trackers to a control system
▪ Q&A
![Page 8: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/8.jpg)
8
Agenda
▪ Introduction
▪ Technology overview of perception
▪ Sensor models for sensor fusion and tracking
▪ Building simulation scenarios
▪ Developing a Multi-Object Tracker
▪ Tracking from multiple platforms
▪ Connecting trackers to a control system
▪ Q&A
![Page 9: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/9.jpg)
9
Sensor Fusion and
Tracking
Self- awareness Situational awareness
Accelerometer, Magnetometer,
Gyro, GPS…
Radar, Camera, IR, Sonar, Lidar,
…
Signal and Image
ProcessingControl
Sensor fusion and tracking is…
![Page 10: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/10.jpg)
10
Timeline of Technology Advances
Military Commercial Ubiquitous
TodayTimeline
Multi-sensor Fusion
for Autonomous Systems
Computer Vision
for Transportation
Multi-object
tracking
Localization
Air Traffic Control
![Page 11: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/11.jpg)
11
Fusion Combines the Strengths of Each Sensor
Vision measurement
at time step k
Radar measurement
at time step k
Fused
estimate at
time step k
Fused estimate
at time step k-1
Predicted estimate
at time step k
Cross range
Down range
Vision
Measurement
Radar
measurement
Track (fused
estimate)
Ellipse
represents
uncertainty
Legend
![Page 12: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/12.jpg)
12
What is Localization?
PositionInertial Sensor Attitude
![Page 13: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/13.jpg)
13
Agenda
▪ Introduction
▪ Technology overview of perception
▪ Sensor models for sensor fusion and tracking
▪ Building simulation scenarios
▪ Developing a Multi-Object Tracker
▪ Tracking from multiple platforms
▪ Connecting trackers to a control system
▪ Q&A
![Page 14: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/14.jpg)
14
Sensor Models for Sensor Fusion and Tracking
![Page 15: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/15.jpg)
15
Exploring gyro model in Sensor Fusion and Tracking Toolbox
![Page 16: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/16.jpg)
16
Exploring gyro model in Sensor Fusion and Tracking Toolbox
Imposing ADC Quantisation Effects Imposing White Noise
![Page 17: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/17.jpg)
17
Exploring gyro model in Sensor Fusion and Tracking Toolbox
Imposing Brown Noise Imposing Pink Noise
![Page 18: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/18.jpg)
18
Exploring gyro model in Sensor Fusion and Tracking Toolbox
Imposing Temperature Scaled Bias
For more information, see example.
![Page 19: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/19.jpg)
19
radar = monostaticRadarSensor(‘Rotator’) radar = monostaticRadarSensor(‘Sector’)
radar = monostaticRadarSensor(‘Mechanical Raster’) radar = monostaticRadarSensor(‘Electronic Raster’)
Levels of Fidelity: Radar Detections vs. I/Q Samples Simulation
![Page 20: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/20.jpg)
20
Generating Radar Detections in MATLAB
Target positions Simulation time
Sensor ID
Detections (time,
measurement, etc…)
For more information,
see example hereTBC – we will see how to build
this as a scenario later…
![Page 21: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/21.jpg)
21
Fusing Sensor Data Improves Localization
Error MeasurementsSensorsGround truth vs. Estimate
Example here
![Page 22: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/22.jpg)
22
Fuse IMU & GPS for Self-Localization of a UAV
Sense
Perceive
Decide
& Plan
Act
Locate
Self
Track
Obstacles
Example here
![Page 23: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/23.jpg)
24
Fuse IMU & Odometry for Self-Localization in GPS-Denied Areas
VO estimate off
by a scale factor
IMU dead reckoning
drift
![Page 24: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/24.jpg)
25
Fuse IMU & Odometry for Self-Localization in GPS-Denied Areas
Sense
Perceive
Decide
& Plan
Act
Locate
Self
Track
Obstacles
![Page 25: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/25.jpg)
26
Scenario Definition and Sensor Simulation
Flexible Workflows Ease Adoption: Wholesale or Piecemeal
Ownship
Trajectory
Generation
INS Sensor
Simulation
Recorded
Sensor Data
Visualization
&
Metrics
Algorithms
gnnTrackergnnTrackerINS Filter,
Tracker, etc..
Actors/
PlatformsRadar, IR,
& Sonar
Sensor
Simulation
Documented
Interface
for detections
Documented
Interface
for tracks
![Page 26: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/26.jpg)
27
Stream Data to MATLAB from IMUs Connected to Arduino
MEMS Devices
▪ 9-axis (Gyro + Accelerometer + Compass)
▪ 6 axis (Gyro + Accelerometer)
▪ Up to 200 Hz sampling rate
![Page 27: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/27.jpg)
28
New Hardware and Multisensor Positioning Examples
![Page 28: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/28.jpg)
29
Agenda
▪ Introduction
▪ Technology overview of perception
▪ Sensor models for sensor fusion and tracking
▪ Building simulation scenarios
▪ Developing a Multi-Object Tracker
▪ Tracking from multiple platforms
▪ Connecting trackers to a control system
▪ Q&A
![Page 29: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/29.jpg)
30
Building a Simulation Scenario
Example here
![Page 30: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/30.jpg)
31
Test Tracker Performance on Pre-Built Benchmark Trajectories
Reference
W.D. Blair, G. A. Watson, T. Kirubarajan, Y. Bar-Shalom, "Benchmark for Radar Allocation and Tracking in
ECM." Aerospace and Electronic Systems IEEE Trans on, vol. 34. no. 4. 1998
![Page 31: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/31.jpg)
32
To go further on localization, see also
![Page 32: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/32.jpg)
33
Agenda
▪ Introduction
▪ Technology overview of perception
▪ Sensor models for sensor fusion and tracking
▪ Building simulation scenarios
▪ Developing a Multi-Object Tracker
▪ Tracking from multiple platforms
▪ Connecting trackers to a control system
▪ Q&A
![Page 33: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/33.jpg)
34
Detections
Multitarget Tracker
TracksTracking
Filter
Track
Association
and
Management
From various sensors at
various update rates
▪ Assigns detections to tracks
▪ Creates new tracks
▪ Updates existing tracks
▪ Removes old tracks
A Multi-object Tracker is More than a Kalman Filter
▪ Fuses measurements with
the track state
![Page 34: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/34.jpg)
35
Scenario Definition and Sensor Simulation
Tracking Algorithm Development Workflow
Ownship
Trajectory
Generation
INS Sensor
Simulation
Recorded
Sensor Data
Visualization
&
Metrics
Tracking Algorithms
gnnTrackergnnTrackerGNN, MHT, etc..
Actors/
Platforms
Radar, IR, &
Sonar Sensor
Simulation
objectDetection tracks
, JPDA, PHD
![Page 35: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/35.jpg)
36
Components of a Multi-Object Tracker
Inertial Navigation
System provides radar
sensor platform position,
velocity and orientation
More information here
Tracking Filter
Track Association
and Management
![Page 36: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/36.jpg)
37
Components of a Multi-Object Tracker
Tracking Filter
More information here
statetime step
output
process noise
(e.g. model error)
Measurement
(sensor) noiseCommon assumption:
Gaussian noise => Kalman Filter
Covariance estimate assumes system is linear:
If y1(t) = H{x1}, y2(t) = H{x2}, then
Ay1(t) + By2(t) = H{Ay1(t) + By2(t)}
Linear Kalman Filter – Assumes linear system
Extended Kalman Filter – Linearizes the nonlinear
system around state estimate
Unscented Kalman Filter – Samples the covariance
distribution and propagates through non-linear model
Gaussian Sum Filter – good for partially observable
cases (e.g. range only measurements)
Interactive Multiple Model Filter – good for tracking
manoeuvring targets
Particle Filter – doesn’t require gaussian noise1. Create the measured positions from a
constant-velocity trajectory
2. Specify initial position and velocity 3. Run Kalman Filter
Linear Kalman Filter Example
State Initialisation (e.g. constant velocity/acceleration/turn
and Motion Models available or use your own
![Page 37: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/37.jpg)
38
Components of a Multi-Object Tracker
Track Association and Management
More information here
One or more sensors generating multiple detections from multiple targets – detections must be:
1. Gated – determine which detections are valid candidates to update existing tracks
2. Assigned* – make a track to detection assignment. Assignment approaches include:
> Global Nearest Neighbour – Minimise overall distance of track to detection assignments
> Joint Probability Data Association – Soft assignment so all gated detections make weighted contributions to a track
> Track Orientated Multiple Hypothesis Tracking – Allows data association to be postponed until more information is received
Track maintenance is required for creation (tentative status), confirmation, deletion of tracks (after coasting)
> Can use history or score based logic
Advanced Topic – Track to Track Fusion:
Lowest
Complexity
Best
Performance
* Some trackers (e.g. PHD
Filter) don’t require assignment
![Page 38: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/38.jpg)
39
Components of a Multi-Object Tracker
Track Association and Management
Time
Data
Track Creation
Track Deletion
More information here
![Page 39: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/39.jpg)
40
Search
Track (confirm)
Track (update)
Example of Multi-Object Tracking:
Multifunction Radar: Search and Track
Example here
![Page 40: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/40.jpg)
41
Example of Multi-Object Tracking:
Multifunction Radar: Search and Track
![Page 41: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/41.jpg)
42
Target 1 Detected
Example of Multi-Object Tracking:
Multifunction Radar: Search and Track
![Page 42: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/42.jpg)
43
Detection
Confirmed and
Track 1 Created
Example of Multi-Object Tracking:
Multifunction Radar: Search and Track
![Page 43: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/43.jpg)
44
Track 1 Updated
Example of Multi-Object Tracking:
Multifunction Radar: Search and Track
![Page 44: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/44.jpg)
45
1° Azimuth
Resolution
+ 270m
at 30km
Two targets seen
as one by the radar
Did the trajectories cross?
Example of Multi-Object Tracking:
Performing What-If Analysis
Example here
![Page 45: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/45.jpg)
46
+ 175m
at 10km
+ 9m
at 1km
tracker = trackerGNN( ...
'FilterInitializationFcn',@initCVFilter,...
'MaxNumTracks', numTracks, ...
'MaxNumSensors', 1, ...
'AssignmentThreshold',gate, ...
'TrackLogic', 'Score', ...
'DetectionProbability', pd, ...
'FalseAlarmRate', far, ...
'Volume', vol, 'Beta', beta);
tracker = trackerGNN( ...
'FilterInitializationFcn',@initIMMFilter,...
'MaxNumTracks', numTracks, ...
'MaxNumSensors', 1, ...
'AssignmentThreshold',gate, ...
'TrackLogic', 'Score', ...
'DetectionProbability', pd, ...
'FalseAlarmRate', far, ...
'Volume', vol, 'Beta', beta);
Example of Multi-Object Tracking:
Performing What-If Analysis
![Page 46: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/46.jpg)
47
tracker = trackerTOMHT( ...
'FilterInitializationFcn',@initIMMFilter,...
'MaxNumTracks', numTracks, ...
'MaxNumSensors', 1, ...
'AssignmentThreshold’,[0.2,1,1]*gate, ...
'TrackLogic', 'Score', ...
'DetectionProbability', pd, ...
'FalseAlarmRate', far, ...
'Volume', vol, 'Beta', beta, ...
'MaxNumHistoryScans', 10, ...
'MaxNumTrackBranches', 5,...
'NScanPruning', 'Hypothesis', ...
'OutputRepresentation', 'Tracks');
Example of Multi-Object Tracking:
Performing What-If Analysis
![Page 47: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/47.jpg)
48
False track
Dropped track
Slower Faster
Example of Multi-Object Tracking:
Comparing Trackers and Tracking Filters
![Page 48: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/48.jpg)
49
Simulink Support for Multi-Object Tracking
![Page 49: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/49.jpg)
50
Point object vs. Extended object
▪ Point object
– Distant object represented as a single point
– One detection per object per scan
▪ Extended object
– High resolution sensors generate
multiple detections per object per scan
![Page 50: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/50.jpg)
51
Extended Object Tracking Example
Sense
Perceive
Decide
& Plan
Act
Locate
Self
Track
Obstacles
![Page 51: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/51.jpg)
52
Tracking with Lidar (Even more points!!)
Sense
Perceive
Decide
& Plan
Act
Locate
Self
Track
Obstacles
JPDA Tracker with IMM
Pre-process point cloud data to extract objects of interest. Example here.
![Page 52: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/52.jpg)
53
To go further on tracking with a single sensor, see also
![Page 53: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/53.jpg)
54
Agenda
▪ Introduction
▪ Technology overview of perception
▪ Sensor models for sensor fusion and tracking
▪ Building simulation scenarios
▪ Developing a Multi-Object Tracker
▪ Tracking from multiple platforms
▪ Connecting trackers to a control system
▪ Q&A
![Page 54: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/54.jpg)
55
Multi-platform Scenario Generation
Moving Airborne Radar (red)
2 ULAs mounted above fuselage
Electronically scan 120° az sector on both sides of airframe
Stationary Ground Based Radar (yellow)
Electronically scanned URA
Raster scan surveying +/-60° az and -20 to 0° el
Moving Airborne Radar (blue)
360° mechanical scan in az
No electronic scanning
Example here
![Page 55: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/55.jpg)
56
Airborne ULA can’t
measure Elevation
Multi-platform Scenario Generation
Visualize Detections and Measurement Uncertainties
Mechanically scanning
radar detects target
only 2 times
Ellipsoids
represent
uncertainties
Example here
![Page 56: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/56.jpg)
57
Multi-platform Scenario Generation
Visualize Track Accuracy
Good track
altitude
estimation
despite poor
measurements
Motion model
mismatch
(CV vs. CT)
![Page 57: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/57.jpg)
58
Multi-platform Scenario Generation
Tune and Compare Trackers with Assignment Metrics
Time to
confirm tracks
Track T09
for P1
7 objects
(P1, P2, …)
![Page 58: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/58.jpg)
59
Multi-platform Scenario Generation
Assess Tracker Performance with Assignment Metrics
Dropped
track
Dropped
track
False
track
False
track
![Page 59: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/59.jpg)
60
Multi-platform Scenario Generation
Jamming Scenario
Jammer
prevents
detection
Example here
![Page 60: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/60.jpg)
61
Where are the real targets? How to remove the
ghosts?
Multi-platform Scenario Generation
Positioning with direction only measurements
Example here
![Page 61: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/61.jpg)
62
Agenda
▪ Introduction
▪ Technology overview of perception
▪ Sensor models for sensor fusion and tracking
▪ Building simulation scenarios
▪ Developing a Multi-Object Tracker
▪ Tracking from multiple platforms
▪ Connecting trackers to a control system
▪ Q&A
![Page 62: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/62.jpg)
63
Simulate a lane detection and lane following system
Simulation Environment
Lane
following
controller
Scenarios
Goals &
MetricsSensors
Actions
& EventsActors
SceneryMonocular
camera lane
detector
Sense
Perceive
Decide
& Plan
Act
Locate
Self
Track
Obstacles
![Page 63: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/63.jpg)
64
Monocular camera lane detector
- Based on shipping example
- Lane rejection and tracking added to
improve performance
Visual Perception Using
Monocular Camera
Automated Driving ToolboxTM
Simulate a lane detection and lane following system
![Page 64: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/64.jpg)
65
Simulate a lane detection and lane following system
Challenge with noisy lanes
Approaching car gets confused with road
![Page 65: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/65.jpg)
66
Simulate a lane detection and lane following system
Integrate noisy lane rejection and tracking
Vehicles &
Lanes
With correction
“EnableLaneTracker” to enable noisy lane
rejection
rejectInvalidLanes
Before correction
Property ofhelperMonoSensorWrapper
Method ofhelperMonoSensorWrapper
helperMonoSensorSystem object
Lanes Lanes
helperMonoSensorWrapperSystem Object
![Page 66: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/66.jpg)
67
Simulate a lane detection and lane following system
Integrate noisy lane rejection and tracking
Example: Before Correction Example: After Correction
![Page 67: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/67.jpg)
69
Simulate a lane detection and lane following system
Tracker Setup and Configuration
Kalman Tracker
• Two instances of Kalman tracker to track left and right lanes independently.• Initialize trackers for first valid lanes using “ConfigureKalmanFilter”• Input Parameters for tracking: A,B,C coefficients of parabolic lane boundaries • Motion model : Constant Acceleration• Correct tracker for every valid lane
![Page 68: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/68.jpg)
70
Simulate a lane detection and lane following system
Steering Angle deviation for simulation run
Without noisy lane rejector With noisy lane rejector
![Page 69: Sensor Fusion and Tracking for Autonomous Systems€¦ · Signal and Image Processing Control ... Multi-sensor Fusion for Autonomous Systems Computer Vision for Transportation Multi-object](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed328e46aa2e01f9b12aaaf/html5/thumbnails/69.jpg)
71
Sensor Fusion and Tracking …
Leverages Sensor StrengthsIs Ubiquitous
Sense
Perceive
Decide
& Plan
Act
Sensor Fusion
and Tracking
Signal and Image
ProcessingControl
Enables Autonomy