creating driving scenarios from recorded vehicle data for ... · actuation – electric power...
TRANSCRIPT
![Page 1: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/1.jpg)
1
Creating Driving Scenarios from Recorded Vehicle
Data for Validating Lane Centering System in
Highway Traffic
Gabriel Choi, General Motors LLC
Seo-Wook Park, MathWorks
![Page 2: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/2.jpg)
2
Contents
▪ Super Cruise – History and Future
▪ Lane Centering with Super Cruise
▪ Motivation
▪ Create virtual driving scenario from recorded data
▪ Simulate closed-loop model for lane centering system
▪ Conclusion
▪ Q & A
![Page 3: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/3.jpg)
3
Super Cruise – History and Future
▪ Debuted in 2017 with CT6 Sedan
– Lane Centering in addition to Full Speed Range Adaptive Cruise Control
– Uses High-Definition Map and Front Camera to detect Lane Marks
▪ Automated Lane Change for 2021 Cadillac CT4 / CT5 / Escalade
– Lane Change following Driver Request
– Able to accelerate and decelerate slightly to search gap to change in
▪ Eventually will be expanded to many name plates
![Page 4: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/4.jpg)
4
Lane Centering with Super Cruise on Cadillac CT6
▪ Sensors– Pre-Scanned High Definition Map
– Map matching with GPS
– Camera
– Long Range Radar
– Short Range Radars
▪ Actuation– Electric Power Steering
▪ Driver Monitoring System for Safety– Infra-red Face Recognition
– Steering Wheel Touch Sensor
– Chime and Vibration Seat
CameraView
Long Range RadarShort Range
Radar
GPS
![Page 5: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/5.jpg)
5
System validation for driving automation system
▪ Pains
– Big data size from “tens of thousands of miles” test drive
▪ Time consuming for data analysis
– Not easy to reproduce a real-world traffic situation with
closed-loop simulation
▪ Hazardous test scenarios
▪ Unwanted system behavior
▪ Virtual driving scenario from recorded data
– Reduce development time
– Enable closed-loop simulation to identify the root causes for unwanted system behavior
➔How to create driving scenario from recorded data?
![Page 6: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/6.jpg)
6
Record and select data
Reconstruct road network
Localize ego trajectory
Reconstruct target vehicles
Compare with recorded video
Record and select data
Reconstruct road network
Localize ego trajectory
Reconstruct target vehicles
Compare with recorded video
Record and select data
Reconstruct road network
Localize ego trajectory
Reconstruct target vehicles
Compare with recorded video
Create virtual driving scenario from recorded data
mdf
GPS Vehicle
Data On-board sensors
MDF(Measurement Data Format)
mp4
![Page 7: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/7.jpg)
7
Create virtual driving scenario from recorded data
Record and select data
Reconstruct road network
Localize ego trajectory
Reconstruct target vehicles
Compare with recorded video
(GPS)
High-Definition Map with Tiled Layers
HD Localization Model
mdf
Road Centerline Model
Topology Geometry
Routing Attributes
ADAS Attributes
Speed Attributes
Routing Lane Attributes
External Ref Attributes
HD Lane Model
Lane Topology
Lane Geometry Lane Attributes
Lane Road References
![Page 8: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/8.jpg)
8
Create virtual driving scenario from recorded data
Record and select data
Reconstruct road network
Localize ego trajectory
Reconstruct target vehicles
Compare with recorded video
Create driving scenario
Import road centers,
lane information
(GPS)
mdf
![Page 9: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/9.jpg)
9
Create virtual driving scenario from recorded data
Record and select data
Reconstruct road network
Localize ego trajectory
Reconstruct target vehicles
Compare with recorded video
GPS
Vehicle Speed
Ego car
trajectory(egoCar,waypoints,speeds)
![Page 10: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/10.jpg)
10
Lateral deviation of ego vehicle from lane center
Lateral
deviation
lane center
GPS based trajectory
Camera lane sensor
left lane boundary
right lane boundary
![Page 11: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/11.jpg)
11
Localized ego trajectory by lane sensor data
GPS based trajectory
Localized ego trajectory
Camera lane sensor
Lateral
deviation
lane center
localized
left lane boundary
right lane boundary
![Page 12: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/12.jpg)
12
Visualize data
Map with GPS
position
Video with
time stamp,
ego speed
& yaw rate
Driving scenario view Bird’s-eye plot with
sensor detections
geoplayer
birdsEyePlot
VideoReaderimshow
drivingScenario
![Page 13: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/13.jpg)
13
Create virtual driving scenario from recorded data
Record and select data
Reconstruct road network
Localize ego trajectory
Reconstruct target vehicles
Compare with recorded video
![Page 14: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/14.jpg)
14
Create virtual driving scenario from recorded data
Record and select data
Reconstruct road network
Localize ego trajectory
Reconstruct target vehicles
Compare with recorded video
Radar
Detections
Target
vehicles
Report
targets only
in ego and
neighbor lanes
![Page 15: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/15.jpg)
15
Target vehicle trajectory
▪ Target vehicle trajectory is defined by a series
of actor poses
▪ Actor poses consist of
– Position 𝑋𝑡 , 𝑌𝑡
– Velocity 𝑣𝑡
– Orientation Ψ𝑡
in world coordinate
X
Yworld coordinate
𝑋𝑡, 𝑌𝑡
𝑣𝑡
Ψ𝑡
![Page 16: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/16.jpg)
16
Estimate heading angle of target vehicle
▪ Radar detections provides target position and
velocity in ego coordinate
– 𝑥𝑡 , 𝑦𝑡 , 𝑣𝑥, 𝑣𝑦 , (𝜑𝑡? ) ← 𝑟𝑎𝑑𝑎𝑟 𝑑𝑒𝑡𝑒𝑐𝑡𝑖𝑜𝑛𝑠
▪ Estimate heading angle of target vehicle
– Heading angle in ego coordinate
▪ 𝜑𝑡 = tan−1𝑣𝑦
𝑣𝑥
– Heading angle in world coordinate
▪ Ψ𝑡𝑎𝑟𝑔𝑒𝑡 = 𝜑𝑡 + Ψ𝑒𝑔𝑜
▪ Target position in world coordinate
– 𝑋𝑡 , 𝑌𝑡 = 𝑋𝑒𝑔𝑜 , 𝑌𝑒𝑔𝑜 + ℝ 𝛹𝑒𝑔𝑜 ⋅ 𝑥𝑡 , 𝑦𝑡𝑇
X
Y
x
y
Ψ𝑒𝑔𝑜
(𝑋𝑒𝑔𝑜, 𝑌𝑒𝑔𝑜)
world coordinate
𝑥𝑡 , 𝑦𝑡𝑣𝑦
𝑣𝑥
𝜑𝑡
𝑣𝑒𝑔𝑜
𝑣𝑡
Ψ𝑡
![Page 17: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/17.jpg)
17
Extract target trajectory from radar detections
ID=50
Extract target trajectory
![Page 18: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/18.jpg)
18
Example of estimated yaw angle
![Page 19: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/19.jpg)
19
Create virtual driving scenario from recorded data
Record and select data
Reconstruct road network
Localize ego trajectory
Reconstruct target vehicles
Compare with recorded video
Report radar
detections
in ego and
neighbor lanes
![Page 20: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/20.jpg)
20
Synthesize
sensors
Synthesize sensors with virtual driving scenario
Record and select data
Reconstruct road network
Localize ego trajectory
Reconstruct target vehicles
Compare with recorded video
![Page 21: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/21.jpg)
21
Synthesize
sensors
Integrate driving scenario + sensors with a close-loop system
Record and select data
Reconstruct road network
Localize ego trajectory
Reconstruct target vehicles
Compare with recorded video
CameraView
Long Range RadarShort Range
Radar
GPS
Integrate driving scenario + sensor models
with a close-loop system
![Page 22: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/22.jpg)
22
Driving
Scenario
Closed-loop system for lane centering
Virtual driving scenario
from recorded data
Sensor Models
Ego Vehicle
Dynamics
Speed Ego pose
Sensor
Fusion and
TrackingVision
Estimate
Lane Center
Preview
Curvature
Find
Lead CarModel
Predictive
Control
Lane
RadarAccel
Steering
Lane Centering System Under TestUse surrogate algorithm
Vision Detection
Generator
Radar Detection
Generator targets
![Page 23: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/23.jpg)
23
Surrogate closed-loop system for lane centering
Lane following
controller
Vehicle and
environment
Collision
detection
Lane Following Control with Sensor Fusion and Lane Detection
Automated Driving ToolboxTM
![Page 24: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/24.jpg)
24
Surrogate closed-loop system for lane centering
Lane following
controller
Vehicle and
environment
Collision
detection
Ego vehicle
dynamics
Radar detection
sensor model
Vision detection
sensor modelDriving scenario reader
(target vehicle poses)
![Page 25: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/25.jpg)
25
Lane centering test bench with recorded target vehicles
![Page 26: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/26.jpg)
26
Closed-loop simulation using reconstructed virtual driving scenario
![Page 27: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/27.jpg)
27
Simulation result (longitudinal & lateral control performance)
longitudinal control performance lateral control performance
a b c
a,b,c : problem cases where headway distance drops below the safe distance.
![Page 28: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/28.jpg)
28
Driving case (b): cut-in vehicle at low speed
b
longitudinal control performance
a,b,c : problem cases where headway distance drops below the safe distance.
![Page 29: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/29.jpg)
29
Driving case (c): cut-in vehicle with too close distance
c
longitudinal control performance
a,b,c : problem cases where headway distance drops below the safe distance.
![Page 30: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/30.jpg)
30
Closed-loop
Simulation
Conclusion
▪ Created virtual driving scenario from recorded data
▪ Reproduced real-world driving scenario in the virtual simulation environment
– Assess functional behavior and identify root cause for problem cases
– Reduce development time with limited resources
– Enable repetitive tests for hazardous scenarios
Record and select data
Reconstruct road network
Localize ego trajectory
Reconstruct target vehicles
Compare with recorded video
GPS
Vehicle SpeedHD Map
OpenStreetMap
Radar
Camera
Lidar
OpenDRIVE
![Page 31: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/31.jpg)
31
Remark
▪ Collaborative effort between GM and MathWorks.
▪ This study has been published in the SAE paper.
![Page 32: Creating Driving Scenarios from Recorded Vehicle Data for ... · Actuation – Electric Power Steering Driver Monitoring System for Safety – Infra-red Face Recognition – Steering](https://reader034.vdocument.in/reader034/viewer/2022042400/5f0f6cc37e708231d444186c/html5/thumbnails/32.jpg)
32
Presenter contact info and poll questions
▪ Poll questions : I found this
technique the most interesting
1. Access mdf data
2. Road network creation from HD map
3. Ego vehicle localization
4. Reconstruct target vehicles
5. Data visualization
6. Close-loop system integration for lane
centering with Simulink
▪ If you would like to an individual
follow-up, please provide your
name and email address in the
WebEx poll area.
Please contact us with questions• Gabriel Choi, General Motors LLC ([email protected])
• Seo-Wook Park, MathWorks ([email protected])