perception for automated and assisted...

PERCEPTION FOR AUTOMATED AND ASSISTED DRIVING

DR.-ING. MICHAEL DARMS GROUP RESEARCH / HEAD OF SENSORS AND FUSION IROS – 28.09.2015

2 Volkswagen AG | Konzernforschung

ENVIRONMENT PERCEPTION

Motivation

• Highly automated driving functions impose increased requirements on the performance of the perception module

• Perception module has to cope with contradictory requirements (comport vs. safety systems)

• Development of a modular and extensible perception architecture

• Environment model has to be independent of specific ADAS function

• Implementation of unified interface to function

Goals

3 Volkswagen AG | Konzernforschung

Representation of the world around the vehicle

Static Environment

Road / Lanes

Dynamic Objects

Road Graph

WORLD MODEL

4 Volkswagen AG | Konzernforschung

OBJECT ESTIMATION

Mono Camera Laser Radar

• Estimating of moving/ movable objects and dynamic model

• Benefit from individual sensor detection capabilities

5 Volkswagen AG | Konzernforschung

GRID ESTIMATION

Ultrasonic Front Camera

2D Layer Fusion

• Estimate Free Space

• Estimate Occupied Space

• Exclude Movable/ Moving Objects

6 Volkswagen AG | Konzernforschung

ROAD ESTIMATION

Highway Lane Markings Other Vehicles

Rural Roads Lane Markings Other Vehicles Curbs, sward

Inner City Lane Markings Other Vehicles Curbs Traffic Lights, Trafic Signs Digital Maps

• Interpret where in the world vehicles should drive using feature cues from the environment

7 Volkswagen AG | Konzernforschung

SCENE ESTIMATION

RoadGraph • Output of the perception modules is

integrated into one model • Roadgraph is main interface to

driving functions Challenges • Fusion of road estimation and

context knowledge into unified, consistent and comprehensive model

• Scene Estimation for urban areas, especially in complex intersection scenarios

• Handling of traffic participants that do not behave as expected

8 Volkswagen AG | Konzernforschung

V-CHARGE: AUTONOMOUS VALET PARKING AND CHARGING FOR E-MOBILITY

Automated valet parking and charging • no time-consuming search for parking spots any more • driverless valet service • no human intervention Fully automated driving • in mixed-traffic scenarios • in indoor and outdoor parking lots and parking garages

without GPS

9 Volkswagen AG | Konzernforschung

V-CHARGE: AUTONOMOUS VALET PARKING AND CHARGING FOR E-MOBILITY Close-to-series sensors

Stereo- Camera

Area- view

Ultra- sonic

Car2x

10 Volkswagen AG | Konzernforschung

OBJECT ESTIMATION USING FISH-EYE CAMERAS

University of Parma, VisLab

11 Volkswagen AG | Konzernforschung

GRID ESTIMATION USING FISH-EYE CAMERAS

C. Häne, T. Sattler, M. Pollefeys, Obstacle Detection for Self-Driving Cars Using Only Monocular Cameras and Wheel Odometry

12 Volkswagen AG | Konzernforschung

GRID ESTIMATION WITH ALL SENSORS

06/25/2015 22

Ultrasonic Grid Stereo Grid

Motion Stereo Grid Fused Grid

13 Volkswagen AG | Konzernforschung

GRID ESTIMATION WITH ALL SENSORS

06/25/2015 22

Ultrasonic Grid Stereo Grid

Motion Stereo Grid Fused Grid

14 Volkswagen AG | Konzernforschung

ROAD ESTIMATION: KNOWING WHERE THE ROAD IS VIA LOCALIZATION

14

Setup

15 Volkswagen AG | Konzernforschung

APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION

15

Survey Run

Furgale et al., ICRA2014

16 Volkswagen AG | Konzernforschung

APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION

16

Survey Run

Online Run

Survey Run

Furgale et al., ICRA2014

17 Volkswagen AG | Konzernforschung 17

Furgale et al., ICRA2014

18 Volkswagen AG | Konzernforschung

APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION

18

Furgale et al., ICRA2014

19 Volkswagen AG | Konzernforschung

APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION

19

Furgale et al., ICRA2014

20 Volkswagen AG | Konzernforschung

APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION

20

Furgale et al., ICRA2014

21 Volkswagen AG | Konzernforschung

October 13, 2013 15:15

22 Volkswagen AG | Konzernforschung

November 8, 2013 10:26

23 Volkswagen AG | Konzernforschung

December 6, 2013 13:50

24 Volkswagen AG | Konzernforschung

July 31, 2014 15:45

25 Volkswagen AG | Konzernforschung

APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION

25

Summary

Mapping / Fusion Localization

Database

Summary map for localization

Localization output & raw data

Full-scale map

Mühlfellner, Peter : Lifelong Visual Localization for Automated Vehicles, Doctoral thesis

26 Volkswagen AG | Konzernforschung

APPROACH: KNOWING WHERE THE ROAD IS - LOCALIZATION

26

Find the most useful Landmarks:

Most observed landmarks from a single session:

Landmarks observed over the most different sessions:

Mühlfellner, Peter : Lifelong Visual Localization for Automated Vehicles, Doctoral thesis

27 Volkswagen AG | Konzernforschung

APPROACH: INTERPRETING WHERE THE ROAD IS – ROAD ESTIMATION

Mühlfellner, Peter : Lifelong Visual Localization for Automated Vehicles, Doctoral thesis Mühlfellner et al.: Summary Maps for Lifelong Visual Localization, JFR, 2015

19.0

9.

04.1

0.

11.1

0.

17.1

0.

23.1

0.

31.1

0.

07.1

1.

08.1

1.

12.1

1.

19.1

1.

03.1

2.

06.1

2.

10.1

2.

12.1

2.

14.0

1.

04.0

4.

14.0

4.

25.0

5.

28.0

4.

30.0

4.

05.0

5.

14.0

5.

21.0

5.

26.0

5.

28.0

5.

11.0

6.

30.0

6.

02.0

7.

16.0

7.

23.0

7.

30.0

7.

31.0

7.

100%

75%

50%

25%

0%

1 2 3 4 5 8 9 6 7 10

11

12

Full Multi-Session Map Summary Map

Date, Arrows Indicate Mapping Sessions

% S

ucce

ssfu

l Loc

aliz

atio

ns

Success Rate

28 Volkswagen AG | Konzernforschung

APPROACH: INTERPRETING WHERE THE ROAD IS – ROAD ESTIMATION

Mühlfellner, Peter : Lifelong Visual Localization for Automated Vehicles, Doctoral thesis Mühlfellner et al.: Summary Maps for Lifelong Visual Localization, JFR, 2015

19.0

9.

04.1

0.

11.1

0.

17.1

0.

23.1

0.

31.1

0.

07.1

1.

08.1

1.

12.1

1.

19.1

1.

03.1

2.

06.1

2.

10.1

2.

12.1

2.

14.0

1.

04.0

4.

14.0

4.

25.0

5.

28.0

4.

30.0

4.

05.0

5.

14.0

5.

21.0

5.

26.0

5.

28.0

5.

11.0

6.

30.0

6.

02.0

7.

16.0

7.

23.0

7.

30.0

7.

31.0

7.

1e7

1e6

1e5

1 2 3 4 5 8 9 6 7 10

11

12

Full Multi-Session Map Summary Map

Date, Arrows Indicate Mapping Sessions

Land

mar

ks T

otal

(Log

. Sca

le)

Number of Landmarks per Session

29 Volkswagen AG | Konzernforschung

CES DEMO DRIVE 2015

− About 900 km on Highways − 5 Journalists as drivers using the automated vehicle

Bakersfield / 5.1.2015 / 7:00

San Francisco / 4.1.2015 / 11:00

Las Vegas / 5.1.2015 / 16:00

CANV / 5.1.2015 / 12:00 Interstate I5 / 4.1.2015 / 15:00

30 Volkswagen AG | Konzernforschung

1 4

5

2x Laser-scanner

4x Ultrasonic 1x 3D-video-camera

4x Top View 4x Mid-range-radar

2x Long-range-radar

1x Stock GPS 2x Short Range Radar

2

2

3

3

4

2

1 2 3 4 5 6 7 8

6

7

8

1

TECHNOLOGY – TODAY

31 Volkswagen AG | Konzernforschung

KEY CHALLENGE: WHERE IS THE ROAD?

Simple Marking features, simple lane geometries

Medium Marking features, more complex geometries (smaller radius of curvature, not parallel geometries)

Complex Arbitrary features, arbitrary lane geometries

Robust and highly available marking features low demands

Marking features with more complex geometries medium-level demands

No clear road markings High demands on the scene interpretation

31

Töpfer, Spehr et al: Efficient Scene Understanding for Intelligent Vehicles Using a Part-Based Road Representation

32 Volkswagen AG | Konzernforschung

INPUT AND OUTPUT OF THE ROAD ESTIMATION • Spatial and temporal reasoning

Features

Patches

Dig. map

Result

Lanes

Road Estimation

Spat

ial a

nd te

mpo

ral

reas

onin

g

Standards

A-priori

Sensors

Road model

Credits Spehr et al

33 Volkswagen AG | Konzernforschung

INTEGRATION OF A-PRIORI KNOWLEDGE

• Standards: • Guidelines for building roads, freeways and street • Use of country-dependent policies (e.g. in Germany „Straßenbaunormen DIN

EN 1423 und DIN EN 1424“)

• Deviations are modeled with appropriate probability distributions

• Digital maps:

• Information beyond the detection range of the sensors • Used for integrating the expected road geometry and topology during

spatial reasoning.

34 Volkswagen AG | Konzernforschung

SENSOR SETUPS

• High-level sensor information (Lanes) • Covers a wide range in the vehicle’s environment • Trajectories of other vehicles • Lane output of external preprocessing units

• Low-level sensor information (Patches/ Features) • Spatially restricted features in the vehicle’s environment such as

boundary features • But also features like grid cells (occupied yes/no), color values of a

camera image

35 Volkswagen AG | Konzernforschung

HIERARCHICAL MODEL OF A SINGLE SCENE

35

The graphical model comprises hidden random variables 𝒙𝒙𝑖𝑖 and observations 𝒚𝒚𝑖𝑖 • Hidden variables represent parts and sub-parts of a scene encoded by the root node • Variables are continuous, multi-dimensional, and multi-modal

Edges encode probabilistic dependencies between pairs of variables

2-Lane Road Scene

Features

Right Lane Left Lane

Lane-Segment Lane-Segment

Patches

36 Volkswagen AG | Konzernforschung

INFERENCE DEPTH-FIRST MESSAGE PASSING bottom-up = generating a

hypothesis top-down = verifying the

hypothesis Example: 11. Verifying the road hypothesis (top-down)

36

37 Volkswagen AG | Konzernforschung

IS THE ROAD DRIVEABLE – DISTANCE TO STOP

Velocity

50 km/h

70 km/h

100 km/h

130 km/h

Deceleration

2 m/s2 54 m 103 m 205 m 342 m

10 m/s2 23 m 38 m 66 m 101 m

0

100

200

300

2 4 6 8 10

Ove

rall

Stop

ping

Dis

tanc

e in

m

Deceleration in m/s2

Geschwindigkeit in km/h

50

70

130

100

t1=0,3s r0=10ms-2 / 0,7s

38 Volkswagen AG | Konzernforschung

Sensors G

atew

ay

Vehicle

zFAS - modular & scalable architecture

CAN

/ F

LEXR

AY /

ETH

ERN

ET /

…

APPLICATION FOR SERIES PRODUCTS

39 Volkswagen AG | Konzernforschung

Today: Release testing of todays ADAS with up to 2 million test km and 1.000 test drivers

data basis: 2.000.000 km, > 1.000 drivers; source: Dr. Markus Fach et al., Daimler AG, VDI/VW Gemeinschafts-Tagung, 2010)

Tomorrow: Increasing system complexity of future ADAS will increase diversity of relevant test scenarios Forecast for high automation: 100 Million km = 0,67 x average distance sun earth = 5,6 light minutes Costs for such release testing: several 100 Million EUR source: Prof. Winner et al., Darmstädter Kolloquium „Mensch und Fahrzeug“, 2011

Objective: Sustainable and affordable concept for test and release of future ADAS

System Activation frequency per 10.000 km km till activation Distance Warning 40 - 60 170 – 250

Breaking Assistance (BAS) Plus 0,5 - 1 10.000 – 20.000

PRE-SAFE breaking, level 1 0,1 – 0,2 50.000 – 100.000

PRE-SAFE breaking, level 2 0 -

Ensuring reliability – challenges

40 Volkswagen AG | Konzernforschung

Extension Function range /

Security requirements

SW Module- Tests

SIL- Tests

HIL- Tests

Veh. Tests

Test levels/ Function range

Com

plex

ity

SW Module-Tests

SIL-Tests

HIL-Tests

Veh. Tests

The effort of vehicle tests rises disproportional with increasing functions range/ safety requirements

Reduction of the vehicle tests effort, through shifting the test from the street to the simulation. => Virtual Test Drive virtual

test drive

Test levels and complexity

41 Volkswagen AG | Konzernforschung

vehicle model

sensor models

virtual camera environment model video Inter- face

image processing

sensor data fusions

application SW - longit. control

- lateral control - path planner

vehicle SW

Simulation via Virtual Test Drive

42 Volkswagen AG | Konzernforschung

CONCLUSION

Knowledge about the location of the road is a key factor for automated driving and future driver assistance systems

Interpretation based approaches using environment sensors work well in easy to medium challenging scenarios

Using additional map information leads to more robust results

Localization techniques are currently used to solve the most complex scenarios

Drivability estimation at long ranges for high speed driving is still challenging

New sensor principles and machine learning approaches are one way for solving this topic

Testing environment perception is one of the key challenges

Shifting tests from street to simulation reduces vehicle test efforts significantly

Centralized ECUs like the zFAS help facilitating testing procedures

THANK YOU FOR YOUR ATTENTION