motivation and objective process pipeline · 2016-05-11 · naveen appiah, nitin bandaru department...

Obstacle detection using stereo vision for self-driving cars Naveen Appiah, Nitin Bandaru Department of Mechanical Engineering, Stanford University Motivation and Objective Process Pipeline References Obstacle detection algorithm Perceiving surroundings accurately is essential for autonomous systems like self-driving cars. Lidar systems used for perception are expensive. Replace them with inexpensive vision-based systems. The objective is to detect obstacles in the surroundings using stereo vision from 360 0 spherical images captured from vertically displaced cameras. Source: http://www.webpages.uidaho.edu/vakanski/ How%20Driverless%20Cars%20Work.html Vertically displaced 360 0 spherical image pair Optical flow Disparity estimation Depth map generation Obstacle detection Post-processing Polar map for motion planning Results 360 0 by 180 0 spherical panoramas captured using Ricoh Theta camera mounted at two different heights. Trigonometrically transform disparity values to depth values. Classifying obstacles from the ground by making use of the relative positions of points in the 3D space. Median filtering and morphological operations to fill small holes in the obstacle map. Detected obstacle points projected on a 2D plane for motion planning. 10 20 30 30 210 60 240 90 270 120 300 150 330 180 0 Original lower image: Equirectangular projection (altitude -30 0 to 30 0 and azimuth 0 0 to 360 0 ) with good mix of obstacles at various ranges. Disparity map: We observe uniform gradient of disparity along the ground plane, in the bottom half of the image. Depth map: Darker regions correspond to smaller depths. Obstacle map: Obstacles, both closer and farther away, are detected sufficiently well. Detected obstacles overlaid on the original image Polar map: The agent is at the center of the map, facing 0 0 . The blue points correspond to polar positions of the obstacle points around the agent. 1. Talukder, A., et al. "Fast and reliable obstacle detection and segmentation for cross-country navigation." Intelligent Vehicle SympoTalukder, A., et al. "Fast and reliable obstacle detection and segmentation for cross-country navigation." Intelligent Vehicle Symposium, 2002. IEEE. Vol. 2. IEEE, 2002. 2. Sun, Deqing, Stefan Roth, and Michael J. Black. "Secrets of optical flow estimation and their principles." Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010. 3. Bernini, Nicola, et al. "Real-time obstacle detection using stereo vision for autonomous ground vehicles: A survey." Intelligent Transportation Systems (ITSC), 2014 IEEE 17th International Conference on. IEEE, 2014. 4. Broggi, Alberto, et al. "Stereo obstacle detection in challenging environments: the VIAC experience." Intelligent Robots and Systems (IROS), 2011 IEEE/RSJ International Conference on. IEEE, 2011. Key idea: Obstacles are points that are at a height from the dominant ground plane. Definition: Two points P 1 and P 2 belong to the same obstacle and are said to be compatible if: and H T < P 2 Z − P 1Z < H max P 2 P 1 P 3 H max ( P 2 − P 1 ) • ( P 3 − P 1 ) P 2 − P 1 P 3 − P 1 > cosθ T Source: [1] OD algorithm: • Classify all pixels as non-obstacles • Scan through all pixels • Determine set of pixels, T P in the projected triangle on 2D image • Examine all points in T P and determine set O P of points P i in T P , compatible with P 1 • If O P is not empty, classify all points of O P as obstacle points. P 1 Vertical optical flow at every pixel is the disparity value for that pixel in the bottom image. H max /d H T /d P 2 2D image plane

Upload: others

Post on 21-Jul-2020

1 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Page 1: Motivation and Objective Process Pipeline · 2016-05-11 · Naveen Appiah, Nitin Bandaru Department of Mechanical Engineering, Stanford University Motivation and Objective Process

Obstacle detection using stereo vision for self-driving carsNaveen Appiah, Nitin Bandaru

Department of Mechanical Engineering, Stanford University

Motivation and Objective Process Pipeline

References

Obstacle detection algorithm

Ø  Perceiving surroundings accurately is essential for autonomous systems like self-driving cars.

Ø  Lidar systems used for perception are expensive. Replace them with inexpensive vision-based systems.

The objective is to detect obstacles in the surroundings using stereo vision from 3600 spherical images captured from vertically displaced cameras.

Source: http://www.webpages.uidaho.edu/vakanski/How%20Driverless%20Cars%20Work.html

Vertically displaced 3600 spherical image pair

Optical flow à Disparity estimation

Depth map generation

Obstacle detection Post-processing Polar map for motion planning

Results

3600 by 1800 spherical panoramas captured using Ricoh Theta camera mounted at two different heights.

Trigonometrically transform disparity values to depth values.

Classifying obstacles from the ground by making use of the relative positions of points in the 3D space.

Median filtering and morphological operations to fill small holes in the obstacle map.

Detected obstacle points projected on a 2D plane for motion planning.

210

240

270

120

300

150

330

180 0

Original lower image: Equirectangular projection (altitude -300 to 300 and azimuth 00 to 3600) with good mix of obstacles at various ranges.

Disparity map: We observe uniform gradient of disparity along the ground plane, in the bottom half of the image.

Depth map: Darker regions correspond to smaller depths. Obstacle map: Obstacles, both closer and farther away, are detected sufficiently well.

Detected obstacles overlaid on the original image

Polar map: The agent is at the center of the map, facing 00. The blue points correspond to polar positions of the obstacle points around the agent.

1.  Talukder, A., et al. "Fast and reliable obstacle detection and segmentation for cross-country navigation." Intelligent Vehicle SympoTalukder, A., et al. "Fast and reliable obstacle detection and segmentation for cross-country navigation." Intelligent Vehicle Symposium, 2002. IEEE. Vol. 2. IEEE, 2002.

2.  Sun, Deqing, Stefan Roth, and Michael J. Black. "Secrets of optical flow estimation and their principles." Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010.

3.  Bernini, Nicola, et al. "Real-time obstacle detection using stereo vision for autonomous ground vehicles: A survey." Intelligent Transportation Systems (ITSC), 2014 IEEE 17th International Conference on. IEEE, 2014.

4.  Broggi, Alberto, et al. "Stereo obstacle detection in challenging environments: the VIAC experience." Intelligent Robots and Systems (IROS), 2011 IEEE/RSJ International Conference on. IEEE, 2011.

Key idea: Obstacles are points that are at a height from the dominant ground plane.

Definition: Two points P1 and P2 belong to the same obstacle and are said to be compatible if: and HT < P2Z −P1Z < Hmax

Hmax

(P2 −P1)•(P3 −P1) P2 −P1 P3 −P1 > cosθT

Source: [1]

OD algorithm: •  Classify all pixels as non-obstacles •  Scan through all pixels

•  Determine set of pixels, TP in the projected triangle on 2D image •  Examine all points in TP and determine set OP of points Pi in TP,

compatible with P1 •  If OP is not empty, classify all points of OP as obstacle points. P1

Vertical optical flow at every pixel is the disparity value for that pixel in the bottom image.

Hmax/d

HT/d

2D image plane