active pedestrian protection system, project review - …web.yonsei.ac.kr/hgjung/ho gi jung...

17th ITS World Congress, Busan 2010 MANDO Corp. Yonsei Univ.

T_AP00966

Active Pedestrian Protection System, Project ReviewActive Pedestrian Protection System, Project Review

2010. 10. 29

Ho Gi Jung

The School of Electronic and Electrical Engineering

Yonsei University, South Korea


Project OverviewProject Overview

Objectives

To develop active pedestrian protection system (APPS),detecting pedestrian using sensor fusion-based,assessing risk of vehicle-pedestrian collision,and actuating countermeasures to avoid the collision.

Duration

1st Phase (Prototype development): Nov. 2006 – Oct. 2009 (36 months)2nd Phase (Product development): Nov. 2009 – Oct. 2011 (24 months)

Fund

Co-funded by participating companies and government, MKE (Ministry of Knowledge and Economy)For the 1st phase, total budget is 4.7 million USD.


Team OrganizationTeam Organization

MANDO

Project management

System design

Vision-based pedestrian classification

Sensor fusion of NIR vision and range sensor

Sensor fusion of FIR stereo and visible vision

Risk assessment

Actuation

LG Innotek

TOF camera module

TOF camera-based pedestrian recognition

SL

NIR camera module

NIR headlamp

Yonsei Univ.

Radar array-based pedestrian recognition

VisLab

System design

Sensor fusion of NIR vision and range sensor

Ho Gi Jung


ScenarioScenario--Driven MethodDriven Method

Almighty classifier is replaced with a combination of strict experts.

• Alberto Broggi, Pietro Cerri, Stefano Ghidoni, Paolo Grisleri, and Ho Gi Jung, “A New Approach to Urban Pedestrian Detection for Automatic Braking,” IEEE Transactions on Intelligent Transportation Systems, Vol. 10, Issue 4, Dec. 2009, pp. 594-605.

• Alberto Broggi, Pietro Cerri, Stefano Ghidoni, Paolo Grisleri, Ho Gi Jung, “Localization and Analysis of Critical Areas in Urban Scenarios”, 2008 IEEE Intelligent Vehicle Symposium, Eindhoven University of Technology, Eindhoven, The Netherlands, June 4-6, 2008, pp. 1074-1079.

• Alberto Broggi, Pietro Cerri, Luca Gatti, Paolo Grisleri, Ho Gi Jung, JunHee Lee, “Scenario-Driven Search for Pedestrians Aimed at Triggering Non-Reversible Systems,” 2009 IEEE Intelligent Vehicle Symposium, 2009 IEEE Intelligent Vehicle Symposium, June 3-5, 2009, Xi’an, Shaanxi, China, pp. 285-291.

False negative False positive


Critical AreaCritical Area--Centered Pedestrian RecognitionCentered Pedestrian Recognition

The top ranked situation is when a pedestrian popped up from the behind of vehicle parked along a road-side.

We assume that pedestrians in front of ego-vehicle without occlusion would be easily detected by the driver.

Examples of critical area. The second row shows the critical area of situations of the first row.

““Critical areaCritical area””


Comparison of Four Sensing MethodsComparison of Four Sensing Methods

Four sensing methods were investigated and compared:1) Sensor fusion of NIR vision and range sensor (scanning laser radar)2) TOF (Time Of Flight) camera-based3) Range sensor array-based (mm-wave radar)4) Sensor fusion of FIR stereo and visible vision

Three Criteria

1) Critical area localization performance2) Latency of popping up pedestrian detection3) System cost and possibility of domestic series production

Sensor fusion of NIR vision and range sensor was selected for the next phase.


Sensor Fusion of NIR Vision and Range SensorSensor Fusion of NIR Vision and Range Sensor

• Pietro Cerri, Luca Gatti, Luca Mazzei, Fabio Pigoni, Ho Gi Jung, “Day and Night Pedestrian Detection Using Cascade AdaBoostSystem,” the 13th IEEE International Conference on Intelligent Transportation Systems, Maderia Island, Portugal, 19-22 Sep. 2010, accepted on 2 Jul. 2010.

Vision-based pedestrian classifier

- Haar-feature-based Adaboost

- Ad-hoc-features-based Adaboost


(a) General headlamp


HDRC Camera Module with NIR Sensitivity

NIR Lamp with horizontally wider FOV

(b) Newly developed headlamp

Bi-functional module NIR lamp


Detected critical area Result of vision-based pedestrian verification



TOF CameraTOF Camera--basedbased

Although prototype was made, this approach was rejected because of several reasons:1) Light power in the near area does not satisfy US FDA eye-safe regulation.2) Required power for the modulating light source is too high.3) Too large size and cooling method.


Radar Sensor ArrayRadar Sensor Array--basedbased

• Seongkeun Park, Jae Pil Hwang, Euntai Kim, Heejin Lee, and Ho Gi Jung, “A neural network approach to target classification for active safety system using microwave radar,” Expert Systems with Applications, Vol. 37, Issue 3, 15 March 2010, pp. 2340-2346.

1) Vehicle corner localization is instable because radar is too sensitive to the vehicle’s pose and contains relatively large variation.

2) Filtering algorithm embedded into the radar prevents rapid detection of popping up pedestrian and it causes significant latency.

Although we developed radar signal-based object classification and tracking, this approach was rejected because it can not address popping up pedestrian scenarios.


Sensor Fusion of FIR Stereo and Visible VisionSensor Fusion of FIR Stereo and Visible Vision

Input stereo images Segmentation result Stereo matching and object detection

HOG-based feature extraction SVM-based classification

Although we recognized this approach was robust to day and night situations, it was rejected because of system price and domestic component sourcing.


Additional Results about Pedestrian ClassifierAdditional Results about Pedestrian Classifier

Sub-window Splitting• In order to extract local characteristics of pedestrian images

36x18 image 18x9 nine overlapped sub-images

• 24 Gabor filters (6 sections in orientation, 4 sections in scale)

• 3 statistical values of filtered result: mean, variance, skewness

• Feature vector dimension: 9x24x3=648

• Ho Gi Jung, Jaihie Kim, “Constructing a pedestrian recognition system with a public open database, without the necessity of re-training: an experimental study,“ Pattern Analysis and Applications, Vol. 13, No.2, May 2010, pp. 223-233.

Gabor Filter Bank (GFB)-based Feature


Sub-window Splitting

Filtering Extraction with Gabor Filter Bank

Candidate Image

Optimization of SVM Learning Parameters

SVM Learning with Training Data 2

22

1 1 1

0

1maximize ( )2

subject to 0, 0

i jn n n

i i j i ji i j

n

i i ii

W y y e

C y

σα α α

α α

−−

= = =

=

= − ⋅

≥ ≥ =

∑ ∑∑

∑

x x

α

SVM Execution with Cross-Validation Data 2

22

1( )

( | ) ( | )

iNs

i ii

error

f y e

P P non ped P ped non

σα−

−

=

= ⋅

= +

∑s x

x

GA-based Minimization of Perror

With respect to C and σ

• Kernel parameter and regulation parameter are critical

for classification performance.

Parameter Optimization is needed!

• GA is robust to non-linear and discontinuous optimization.

• Gene is composed of two parameters.

• Fitness function is defined by cross-validation.

• When Joachims’ performance estimator is used instead of

cross-validation, all learning database can be used for the

training. Consequently, it leads to improvement of

recognition performance.

Radial Basis Function (RBF)-based SVM and

Genetic Algoritm (GA)-based Optimization



• DCX pedestrian database

- 5 datasets 3 for training (1 for learning and 2 for cross validation), 2 for final test.

- One dataset: 4,800 pedestrian images (800 persons), 5,000 non-pedestrian images.

• Better performance and lower complexity

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

False Positive Rate

Det

ectio

n R

ate

Comparison of ROCs. The dotted line is ROC of LRF–quadratic SVM shown in Fig. 5(d) of [1] and the solid line is ROC of GFB–RBF SVM.

S. Munder and D. M. Gavrila, “An Experimental Study on Pedestrian Classification”, IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 28, No. 11, pp. 1863-1868, Nov. 2006.




1) GFB-RBF SVM is competent in pedestrian recognition.2) Performance estimator can replace cross-validation. In this case, dataset used for

cross-validation can be used for learning, and increases resultant performance.3) If sampling method and pre-processing are common, a pedestrian recognition system

constructed with a database can be used for actual application without re-training with the newly acquired database. In other words, publicly open database could be used for a general pedestrian recognition system.

4) A posteriori probability-based post-processing enhances recognition rate while losing a little in the false positive rate.

Four useful facts were found:


Risk AssessmentRisk Assessment

Considering the possibility of collision avoidance using braking and steering, 3 risk levels were defined.


Risk Levels and Counter MeasuresRisk Levels and Counter Measures

Typical 5-step collision scenario and countermeasures were investigated.


Counter MeasuresCounter Measures

To reduce danger of false actuation, pedestrian protection measures are activated according to a hierarchical strategy.


Lessons from PCDN StudyLessons from PCDN Study

(a) Without PCDN and AHS• Ho Gi Jung, Byung Moon Kwak, Jeong Soo Shim, Pal Joo Yoon, Jaihie Kim, “Pre-Crash Dipping Nose (PCDN) Needs Pedestrian

Recognition”, IEEE Transactions on Intelligent Transportation Systems, VOL. 9, NO. 4, Dec. 2008, pp. 678-687.

• Feasibility of pre-crash dipping nose (PCDN), developed for vehicle-vehicle collision, was investigated. PCDN showed severity increase in vehicle-pedestrian collision situation.

• Although PCDN was developed for vehicle-vehicle side crashes, there is a possibility that the range sensor for crash detection fail to distinguish a group of pedestrians from a side-faced vehicle. vehicle-vehicle collision countermeasure also needs pedestrian recognition.

(b) With PCDN and AHS


Experimental ResultsExperimental Results

• For cooperative development, two test vehicles in each site.• SICK LMS 211, NIR sensitive camera.• Active braking by MANDO’s MGH-40 ESCplus via CAN.

Parma University, Italy

MANDO, Korea



Fig. 10. Some suddenly appearing pedestrians correctly detected (a) in an underground parking, (b) in the rain, (c) behind a misaligned vehicle, (d) behind a wall, and (e) at night, and (f) a suddenly appearing pedestrian detected as a non-dangerous pedestrian (false negative).

10 hours in complex urban scenarios

236km

Various situations were included.

24 true positives

1 false positive

(2×10-6 false positive/frame)

11 false negative

(1 missing, others are alert missing or delayed detection)



In Italy

In Korea


Thank You for Your Attention!Thank You for Your Attention!

EE--mail: mail: [email protected]@yonsei.ac.kr

Homepage: http://Homepage: http://web.yonsei.ac.kr/hgjungweb.yonsei.ac.kr/hgjung

active pedestrian protection system, project review - …web.yonsei.ac.kr/hgjung/ho gi jung...

Documents