![Page 1: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/1.jpg)
Perspective Multiscale Detection and
Tracking of Persons
11
Tracking of Persons
Marcos Nieto, Juan Diego Ortega, Andoni Cortés, and Seán Gaines
MMM 2014 – The 20th Anniversary International Conference on
Multimedia Modeling, Dublin (Ireland), 6,7,8-10th January 2014
![Page 2: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/2.jpg)
1. Motivation
2. Perspective calibration
3. Approach
4. Results
Outline
22
5. Conclusions
![Page 3: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/3.jpg)
Outline
1. Motivation
1. Object detection in images
2. Real-time application
3. Contextual information
2. Perspective calibration
33
2. Perspective calibration
3. Approach
4. Results
5. Conclusions
![Page 4: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/4.jpg)
Motivation
• Object detection in images
Detection-by-classification
Supervised learning
Feature extraction
Binary or multiclass
Multiscale detection
Sliding window
Spans position & size
Bounding boxes
44
Open Open
Close Close
![Page 5: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/5.jpg)
• Real-time applications
Motivation
Multiscale detection
Kind of brute-force
Too many evaluations
Some are absurd given the context
0
10
20
30
40
50
60
70
80
90
100
1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61
Nu
m.
Ev
alu
ati
on
s
Th
.
Levels
1,02
1,05
1,1
55
Parameters
Initial (smallest) size
Number of scales
Factor between scales
Offset (stride)
…
Therefore, some
knowledge about the
scene must be provided
![Page 6: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/6.jpg)
• Contextual information
Motivation
Color, motion, depth
Low generality
Particular to each application
Perspective of the scene
High generality
Allows to maintain multiscale technique
Applicable in real-time
Two assumptions
There is a dominant ground plane
66
There is a dominant ground plane
Objects lie on the plane, and their 3D size is app. known
Surveillance, ADAS
Vehicles, persons
![Page 7: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/7.jpg)
Outline
1. Motivation
2. Perspective Calibration
1. Plane view calibration
2. GUI
3. Projection of objects
77
3. Projection of objects
3. Approach
4. Results
5. Conclusions
![Page 8: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/8.jpg)
Perspective calibration
• Plane view calibration
Homography calculation
4-points
2 metric references
Extrinsics from Homography
Rotation and translation of
camera
88
1 DoF Camera model
Focal length from homography
Refinement using Lev.-Marq.
![Page 9: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/9.jpg)
Perspective calibration
GUI
Useful to calibrate videos
Quick (2-5 minutes)
Also lens distortion
correction
99
![Page 10: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/10.jpg)
Perspective calibration
• Projection of objects
Farthest size of object
1010
Closest size of object
![Page 11: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/11.jpg)
Outline
1. Motivation
2. Perspective Calibration
3. Approach
1. Overview
2. Perspective Multiscale
1111
2. Perspective Multiscale
3. Perspective Grid
4. Results
5. Conclusions
![Page 12: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/12.jpg)
• Define the perspective of the scene
• Define the 3D size of the object to search
Approach
Camera calibration
Intrinsic parameters
Camera pose
Extrinsic parameters
Homography
calibration
1212
• Define the 3D size of the object to search
• A) Calculate the best parameters for multiscale
• B) Define a fixed grid of positions in the plane
Persons
1700 x 500 x 500Car
1500 x 1700 x 3500
![Page 13: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/13.jpg)
• A) Perspective multiscale
• Rescale original
image so model size
fits farthest object
• Compute scale
factor so that model
size coincides with
Approach
Multiscale Perspective Multiscale
1313
size coincides with
closest object at the
smallest image
• Filter out invalid
positions
Focused effort: less
number of levels are
required
![Page 14: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/14.jpg)
• It is still necessary to filter out invalid positions-sizes
• The advantage of using this approach is that traditional multiscale
implementations can still be used with much less number of levels
Approach
60
70
80
90
100
Nu
m.
Ev
alu
ati
on
s
Th
.
1,02
1414
0
10
20
30
40
50
1 6 11 16 21 26 31 36 41 46 51 56 61
Nu
m.
Ev
alu
ati
on
s
Levels
1,02
1,05
1,1
Focused effort: less number
of levels are required
(typically 3 to 5)
![Page 15: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/15.jpg)
• B) Grid of fixed positions
• Predefine feasible
locations of objects
• No need to filter
• Can not be used in
multiscale
Approach
1515
Can not be used in
multiscale
implementations.
One evaluation per
candidate
Much more focused
effortBounding boxesProjected boxes
![Page 16: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/16.jpg)
Outline
1. Motivation
2. Perspective Calibration
3. Approach
4. Results
1. Case study: person detection
1616
1. Case study: person detection
2. Case study: vehicle detection
5. Conclusions
![Page 17: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/17.jpg)
• Case study: Person detection
– Full-body and Head & Shoulder SVM-HOG detector
– Perspective Multiscale
– Linear multiobject tracking
– Active Vision Group dataset (1920x1080, 4500 frames, 71460 persons
labeled)
Results
1717
labeled)
![Page 18: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/18.jpg)
Results
0,998
1
• Performance
– Reduction from
144880 to 46226
(68%) for similar
performance
Using 3 levels is
Multiscale Perspective Multiscale
1818
0,978
0,98
0,982
0,984
0,986
0,988
0,99
0,992
0,994
0,996
0,998
-0,1 6E-16 0,1 0,2 0,3 0,4 0,5 0,6
Pre
cisi
on
Recall
FB
FBUB
FBUB*
DAFFiltering
TrackingLess FN
Less FP but also
some
missdetections
L=3, 5, 7
– Using 3 levels is
enough because
perspective effect is
soft
![Page 19: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/19.jpg)
• Case study: Vehicle detection
– Vehicle detection application for embedded vision system
– Road can be assumed as planar in the short distance
– Ground truth sequence 2 minutes
– Grid of fixed positions
Results
1919
![Page 20: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/20.jpg)
• Case study: Vehicle detection
– Detections are sparse and noisy
– Tracking is still necessary
Results
2020
![Page 21: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/21.jpg)
Results
•1000x less evaluations
•7x speed in PC
•Same TP
•5 times less FP
2121
![Page 22: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/22.jpg)
Results
Type Processor RAM CPU OS Language
PC Intel Core
i5
8 GB 3.0 GHz Windows 7
Ubuntu 12.04
C++
Embedded
HW 1
ARM
Cortex
512 MB 800 MHz Xilinx Zynq
Linux
C++
2222
30 - 40 ms in ARM Cortex30 - 40 ms in ARM Cortex
FastSlow
11 – 40 ms in PC11 – 40 ms in PC
25 fps real-time
Perspective
multiscale
Brute-force
multiscale
2 - 10 ms in PC 2 - 10 ms in PC
![Page 23: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/23.jpg)
Conclusions
• Perspective is a contextual information available in many situations
• Assumptions: dominant ground plane and known object size
• Its computation is easy (K, R, t) using homographies
• It can be used for object detection to focus computational Twoways of applying it
2323
ways of applying it
• A) Perspective Multiscale: Wrapping multiscale function (~60% reduction in typical surveillance scene)
• B) Grid of fixed positions: for even more reduction of complexity (x7 speed up in low perspective scenes like onboardvehicle detection)
![Page 25: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/25.jpg)
2525
![Page 26: Perspective Multiscale Detection and Tracking of Persons](https://reader033.vdocument.in/reader033/viewer/2022052602/559b6ef91a28ab4b778b45da/html5/thumbnails/26.jpg)
2626
Offline process
Online process