![Page 1: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/1.jpg)
EVENT DETECTION USING AN ATTENTION-BASED TRACKER
22 Outubro 2007. Universidade Federal do Paraná.Gerald Dalley, Xiaogang Wang, and W. Eric L. Grimson
![Page 2: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/2.jpg)
2
Glasgow Airport
4 cameras
9 video clips 1 for training 8 for testing
Dataset Description
![Page 3: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/3.jpg)
3
Loitering Staying in the scene for > 1 minute
Theft Taking someone else’s baggage
Left luggage Leaving the scene without your luggage
Evaluation Correct event times Correct 3D locations Without false detections
Challenge Problem
![Page 4: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/4.jpg)
4
Basic tracking first
Detect interesting tracks
Improve interesting tracks to detect events
Our Approach
![Page 5: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/5.jpg)
5
Processing Pipeline
Tracking & Event
Detection
Input Data
Background Subtraction
BackgroundModelingIllumination
Normalization
![Page 6: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/6.jpg)
6
Illumination ChangesBA
CKG
ROU
ND
126 999
2999
S08
![Page 7: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/7.jpg)
7
Illumination Changes by Clip
0
20
40
60
80
100
120
Intra-Clip Time
Mea
n In
tens
ity
![Page 8: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/8.jpg)
8
Background Modeling
Tracking & Event
Detection
Input Data
Background Subtraction
Robust Gaussian Fit
Ground-Plane Registration
Background Model Search
NormalizedBACKGROUND
NormalizedS01-S08
BackgroundModeling
Background Models
IlluminationNormalization
![Page 9: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/9.jpg)
9
Fundamental assumption foreground is rare at every pixel
Reality for PETS 2007… background is rare for the pixels we care about
most Some pixels: foreground as much as 90% of the time
Breaking Adaptive Background Subtraction
02000
30004000
![Page 10: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/10.jpg)
10
Robust Gaussian Fit (per pixel) BACKGROUND clip
Foreground is rare everywhere
Fit a Gaussian Refit to inliers
![Page 11: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/11.jpg)
11
Need for Model Adaptation Another clip (S02)
BACKGROUND’s model: Suboptimal fit
BACKGROUND’s Gaussian modelBackground samplesForeground samples
![Page 12: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/12.jpg)
12
Model Adaptation Until convergence
Find inliers Shift Gaussian center
![Page 13: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/13.jpg)
13
Improvement Adaptation
Is robust Improves FG/BG
classification rates
Before adaptationAfter adaptation
![Page 14: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/14.jpg)
14
Background Subtraction
Tracking & Event
Detection
Input Data
Background Subtraction
BackgroundModelingIllumination
Normalization
MRF blobsBlob
Extraction
Mahalanobis Distance
Background Models
NormalizedS01-S08
![Page 15: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/15.jpg)
15
Background Subtraction
Frame 2000 With Foreground Mask Foreground Mask
Mahalanobis DistancesAdapted ModelNormalized
- =
MRF
![Page 16: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/16.jpg)
16
Tracking & Event Detection
Tracking & Event
Detection
Input Data
Background Subtraction
BackgroundModelingIllumination
Normalization
alerts
blobs Kalman Tracking
Meanshift
TrackingEvent
Detection
![Page 17: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/17.jpg)
17
Idea: Focus on tracking what we care about. Loitering humans Dropped luggage that becomes dissociated from
its owner
Kalman tracking Constant velocity Low false positive rate
Blob Tracking
![Page 18: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/18.jpg)
18
Loitering humans Remain in the scene for a long time Likely to create isolated tracks
Dispossessed luggage Likely to create at least an isolated blob detection
Detecting Humans and Luggage
Object Type
Min. Blob Area (% of
frame)
Max. Blob Area (% of
frame)Min. Blob Track
Length
humans 1.5% 3.0% 16sluggage 0.2% 1.0% 1 frame
![Page 19: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/19.jpg)
19
Blob tracking Yields high-quality tracks (good) Requires isolated blobs (bad)
Meanshift tracker Learn a model (color histogram) from the good blob tracks Tracks through occlusions
Humans Find scene entry/exit times
Luggage Find drop/pickup times Associate with human owners
Mean-Shift Tracking
![Page 20: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/20.jpg)
20
Results
![Page 21: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/21.jpg)
21
No events occur
None detected
S00 – No Defined Behavior
![Page 22: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/22.jpg)
Staged loitering 5.1s late
S01 – General Loitering 1 (Easy)
![Page 23: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/23.jpg)
23
Staged loitering 1.4s late
S02 – General Loitering 2 (Hard)
![Page 24: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/24.jpg)
24
Staged loitering 8.2s late
Dropped luggage Should not trigger an alarm No alarm triggered
Staged loitering man near purple-outlined woman Missed
Unscripted loitering Detected
S03 – Bag Swap 1 (Easy)
![Page 25: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/25.jpg)
25
S04 – Bag Swap 2 (Hard)
Stay close to each other the whole time
![Page 26: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/26.jpg)
26
S05 – Theft 1 (Easy)
Victim enters 19.2s late
Luggage stolen 0.08s late
Thief exits 0.08s late
![Page 27: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/27.jpg)
27
S06 – Theft 2 (Hard)
Thief
Luggage (never isolated)
Victims
Assistant thief
![Page 28: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/28.jpg)
28
S07 – Left Luggage 1 (Easy)
Luggage dropped 0.12s late
Owner tracked Luggage taken
0.08s late By owner
![Page 29: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/29.jpg)
29
S08 – Left Luggage 2 (Hard)
Owner drops: 0.2s late
Owner picks up: 0.12s late
![Page 30: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/30.jpg)
30
PETS 2007 Problem description, datasets, etc. http://www.pets2007.net
My Website The paper http://people.csail.mit.edu/dalleyg
Websites
![Page 31: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/31.jpg)
31
Perguntas
![Page 32: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/32.jpg)
32
Illumination Normalization~ci ;t = § ¡ 1
2t (ci ;t ¡ ¹ct);~ci ;t = § ¡ 12t (ci ;t ¡ ¹ct);
whereci ;t = thecolor of pixel i in frame t;
¹ct = 1N
NX
i=1ci ;t; and
§ t = 1N ¡ 1
NX
i=1(ci ;t ¡ ¹ct)2:
ci ;t = thecolor of pixel i in frame t;
¹ct = 1N
NX
i=1ci ;t; and
§ t = 1N ¡ 1
NX
i=1(ci ;t ¡ ¹ct)2:
![Page 33: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/33.jpg)
33
Region-of-Interest Masks
BG-S04
S06
S05
S07-S08
![Page 34: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/34.jpg)
34
Background Subtraction
Tracking & Event
Detection
Input Data
Background Subtraction
BackgroundModelingIllumination
Normalization
BG-Biased MRF
FG-Biased MRF
Blob Extractio
nfragmented blobs
merged blobs
Blob Extractio
n
Mahalanobis Distance
Background Models
NormalizedS01-S08
![Page 35: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/35.jpg)
35
Dual Background Subtraction
Low fragmentation
But blobs merged Good for human
tracking
Mahalanobis Distance Map
Background-Biased Blobs
Foreground-Biased Blobs
Sharp boundaries But fragmented blobs Good for dropped
luggage detection
![Page 36: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/36.jpg)
36
Tracking & Event Detection
Tracking & Event
Detection
Input Data
Background Subtraction
BackgroundModelingIllumination
Normalization
alerts
fragmented blobs
mergedblobs
Kalman Tracking
Kalman Tracking
Meanshift
TrackingEvent
Detection
![Page 37: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/37.jpg)
37
Luggage: it often doesn’t travel far from the owner, so we need BG-biased to
avoid merging the dropped luggage blob with the owner The floor is more boring (except for specularities), so camouflaging
doesn’t occur much there, relative to the vertical surfaces Easy to tell people fragments from luggage: small people fragments
move They move when they’re isolated They move before and after isolation
Humans With the busyness of the scene, a BG-biased MRF produces a lot of
fragments and many-to-many blob matching quickly becomes impractical
A FG-biased MRF avoids the fragmentation issue but merges lots of blobs We only care about loiterers
Loiterers are in the scene for a long time They’re likely to be isolated from other people at least at some point in time
Why Dual Trackers/Motion Blobs
![Page 38: Event Detection Using an Attention-Based Tracker](https://reader036.vdocument.in/reader036/viewer/2022062815/5681694b550346895de0e3b3/html5/thumbnails/38.jpg)
38
Oriented Ellipsoids