![Page 1: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/1.jpg)
The PASCAL Visual Object Classes Challenge 2012 (VOC2012)
Part I – Detection Challenge
Mark Everingham, Luc Van Gool Chris Williams, John Winn
Andrew Zisserman Yusuf Aytar, Ali Eslami
![Page 2: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/2.jpg)
Detection challenge
§ Predict the bounding boxes of all objects of a given class in an image (if any)
§ Competition 3: Train on the supplied data § Which methods perform best given specified training
data?
§ Competition 4: Train on any (non-test) data § How well do state-of-the-art methods perform on
these problems?
![Page 3: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/3.jpg)
Examples
Aeroplane
Bus
Bicycle Bird Boat Bottle
Car Cat Chair Cow
![Page 4: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/4.jpg)
Examples
Dining Table
Potted Plant
Dog Horse Motorbike Person
Sheep Sofa Train TV/Monitor
![Page 5: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/5.jpg)
Annotation
§ Complete annotation of objects from 20 categories
Truncated Object extends beyond BB
Occluded Object is significantly occluded within BB
Pose Facing left
Difficult Not scored in evaluation
![Page 6: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/6.jpg)
Evaluating bounding boxes
§ Area of overlap (AO) measure
§ Need to define a threshold t such that AO(Bgt,Bp) implies a correct detection: 50%
![Page 7: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/7.jpg)
Dataset statistics
§ Same size as VOC2011.
§ Minimum ~600 training objects per category § ~2,000 cars, 1,500 dogs, 8,500 people § Approximately equal distribution across training
and test datasets
Training Testing Images 11,540 10,994 Objects 27,450 27,078
![Page 8: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/8.jpg)
Submitted methods
§ 8 methods, 7 groups § VOC2011: 13 methods, 15 groups
§ Common approach: § Deformable Part Model (Girshick, Felzenszwalb,
McAllester) with variations, e.g. § HOG-LBP features § Colour features § Multiple kernel learning
§ New approaches: § Selective search (UVA, NEC_STANFORD) § Dynamic AND-OR tree
![Page 9: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/9.jpg)
Average precision by class
0
10
20
30
40
50
60
70Av
erag
e Pr
ecis
ion
aerop
lane
motorbi
ke bus
bicycl
e cat
train
horse
tvmon
itor
car
perso
ndo
gsh
eep
sofa
dining
table co
wbo
ttle bird
boat
potte
dplan
tch
air
maxmedian
![Page 10: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/10.jpg)
Improvement over VOC2011
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Aver
age
Prec
isio
n
aerop
lanebic
ycle
bird
boatbo
ttle bus ca
r catch
air cow
dining
table do
gho
rse
motorbi
kepe
rson
potte
dplan
tsh
eep
sofa
train
tvmon
itor
20112012
![Page 11: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/11.jpg)
AP by class and method
aeroplan
e
bicycle
bird
boat
bo/le
bus
car
cat
chair
cow
diningtable
dog
horse
motorbike
person
po/ed
plan
t
sheep
sofa
train
tvmon
itor
CVC_BOW_COLOR_HOG 45.4 49.8 15.7 16.0 26.3 54.6 44.8 35.1 16.8 31.3 23.6 26.0 45.6 49.6 42.2 14.5 30.5 28.5 45.7 40.0 MISSOURI_HOGLBP_MDPM_CONTEXT 51.4 53.7 18.3 15.6 31.6 56.5 47.1 38.6 19.5 32.0 22.1 25.0 50.3 51.9 44.9 11.9 37.7 30.6 50.8 39.3 NEC_STANFORD_OCP 65.1 46.8 25.0 24.6 16.0 51.0 44.9 51.5 13.0 26.6 31.0 40.2 39.7 51.5 32.8 12.6 35.7 33.5 48.0 44.8 OLB_FT_DPM_R5 47.5 51.7 14.2 12.6 27.3 51.8 44.2 25.3 17.8 30.2 18.1 16.9 46.9 50.9 43.0 9.5 31.2 23.6 44.3 22.1 SYSU_DYNAMIC_AND_OR_TREE 50.1 47.0 7.9 3.8 24.8 47.2 42.8 31.2 17.5 24.2 10.0 21.3 43.5 46.4 37.5 7.9 26.4 21.5 43.1 36.7 UOC_OXFORD_DPM_MKL 59.6 54.5 21.9 21.6 32.1 52.5 49.3 40.8 19.1 35.2 28.9 37.2 50.9 49.9 46.1 15.6 39.3 35.6 48.9 42.8 UVA_DETECTOR_MERGING 47.2 50.2 18.3 21.4 25.2 53.3 46.3 46.3 17.5 27.8 30.3 35 41.6 52.1 43.2 18 35.2 31.1 45.4 44.4 UVA_HYBRID_CODING_APE 61.8 52 24.6 24.8 20.2 57.1 44.5 53.6 17.4 33 38.3 42.8 48.8 59.4 35.7 22.8 40.3 39.5 51.1 49.5
![Page 12: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/12.jpg)
Precision/recall curves (aeroplane)
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Prec
isio
n
NEC_STANFORD_OCP (65.0)UVA_HYBRID_CODING_APE (61.8)UOC_OXFORD_DPM_MKL (59.6)MISSOURI_HOGLBP_MDPM_CONTEXT (51.4)SYSU_DYNAMIC_AND_OR_TREE (50.1)OLB_FT_DPM_R5 (47.5)UVA_DETECTOR_MERGING (47.2)CVC_BOW_COLOR_HOG (45.4)
![Page 13: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/13.jpg)
Precision/recall curves (bicycle)
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Prec
isio
n
UOC_OXFORD_DPM_MKL (54.5)MISSOURI_HOGLBP_MDPM_CONTEXT (53.6)UVA_HYBRID_CODING_APE (52.0)OLB_FT_DPM_R5 (51.6)UVA_DETECTOR_MERGING (50.2)CVC_BOW_COLOR_HOG (49.8)SYSU_DYNAMIC_AND_OR_TREE (47.0)NEC_STANFORD_OCP (46.8)
![Page 14: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/14.jpg)
Precision/recall curves (person)
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Prec
isio
n
UOC_OXFORD_DPM_MKL (46.1)MISSOURI_HOGLBP_MDPM_CONTEXT (44.9)UVA_DETECTOR_MERGING (43.2)OLB_FT_DPM_R5 (43.0)CVC_BOW_COLOR_HOG (42.2)SYSU_DYNAMIC_AND_OR_TREE (37.5)UVA_HYBRID_CODING_APE (35.7)NEC_STANFORD_OCP (32.8)
![Page 15: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/15.jpg)
Precision/recall curves (bottle)
0 0.2 0.4 0.6 0.8 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Recall
Prec
isio
n
UOC_OXFORD_DPM_MKL (32.1)MISSOURI_HOGLBP_MDPM_CONTEXT (31.6)OLB_FT_DPM_R5 (27.3)CVC_BOW_COLOR_HOG (26.3)UVA_DETECTOR_MERGING (25.2)SYSU_DYNAMIC_AND_OR_TREE (24.8)UVA_HYBRID_CODING_APE (20.2)NEC_STANFORD_OCP (16.0)
![Page 16: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/16.jpg)
Median average precision by method
0
10
20
30
40
50
UVA_HYBRID_C
ODING_APE
OXFORD_DPM_M
KL
UVA_DETECTOR_M
ERGING
MISSOURI_HOGLB
P_MDPM_C
ONTEXT
NEC_STANFORD_O
CP
CVC_BOW_C
OLOR_H
OG
OLB_F
T_DPM_R
5
Aver
age
Prec
isio
n
![Page 17: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/17.jpg)
Prizes § Winner
§ UVA_HYBRID_CODING_APE Koen E. A. van de Sande, Jasper R. R. Uijlings, Cees G. M. Snoek, Arnold W. M. Smeulders University of Amsterdam
§ Honourable mention § OXFORD_DPM_MKL
Ross Girshick, Andrea Vedaldi, Karen Simonyan University of Oxford
![Page 18: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/18.jpg)
The PASCAL Visual Object Classes Challenge 2012 (VOC2012)
Part I – Detection Ranking Uncertainty
Mark Everingham, Luc Van Gool Chris Williams, John Winn
Andrew Zisserman Yusuf Aytar, Ali Eslami
![Page 19: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/19.jpg)
Ranking uncertainty
§ Only one AP curve per class and method § However, we can use bootstrap resampling to
obtain multiple AP curves (see e.g. blog post by Brendan O’Connor, 2010)
§ Compare AP or rank of two methods A and B § Can obtain a confidence interval for AP § If rank(A) < rank(B) with high probability
then A is significantly different from B
![Page 20: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/20.jpg)
Ranking uncertainty
for each replication 1. sample a subset of the test images 2. compute AP of each submission on sample 3. compute rank of each submission based on APs for each pair m1 and m2
1. m1 and m2 equivalent if rank of one method is not higher than the rank of the other in at least in 95% of replications
![Page 21: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/21.jpg)
Equivalencies by class and method
aeroplan
e
bicycle
bird
boat
bo/le
bus
car
cat
chair
cow
diningtable
dog
horse
motorbike
person
po/ed
plan
t
sheep
sofa
train
tvmon
itor
CVC_BOW_COLOR_HOG 45.4 49.8 15.7 16.0 26.3 54.6 44.8 35.1 16.8 31.3 23.6 26.0 45.6 49.6 42.2 14.5 30.5 28.5 45.7 40.0 MISSOURI_HOGLBP_MDPM_CONTEXT 51.4 53.7 18.3 15.6 31.6 56.5 47.1 38.6 19.5 32.0 22.1 25.0 50.3 51.9 44.9 11.9 37.7 30.6 50.8 39.3 NEC_STANFORD_OCP 65.1 46.8 25.0 24.6 16.0 51.0 44.9 51.5 13.0 26.6 31.0 40.2 39.7 51.5 32.8 12.6 35.7 33.5 48.0 44.8 OLB_FT_DPM_R5 47.5 51.7 14.2 12.6 27.3 51.8 44.2 25.3 17.8 30.2 18.1 16.9 46.9 50.9 43.0 9.5 31.2 23.6 44.3 22.1 SYSU_DYNAMIC_AND_OR_TREE 50.1 47.0 7.9 3.8 24.8 47.2 42.8 31.2 17.5 24.2 10.0 21.3 43.5 46.4 37.5 7.9 26.4 21.5 43.1 36.7 UOC_OXFORD_DPM_MKL 59.6 54.5 21.9 21.6 32.1 52.5 49.3 40.8 19.1 35.2 28.9 37.2 50.9 49.9 46.1 15.6 39.3 35.6 48.9 42.8 UVA_DETECTOR_MERGING 47.2 50.2 18.3 21.4 25.2 53.3 46.3 46.3 17.5 27.8 30.3 35 41.6 52.1 43.2 18 35.2 31.1 45.4 44.4 UVA_HYBRID_CODING_APE 61.8 52 24.6 24.8 20.2 57.1 44.5 53.6 17.4 33 38.3 42.8 48.8 59.4 35.7 22.8 40.3 39.5 51.1 49.5
![Page 22: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/22.jpg)
Equivalencies by class and method
0
5
10
NEC_STANFORD_O
CP > UVA_H
YBRID_CODING_A
PE
UVA_HYBRID_C
ODING_APE >
NEC_STANFORD_O
CP
Difference is statistically significant
![Page 23: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/23.jpg)
Equivalencies by class and method
0
5
10
NEC_STANFORD_O
CP > UVA_H
YBRID_CODING_A
PE
UVA_HYBRID_C
ODING_APE >
NEC_STANFORD_O
CP
Difference is not statistically significant
![Page 24: The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ...homes.cs.washington.edu/~shapiro/EE596/notes/Pascal.pdf · The PASCAL Visual Object Classes Challenge 2012 (VOC2012 ) Part](https://reader033.vdocument.in/reader033/viewer/2022043002/5f7efbb0e3b55016c76308d8/html5/thumbnails/24.jpg)
Equivalencies by class and method
aeroplan
e
bicycle
bird
boat
bo/le
bus
car
cat
chair
cow
diningtable
dog
horse
motorbike
person
po/ed
plan
t
sheep
sofa
train
tvmon
itor
CVC_BOW_COLOR_HOG 45.4 49.8 15.7 16.0 26.3 54.6 44.8 35.1 16.8 31.3 23.6 26.0 45.6 49.6 42.2 14.5 30.5 28.5 45.7 40.0 MISSOURI_HOGLBP_MDPM_CONTEXT 51.4 53.7 18.3 15.6 31.6 56.5 47.1 38.6 19.5 32.0 22.1 25.0 50.3 51.9 44.9 11.9 37.7 30.6 50.8 39.3 NEC_STANFORD_OCP 65.1 46.8 25.0 24.6 16.0 51.0 44.9 51.5 13.0 26.6 31.0 40.2 39.7 51.5 32.8 12.6 35.7 33.5 48.0 44.8 OLB_FT_DPM_R5 47.5 51.7 14.2 12.6 27.3 51.8 44.2 25.3 17.8 30.2 18.1 16.9 46.9 50.9 43.0 9.5 31.2 23.6 44.3 22.1 SYSU_DYNAMIC_AND_OR_TREE 50.1 47.0 7.9 3.8 24.8 47.2 42.8 31.2 17.5 24.2 10.0 21.3 43.5 46.4 37.5 7.9 26.4 21.5 43.1 36.7 UOC_OXFORD_DPM_MKL 59.6 54.5 21.9 21.6 32.1 52.5 49.3 40.8 19.1 35.2 28.9 37.2 50.9 49.9 46.1 15.6 39.3 35.6 48.9 42.8 UVA_DETECTOR_MERGING 47.2 50.2 18.3 21.4 25.2 53.3 46.3 46.3 17.5 27.8 30.3 35 41.6 52.1 43.2 18 35.2 31.1 45.4 44.4 UVA_HYBRID_CODING_APE 61.8 52 24.6 24.8 20.2 57.1 44.5 53.6 17.4 33 38.3 42.8 48.8 59.4 35.7 22.8 40.3 39.5 51.1 49.5