design and perceptual validation of performance measures for salient object segmentation
Post on 23-Jan-2016
73 Views
Preview:
DESCRIPTION
TRANSCRIPT
Design and Perceptual Validation
of Performance Measures for Salient Object Segmentation
Vida Movahedi, James H. Elder
Centre for Vision Research
York University, Canada
Evaluation of Salient Object Segmentation
Centre for Vision Research, York University2
Source: Berkeley Segmentation Dataset
Evaluation of Salient Object Segmentation
Centre for Vision Research, York University3
How do we measure success?
Existing literature Salient object segmentation
[Liu07, Zhang07, Park07, Zhuang09, Achanta09, Pirnog09, …]
Evaluation of salient object segmentation algorithms [Ge06,?]
Evaluation of segmentation algorithms [Huang95, Zhang96, Martin01, Monteiro06, Goldmann08,
Estrada09]
Centre for Vision Research, York University4
Contributions
Centre for Vision Research, York University5
Analysis of previously suggested measures
Contour Mapping Measure (CM) Order-preserving matching
A new dataset of salient objects (SOD)
Psychophysics experiments Evaluation of above measures
Matching paradigm in Precision and Recall measures
Evaluation measures in literature
Centre for Vision Research, York University6
Region-based error measures Based on false positive/ false negative pixels [Young05], [Ge06], [Goldmann08], ...
Boundary-based error measures Based on distance between boundaries [Huttenlocher93], [Monteiro06], ...
Mixed measures Based on distance of misclassified pixels to the
boundaries [Young05], [Monteiro06], ...
Region-based error measures[Young05], [Ge06], [Goldmann08], ...
Centre for Vision Research, York University7
A and B two boundaries RA the region corresponding to a boundary A and |
RA| the area of this region,
BA
BA
RR
RRBARI
1),(BA
BAB
BA
BAA
RR
RRR
RR
RRR
||||
False Negatives
False Positives
Not sensitive to shape differences
Boundary-based error measures[Huang95],[Huttenlocher93], [Monteiro06], ...
Centre for Vision Research, York University8
A and B two boundaries Distance of one point a on A from B is
Hausdorff distance:
Mean distance:
),(min)( badadBb
B
)(max),(maxmax),( bdadBAHD ABb
BAa
)(mean),(meanmean),( bdadBAMD ABb
BAa
Not sensitive to shape differences
a
Penalizing the over-detected and under-detected regions by their distances to intersection
Mixture error measures [Young05], [Monteiro06], ...
Centre for Vision Research, York University9
fpfn N
kkB
fp
N
jjA
fndiag
qdN
pdND
BAMM11
)(1
)(1
2
1),(
False Negatives
False Positives
Not sensitive to shape difference
Another example
Centre for Vision Research, York University10
Different shapes with low errors
Comparing two boundaries
Centre for Vision Research, York University11
The two boundaries need to follow each other Thus it is not sufficient to map points to the
closest point on the other boundary The ordering of mapped points must be preserved
B ASmall False
Negative Region
Small False Positive Region
The order of mapped points on the two boundaries must be monotonically non-decreasing.
Allowing for different levels of detail: One-to-one Many-to-one One-to-many
Order-preserving Mapping
nmjibaba njmi then and ,If
Centre for Vision Research, York University12
Contour Mapping Measure
Centre for Vision Research, York University13
Given two contours A=a1a2..an and B=b1b2..bm, Find the correct order-preserving mapping
Contour mapping error measure:
Average distance between matched pairs of points
Bimorphism [Tagare02]
Elastic Matching [Geiger95, Basri98, Sebastian03, ..]
A dynamic programming implementation to find the optimum mapping Closed contours point indices are assigned cyclically
Based on string correction techniques [Maes90]
Complexity: if m<n and m, n points on two boundaries
Contour Mapping Measure
Centre for Vision Research, York University14
)log( mnmO
Contour Mapping Example
Centre for Vision Research, York University15
Ground Truth Boundary
Algorithm Boundary
Matched pairs shown as line segments
CM= average length of line segments
connecting matched pairs
Contour Mapping Measure Order- preserving mapping avoids problems
experienced by other measures
Centre for Vision Research, York University16
SOD: Salient Object Dataset
Centre for Vision Research, York University17
A dataset of salient objects Based on Berkeley Segmentation Dataset
(BSD) [Martin01]
300 images 7 subjects
1
1
1
1
1
Source: Berkeley Segmentation Dataset Available in SOD
Psychophysical experiments
Centre for Vision Research, York University18
Which error measure is closer to human judgements of shape similarity?
9 subjects 5 error measures Regional Intersection (RI) Mean distance (MD) Hausdorff distance (HD) Mixed distance (MM) Contour Mapping (CM)
Psychophysical Experiments
Experiment 1 - SOD
Reference & test shapes all from SOD
Experiment 2 - ALG
Reference from SOD, test shapes algorithm-generated
Centre for Vision Research, York University19
Reference: Human
segmentation
Test cases:
Algorithm-
generated
Test cases: Human
segmentations
Reference: Human
segmentation
Agreement with Human Subjects Human subject chooses Left
or Right
An error measure M also chooses Left or Right, based on their error w.r.t. the reference shape
If M chooses the same as the human, it is a case of agreement
Human-Human consistency: defined based on agreement between human subjects
Centre for Vision Research, York University20
Left Right
Reference
Psychophysical Experiments
Centre for Vision Research, York University21
Experiment 1- SOD
Reference & tests shapes all
from SOD
Experiment 2 - ALG
Reference from SOD, test shapes algorithm-
generated
RI: region intersection, MD: mean distance, HD: Hausdorff distance, MM: mixed measure, CM: contour mapping
Precision and Recall measures
Centre for Vision Research, York University22
For algorithm boundary A and ground truth boundary B
Precision: proportion of true positives on A
Recall: proportion of detected points on B
Martin’s PR (M-PR)[Martin04] Minimum cost bipartite matching, cost proportional to
distance
Estrada’s PR (E-PR)[Estrada09] ‘No intervening contours’ and ‘Same side’ constraints
Contour Mapping PR (CM-PR) Order-preserving mapping
||
),matched(
A
BA
||
),matched(
B
AB
Matching paradigm in Precision/Recall
Centre for Vision Research, York University23
Experiment 1- SOD
Reference & test shapes all
from SOD
Experiment 2 - ALG
Reference from SOD, test shapes algorithm-
generated
Summary
Centre for Vision Research, York University24
Analysis of available measures for evaluation of salient object segmentation algorithms
A new measure- contour mapping measure (CM) Code available online: http://elderlab.yorku.ca/ContourMapping
A new dataset of salient objects Dataset available online: http://elderlab.yorku.ca/SOD
Psychophysical Experiment CM has a higher agreement with human subjects
Order-preserving matching paradigm in Precision/Recall analysis Code available online: http://elderlab.yorku.ca/ContourMapping
Centre for Vision Research, York University25
Thank You!
top related