california institute of technology - pasadena, ca 91125...
TRANSCRIPT
[1] P. Arbelaez, M. Maire, C. Fowlkes, and J. Ma-lik. Contour Detection and Hierarchical ImageSegmentation. PAMI, 2011.
[2] M. Maire, S. X. Yu, and P. Perona. Object De-tection and Segmentation from Joint Embeddingof Parts and Pixels. ICCV, 2011.
[3] D. Martin, C. Fowlkes, D. Tal, and J. Malik. ADatabase of Human Segmented Natural Imagesand its Application to Evaluating SegmentationAlgorithms and Measuring Ecological Statistics.ICCV, 2001.
[4] M. Spain and P. Perona. Measuring and Predict-ing Object Importance. IJCV, 2010.
ONR MURI N00014-10-1-0933 and ARO/JPL-NASA Stennis NAS7.03001 supported this work. Part of Stella Yu’s work was supported by NSF CAREER IIS-1257700.
Benchmarks
Image Groundtruth UCM gPb-UCM [1] Residual
• Construct groundtruth Ultrametric Contour Map (UCM):– Weight each boundary by level at which it appears in object-part hierarchy– Determine boundary visibility by region tree traversal• Does machine hierarchical segmentation [1] respect object-part containment?
– Residuals (above) show differences by:∗ Type: false positive / false negative / incorrect level∗ Severity: color intensity reflects magnitude of error
– Plots (below) measure recovery order of groundtruth boundaries:∗ Ideally recover top-level objects, then parts, then subparts (levels 1, 2, 3)∗ gPb-UCM only recovers groundtruth hierarchy on simpler scenes (portraits)
All
Sce
nes
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Overall Boundary Recall
Leve
l Rec
over
y F
ract
ion
Boundary Recovery Order by Hierarchy Level
Level 1Level 2Level 3
Port
rait
Sce
nes
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Overall Boundary Recall
Leve
l Rec
over
y F
ract
ion
Boundary Recovery Order by Hierarchy Level
Level 1Level 2Level 3
Dataset
Par
tsO
bjec
tsIm
age
• Complete labeling of 97 complex scenes photographed by artist Stephen Shore [4]• Compared to Berkeley Segmentation Dataset [3]: more objects, greater scale range
Interactive Labeling
A
BDC
• Browse object-part hierarchy by expanding/collapsing visible subtrees (A)• Drag and drop to rearrange region hierarchy (B)• Fade occlusion layers by depth to visualize figure/ground ordering (C)• Interactively edit regions (D)
– Editor enforces parent-child region containment invariant– Superpixel selection brush speeds region definition:
Image Superpixels Click Drag Release Touch-up
Scene ModelObject-Part ⇒
⇐OcclusionOrdering
man
head
shirt
arm
watch
hand
arm
envelope
stamp
label glove
Objects/Parts
Figure/G
round
Gnd
Fig
glove
envelope
stamp
label
man
head
V
shirt
arm
V
watch
hand
arm glove
• Image⇒ regions {R1, R2, ..., Rn} (Ri, Rj possibly overlapping)• Map each region Ri to a node in doubly-ordered tree:
– Parent-child node relationships correspond to object-part containment– Relative ordering of sibling nodes resolves occlusion ambiguities• Tree traversal:
– Recovers object-part hierarchy– Converts local occlusion relationships into global figure/ground ordering• Virtual nodes (dotted ovals): parts without visible boundaries• Virtual links (dotted arrows): remap occlusion ordering (handle self-occlusion)
Overview• Annotate multiple modalities:
– Objects, parts, subparts– Object-part containment– Segmentation– Occlusion (figure/ground)
• Unifying abstraction: region trees• Web-based annotation tool:
– Computer-assisted segmentation• Object segmentation dataset & benchmark• Motivation: joint detection & segmentation [2]
appleCCCCCCCCW
glassesPPPPPPPPq
hat�
face�
torso��
�����
eyes HHHHHj
PPPPPPPPPq
beardPPPPPPq
finger
?
shirt��
����
tiePPPP
PPi
jacket����������������
Fig
Gnd
Image Objects/Parts Subparts Figure/Ground Ultrametric Contour Map
Michael Maire1, Stella X. Yu2, and Pietro Perona11California Institute of Technology - Pasadena, CA 91125 2University of California at Berkeley / ICSI - Berkeley, CA 94704
Hierarchical Scene Annotation