a database of human segmented natural images and two applications david martin, charless fowlkes,...
Post on 20-Dec-2015
217 views
TRANSCRIPT
![Page 1: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/1.jpg)
A Database of Human Segmented Natural Images
and Two Applications
David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik
UC Berkeley{dmartin,fowlkes,doron,malik}@eecs.berkeley.edu
![Page 2: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/2.jpg)
David Martin - UC Berkeley - ICCV 2001 2
Motivation
• Berkeley Segmentation Dataset Groundtruth for image segmentation of natural images
• App#1: A segmentation benchmark• App#2: Ecological statistics
![Page 3: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/3.jpg)
David Martin - UC Berkeley - ICCV 2001 3
Benchmark Example for Recognition
MNIST handwritten digit dataset [LeCun, AT&T]http://www.research.att.com/~yann/exdb/mnist/index.html
METHOD ERROR (%)Boosted LeNet-4, [distortions] 0.7Virtual SVM deg 9 poly [distortions] 0.8LeNet-5, [distortions] 0.8LeNet-5, [huge distortions] 0.85LeNet-5, [no distortions] 0.95Reduced Set SVM deg 5 polynomial 1K-NN, Tangent Distance, 16x16 1.1SVM deg 4 polynomial 1.1LeNet-4 1.1LeNet-4 with K-NN instead of last layer 1.1LeNet-4 with local learning instead of ll 1.12-layer NN, 300 HU, [deskewing] 1.6LeNet-1 [with 16x16 input] 1.7K-nearest-neighbors, Euclidean, deskewed 2.4
Training set, test set, evaluation methodology, algorithm ranking
![Page 4: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/4.jpg)
David Martin - UC Berkeley - ICCV 2001 4
The Image Dataset
• 1000 Corel images– Photographs of outdoor scenes– Texture is common– Large variety of subject matter– 481 x 321 x 24b
![Page 5: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/5.jpg)
David Martin - UC Berkeley - ICCV 2001 5
![Page 6: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/6.jpg)
David Martin - UC Berkeley - ICCV 2001 6
![Page 7: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/7.jpg)
David Martin - UC Berkeley - ICCV 2001 7
![Page 8: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/8.jpg)
David Martin - UC Berkeley - ICCV 2001 8
![Page 9: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/9.jpg)
David Martin - UC Berkeley - ICCV 2001 9
Establishing Groundtruth• Def: Segmentation
= Partition of image pixels into exclusive sets
• Manual segmentation by human subjects– Custom Java tool to facilitate task
• Currently: 1000 images, 5500 segmentations, 20 subjects
• Naïve subjects (UCB undergrads) given simple, non-technical instructions
![Page 10: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/10.jpg)
David Martin - UC Berkeley - ICCV 2001 10
Directions to Image Segmentors
• You will be presented a photographic image• Divide the image into some number of
segments, where the segments represent “things” or “parts of things” in the scene
• The number of segments is up to you, as it depends on the image. Something between 2 and 30 is likely to be appropriate.
• It is important that all of the segments have approximately equal importance.
![Page 11: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/11.jpg)
David Martin - UC Berkeley - ICCV 2001 11
![Page 12: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/12.jpg)
David Martin - UC Berkeley - ICCV 2001 12
![Page 13: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/13.jpg)
David Martin - UC Berkeley - ICCV 2001 13
• The segmentations are not identical.
• But are they consistent??
![Page 14: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/14.jpg)
David Martin - UC Berkeley - ICCV 2001 14
Perceptual organization
forms a hierarchyimage
background left bird right bird
grass bush
headeye
beakfar
body headeye
beak
body
Each subject picks a slice through this hierarchy.
![Page 15: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/15.jpg)
David Martin - UC Berkeley - ICCV 2001 15
Quantifying inconsistency
S1 S2
How much is S1 a refinement of S2 at pixel ?
),(
),(\),(),(
1
2121
i
ii
pSR
pSRpSRSSLRE
![Page 16: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/16.jpg)
David Martin - UC Berkeley - ICCV 2001 16
Segmentation Error Measure
• One-way Local Refinement Error:
i
ii pSSLREpSSLREn
SSSE ),,(),,,(min1
),( 122121
• Segmentation Error allows refinement in either direction at each pixel:
),(
),(\),(),(
1
2121
i
ii
pSR
pSRpSRSSLRE
![Page 17: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/17.jpg)
David Martin - UC Berkeley - ICCV 2001 17
Human segmentations are consistent
SE (Color Human Segmentations)
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Segmentation Error (SE)
Same Image
Different Images
Distribution of segmentation error over the dataset.
![Page 18: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/18.jpg)
David Martin - UC Berkeley - ICCV 2001 18
Color Gray InvNeg
![Page 19: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/19.jpg)
David Martin - UC Berkeley - ICCV 2001 19
InvNeg
![Page 20: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/20.jpg)
David Martin - UC Berkeley - ICCV 2001 20
Color Gray InvNeg
![Page 21: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/21.jpg)
David Martin - UC Berkeley - ICCV 2001 21
Gray vs. Color vs. InvNeg Segmentations
SE (gray, gray) = 0.047SE (gray, color) = 0.047
Color may affect attention, but doesn’t seem to affect perceptual organization
SE (gray, gray) = 0.047SE (gray, invneg) = 0.059
InvNeg interferes with high-level cues
(2500 gray, 2500 color,200 invneg segmentations)
![Page 22: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/22.jpg)
David Martin - UC Berkeley - ICCV 2001 22
Benchmark Methodology
• Separate training and test datasets with no images in common
• Generate computer segmentation(s) of each image in test set– Determine error of each computer
segmentation using SE measure– Algorithm scored by mean SE
• Example: – SE (human, human) = 0.05– SE (NCuts, human) = 0.22– SE (different images) = 0.30
![Page 23: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/23.jpg)
David Martin - UC Berkeley - ICCV 2001 23
Ecological Statistics of Image Segmentations
• Validating and quantifying Gestalt grouping factors [Brunswik 1953]
• Priors on region properties
• Recent work on natural image statistics:– Filter outputs [Ruderman 1994, Olshausen & Field 1996,
Yuille et. al. 1999]– Object sizes [Alvarez, Gousseau, Morel 1999]– Shape [Zhu 1999] – Contours [August & Zucker 2000, Geisler et al. 2001]
![Page 24: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/24.jpg)
David Martin - UC Berkeley - ICCV 2001 24
Relative power of cues
• Pairwise grouping cues– Proximity– Luminance similarity– Color similarity– Intervening contour– Texture similarity
![Page 25: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/25.jpg)
David Martin - UC Berkeley - ICCV 2001 25
P (Same Segment | Proximity)
![Page 26: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/26.jpg)
David Martin - UC Berkeley - ICCV 2001 26
P (Same Segment | Luminance)
![Page 27: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/27.jpg)
David Martin - UC Berkeley - ICCV 2001 27
Bayes Risk for Proximity Cue
![Page 28: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/28.jpg)
David Martin - UC Berkeley - ICCV 2001 28
Bayes Risk for Various Cues Conditioned on Proximity
![Page 29: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/29.jpg)
David Martin - UC Berkeley - ICCV 2001 29
Mutual Information for Various Cues Conditioned on Proximity
![Page 30: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/30.jpg)
David Martin - UC Berkeley - ICCV 2001 30
Priors on Region Properties
• Area• Convexity
![Page 31: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/31.jpg)
David Martin - UC Berkeley - ICCV 2001 31
Empirical Distribution of Region Area
y = Kx-
= 0.913
Compare with Alvarez, Gousseau, Morel 1999.
![Page 32: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/32.jpg)
David Martin - UC Berkeley - ICCV 2001 32
Empirical Distribution of Region Convexity
![Page 33: A Database of Human Segmented Natural Images and Two Applications David Martin, Charless Fowlkes, Doron Tal, Jitendra Malik UC Berkeley {dmartin,fowlkes,doron,malik}@eecs.berkeley.edu](https://reader030.vdocument.in/reader030/viewer/2022032704/56649d545503460f94a30765/html5/thumbnails/33.jpg)
David Martin - UC Berkeley - ICCV 2001 33
Conclusion
• Large new database of segmentations of natural images by humans
• A segmentation benchmark• Ecological statistics
– Relative power of grouping cues– Priors on region properties
http://www.cs.berkeley.edu/~dmartin/segbench