lecture 7: segmentationgrauman/courses/378/... · –hierarchical – k-means – mean shift –...
TRANSCRIPT
![Page 1: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/1.jpg)
Lecture 7: SegmentationThursday, Sept 20
![Page 2: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/2.jpg)
Outline• Why segmentation?• Gestalt properties, fun illusions and/or
revealing examples• Clustering
– Hierarchical– K-means– Mean Shift– Graph-theoretic
• Normalized cuts
![Page 3: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/3.jpg)
![Page 4: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/4.jpg)
Grouping
• Segmentation / Grouping / Perceptual organization: gather features that belong together
• Need an intermediate representation, compact description of key image (video, motion,…) parts
• Top down vs. bottom up• Hard to measure success• (Fitting: associate a model with observed
features)
![Page 5: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/5.jpg)
Examples of grouping in vision
[Figure by J. Shi]
[http://poseidon.csd.auth.gr/LAB_RESEARCH/Latest/imgs/SpeakDepVidIndex_img2.jpg]
Determine image regionsFind shot boundaries
Fg / Bg
[Figure by Wang & Suter]
Object-level grouping
![Page 6: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/6.jpg)
Gestalt
• Gestalt: whole or group• Whole is greater than sum of its parts• Psychologists identified series of factors
that predispose set of elements to be grouped
• Interesting observations/explanations, but not necessarily useful for algorithm building
![Page 7: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/7.jpg)
Some Gestalt factors
![Page 8: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/8.jpg)
Muller-Lyer illusion
• http://www.michaelbach.de/ot/sze_muelue/index.html
Gestalt principle: grouping key to visual perception.
![Page 9: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/9.jpg)
![Page 10: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/10.jpg)
![Page 11: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/11.jpg)
Subjective contours
In Vision, D. Marr, 1982
Interesting tendency to explain by occlusion
![Page 12: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/12.jpg)
![Page 13: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/13.jpg)
D. Forsyth
![Page 14: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/14.jpg)
![Page 15: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/15.jpg)
![Page 16: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/16.jpg)
![Page 17: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/17.jpg)
In Vision, D. Marr, 1982; from J. L. Marroquin, “Human visual perception of structure”, 1976.
![Page 18: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/18.jpg)
Outline
• Why segmentation?• Gestalt properties, fun illusions and/or
revealing examples• Clustering
– Hierarchical– K-means– Mean Shift– Graph-theoretic
• Normalized cuts
![Page 19: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/19.jpg)
Histograms vs. clustering
![Page 20: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/20.jpg)
Segmentation as clustering
• Cluster similar pixels (features) together
R=0G=200B=20
R=255G=200B=250 R=245
G=220B=248
R=15G=189B=2
R=3G=12B=2
R
G
B
![Page 21: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/21.jpg)
Segmentation as clustering
• Cluster similar pixels (features) together
R=0G=200B=20X=30Y=20
R=15G=189B=2X=20Y=400 R=3
G=12B=2X=100Y=200
…
![Page 22: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/22.jpg)
Hierarchical clustering
• Agglomerative: Each point is a cluster, Repeatedly merge two nearest clusters
• Divisive: Start with single cluster, Repeatedly split into most distant clusters
![Page 23: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/23.jpg)
Dendrogram
![Page 24: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/24.jpg)
Inter-cluster distances
• Single link: min distance between any elements
• Complete link: max distance between any elements
• Average link
![Page 25: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/25.jpg)
K-means• Given k, want to minimize
error among k clusters• Error defined as distance
of cluster points to its center
• Search space too large• k-means: iterative
algorithm :– Fix cluster centers, allocate– Fix allocation, compute best
centers
![Page 26: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/26.jpg)
K-means slides by Andrew Moore
![Page 27: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/27.jpg)
![Page 28: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/28.jpg)
![Page 29: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/29.jpg)
![Page 30: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/30.jpg)
![Page 31: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/31.jpg)
K-means for color-based segementation
![Page 32: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/32.jpg)
![Page 33: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/33.jpg)
![Page 34: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/34.jpg)
![Page 35: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/35.jpg)
K-means and outliers
![Page 36: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/36.jpg)
K-means
• Use of centroid + spread – doesn’t describe irregularly shaped clusters
![Page 37: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/37.jpg)
K-means• Pros
– Simple– Converges to local minimum of within-cluster squared
error– Fast to compute
• Cons/issues– Setting k?– Sensitive to initial centers (seeds)– Usable only if mean is defined– Detects spherical clusters– Careful combining feature types
![Page 38: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/38.jpg)
Probabilistic clusteringBasic questions
• what’s the probability that a point x is in cluster m?• what’s the shape of each cluster?
K-means doesn’t answer these questions
Probabilistic clustering (basic idea)• Treat each cluster as a Gaussian density function
Slide credit: Steve Seitz
![Page 39: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/39.jpg)
Expectation Maximization (EM)
A probabilistic variant of K-means:• E step: “soft assignment” of points to clusters
– estimate probability that a point is in a cluster• M step: update cluster parameters
– mean and variance info (covariance matrix)• maximizes the likelihood of the points given the clusters• Forsyth Chapter 16 (optional)
Slide credit: Steve Seitz
![Page 40: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/40.jpg)
Outline• Why segmentation?• Gestalt properties, fun illusions and/or
revealing examples• Clustering
– Hierarchical– K-means– Mean Shift– Graph-theoretic
• Normalized cuts
![Page 41: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/41.jpg)
Mean shift• Seeks the mode among sampled
data, or point of highest density– Choose search window size – Choose initial location of
search window– Compute mean location
(centroid) in window– Re-center search window at
mean location– Repeat until convergence
Fukunaga & Hostetler 1975 Comaniciu & Meer, PAMI 2002
![Page 42: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/42.jpg)
Mean shiftRegion ofinterest
Center ofmass
Mean Shiftvector
Slide by Y. Ukrainitz & B. Sarel
![Page 43: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/43.jpg)
Region ofinterest
Center ofmass
Mean Shiftvector
Slide by Y. Ukrainitz & B. Sarel
Mean shift
![Page 44: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/44.jpg)
Region ofinterest
Center ofmass
Mean Shiftvector
Slide by Y. Ukrainitz & B. Sarel
Mean shift
![Page 45: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/45.jpg)
Region ofinterest
Center ofmass
Mean Shiftvector
Slide by Y. Ukrainitz & B. Sarel
Mean shift
![Page 46: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/46.jpg)
Region ofinterest
Center ofmass
Mean Shiftvector
Slide by Y. Ukrainitz & B. Sarel
Mean shift
![Page 47: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/47.jpg)
Region ofinterest
Center ofmass
Mean Shiftvector
Slide by Y. Ukrainitz & B. Sarel
Mean shift
![Page 48: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/48.jpg)
Region ofinterest
Center ofmass
Slide by Y. Ukrainitz & B. Sarel
Mean shift
![Page 49: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/49.jpg)
Real Modality Analysis
Tessellate the space with windows
Run the procedure in parallel
![Page 50: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/50.jpg)
Real Modality Analysis
The blue data points were traversed by the windows towards the modeSlide by Y. Ukrainitz & B. Sarel
![Page 51: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/51.jpg)
Mean shift segmentation
Comaniciu & Meer, PAMI 2002
![Page 52: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/52.jpg)
Comaniciu & Meer, PAMI 2002
![Page 53: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/53.jpg)
CAMSHIFT [G. Bradski]
• Variant on mean shift: “Continuously adaptive mean shift”
• Shown for face tracking for a user interface• Want mode of color distribution in a video scene• Dynamic distribution now, since there is motion,
scale change• Adjust search window size dynamically, based
on area of face
![Page 54: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/54.jpg)
CAMSHIFT [G. Bradski]
![Page 55: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/55.jpg)
CAMSHIFT [G. Bradski]
![Page 56: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/56.jpg)
CAMSHIFT [G. Bradski]
![Page 57: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/57.jpg)
Mean shift• Pros:
– Does not assume shape on clusters (e.g. elliptical)
– One parameter choice (window size)– Generic technique– Find multiple modes
• Cons:– Selection of window size– Does not scale well with dimension of feature
space (but may insert approx. for high-d data…)
![Page 58: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/58.jpg)
Graph-theoretic clustering
![Page 59: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/59.jpg)
Graph representation
a b c d e
a
b
c
d
e
![Page 60: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/60.jpg)
Weighted graph representation
(“Affinity matrix”)
![Page 61: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/61.jpg)
q
Images as graphs
Fully-connected graph• node for every pixel• link between every pair of pixels, p,q• similarity cpq for each link
» similarity is inversely proportional to difference in color and position
p
Cpqc
Slide by Steve Seitz
![Page 62: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/62.jpg)
Segmentation by Graph Cuts
Break Graph into Segments• Delete links that cross between segments• Easiest to break links that have low similarity
– similar pixels should be in the same segments– dissimilar pixels should be in different segments
w
A B C
Slide by Steve Seitz
![Page 63: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/63.jpg)
Measuring affinity• One possibility:
Small sigma: group only nearby points
Large sigma: group distant points
![Page 64: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/64.jpg)
Scale affects affinity
σ=.1 σ=.2 σ=1
σ=.2
Data points
Affinity matrices
![Page 65: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/65.jpg)
Eigenvectors and cuts• Want a vector a giving the association
between each element and a cluster• Want elements within this cluster to have
strong affinity with one another
• Maximize
subject to the constraint
• Eigenvalue problem : choose the eigenvector of A with largest eigenvalue
aT Aa
aTa = 1
![Page 66: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/66.jpg)
Rayleigh Quotient
![Page 67: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/67.jpg)
Example
Data points
Affinity matrix
eigenvector
![Page 68: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/68.jpg)
Eigenvectors and multiple cuts
• Use eigenvectors associated with k largest eigenvalues as cluster weights
• Or re-solve recursively
![Page 69: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/69.jpg)
Scale affects affinity, number of clusters
Eigenvalues of the affinity matrices
Data points
Affinity matrices
![Page 70: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/70.jpg)
Graph partitioning: Min cut
• Select bipartition that minimizes cut value, i.e., total weight of edges removed
A B
Fast algorithms exist for this
A, B are disjoint sets
![Page 71: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/71.jpg)
Min cut
• Problem: weight of cut proportional to number of edges in the cut; min cut penalizes large segments
[Shi & Malik, 2000 PAMI]
![Page 72: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/72.jpg)
Normalized cuts
• First eigenvector of affinity matrix captures within cluster similarity, but not across cluster difference
• Would like to maximize the within cluster similarity relative to the across cluster difference
![Page 73: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/73.jpg)
Normalized cuts
• Minimize
• To get disjoint groups A, B for which within cluster similarity is high compared to their association with rest of graph
= total connection from nodes in A to all nodes in graph (V)
![Page 74: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/74.jpg)
Normalized cuts
• Minimize
• Maximize
= total connection from nodes in A to all nodes in graph (V)
assoc(A, A)assoc(A,V )
⎛ ⎝ ⎜
⎞ ⎠ ⎟ +
assoc(B,B)assoc(B,V )
⎛ ⎝ ⎜
⎞ ⎠ ⎟
![Page 75: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/75.jpg)
Normalized cuts• Exact discrete solution is NP-complete
[Papadimitrou 1997] • But can efficiently approximate via
generalized eigenvalue problem [Shi & Malik] ☺
![Page 76: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/76.jpg)
Figure from “Image and video segmentation: the normalised cut framework”, by Shi and Malik, copyright IEEE, 1998
![Page 77: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/77.jpg)
Figure from “Normalized cuts and image segmentation,” Shi and Malik, copyright IEEE, 2000
![Page 78: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/78.jpg)
![Page 79: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/79.jpg)
Examples of grouping in vision
Shapiro & Stockman, P. Duygulu
Motion segmentation
![Page 80: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/80.jpg)
Features = measure of motion/velocity
Motion Segmentation and Tracking Using Normalized Cuts [Shi & Malik 1998]
![Page 81: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/81.jpg)
K-means vs. graph cuts, mean shift
• Graph cuts / spectral clustering, mean shift: do not require model of data distribution
![Page 82: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/82.jpg)
Scale selection for spectral clustering
• How to select scale for analysis?• What about multi-scale data?
![Page 83: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/83.jpg)
Measuring affinity
Small sigma: group only nearby points
Large sigma: group distant points
![Page 84: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/84.jpg)
Segmentation: Caveats
• Can’t hope for magic• Intertwined with recognition problem• Have to be careful not to make hard
decisions too soon• Hard to evaluate
![Page 85: Lecture 7: Segmentationgrauman/courses/378/... · –Hierarchical – K-means – Mean Shift – Graph-theoretic • Normalized cuts. Mean shift • Seeks the mode among sampled data,](https://reader034.vdocument.in/reader034/viewer/2022042710/5f675909ef284573174bcbc7/html5/thumbnails/85.jpg)
Next
• Fitting for grouping• Read F&P Chapter 15 (ignore
fundamental matrix sections for now)• Problem set 1 due Tues. – estimate time