cs434a/541a: pattern recognition prof. olga veksleroveksler/courses/fall2016/cs9840/lecture… ·...
TRANSCRIPT
![Page 1: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/1.jpg)
CS9840 Machine Learning in Computer Vision
Olga Veksler
Lecture 4 Image Representation
![Page 2: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/2.jpg)
Outline
• How to represent an image as a feature vector? • Basic image features
• intensity, color, gradients, response to filter(s) • dense (at each pixel) • sparse (at a subset of locations)
• Representation of image through basic features • pixelwise • histogram
• Global vs. Local histograms • Spatial pyramids
![Page 3: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/3.jpg)
Basic Image Features • Given image I, first compute basic image features
• e.g. Intensity of a pixel, not enough for most applications
• Other image basic features commonly used
Color: 3 values per pixel Edges: 1 or 2 values per pixel Texture: ≈ 48 values per pixel
• Basic features can be sparse or dense • Common sparse basic features
Corners Stable regions SIFT features
• Consolidate basic features into feature vector x that represents image I
![Page 4: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/4.jpg)
Extracting Texture (Texture Responses)
• Texture filter bank:
scales
orientations
“Edges” “Bars”
“Spots”
• Convolve image with each filter • 48 responses per pixel
![Page 5: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/5.jpg)
Kristen Grauman
![Page 6: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/6.jpg)
Showing magnitude of responses
Kristen Grauman
![Page 7: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/7.jpg)
Kristen Grauman
![Page 8: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/8.jpg)
Kristen Grauman
![Page 9: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/9.jpg)
Kristen Grauman
![Page 10: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/10.jpg)
Kristen Grauman
![Page 11: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/11.jpg)
Kristen Grauman
![Page 12: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/12.jpg)
Kristen Grauman
![Page 13: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/13.jpg)
Kristen Grauman
![Page 14: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/14.jpg)
Kristen Grauman
![Page 15: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/15.jpg)
Kristen Grauman
![Page 16: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/16.jpg)
Kristen Grauman
![Page 17: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/17.jpg)
Kristen Grauman
![Page 18: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/18.jpg)
Kristen Grauman
![Page 19: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/19.jpg)
Kristen Grauman
![Page 20: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/20.jpg)
Kristen Grauman
![Page 21: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/21.jpg)
Kristen Grauman
![Page 22: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/22.jpg)
Kristen Grauman
![Page 23: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/23.jpg)
Kristen Grauman
![Page 24: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/24.jpg)
Kristen Grauman
![Page 25: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/25.jpg)
[r1, r2, …, r38]
Form a descriptor vector from the list of responses at each pixel
Kristen Grauman
Extracting Texture
![Page 26: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/26.jpg)
Extracting Texture
[r1,…, large, …., small, …, r48]
[r1,…, small, …., large, …, r48]
![Page 27: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/27.jpg)
Right features depend on what you want to know
• Object: 2D shape • Local shape info, shading, shadows, texture
• Scene : overall layout • linear perspective, gradients
• Material properties: albedo, feel, hardness, … • Color, texture
• Motion • Optical flow, tracked points
Slide by Derek Hoiem
![Page 28: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/28.jpg)
Basic Image Features • Each basic feature is described by a multi-dimensional descriptor
Color
200120100
SIFT features
11222...33
100
128
dim
ensio
ns
• For machine learning, need to consolidate basic descriptors into feature vector x that represents image I
![Page 29: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/29.jpg)
Pixelwise Representation
10 12 30 20 17 21 33 91 24
249133211720301210
• Pile basic descriptors into one vector, say row order • Example: intensity as a basic image feature
• one value per pixel
![Page 30: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/30.jpg)
Pixelwise Representation
623814477195
• Color as a basic image feature • three values per pixel, descriptor is 3-dimensional
• Pile all color channel into one vector
5 7 8
9 4 3
1 4 2
7 1 6
617241349875
or
![Page 31: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/31.jpg)
Pixelwise Representation
0123623814477195
• Basic image feature has n dimensional descriptor • Sometimes each dimension is called a “channel”
• Pile each channel one after another into one vector
5 7 8 3
9 4 3 2
1 4 2 1
7 1 6 0
![Page 32: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/32.jpg)
Pixel Representations
Slide by Erik Learned-Miller
• Small change in image appearance
![Page 33: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/33.jpg)
Pixel Representations
• Leads to a large change in feature vector
difference image 10 12 30
20 17 21 33 91 24
9 10 12
19 20 17 32 33 91
[10 12 30 20 17 21 33 91 24]
[9 10 12 19 20 17 32 33 91]
Slide modified from Erik Learned-Miller
![Page 34: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/34.jpg)
Pixel Representations
• Pixelwise representations: overly sensitive to position
• Nevertheless it has been successfully used in applications • eigenfaces, first successful face detection system
Slide by Erik Learned-Miller
![Page 35: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/35.jpg)
Global Intensity Histogram • Think of each intensity value as a “bin”
0 1 2 3 4
• Visual plot:
• Histogram counts the number of values that fall in each bin
0 1 2 3 4
100 15 25 0 5
0 1 2 3 4
![Page 36: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/36.jpg)
Global Intensity Histogram
100...4
105
• Insensitive to changes in pixel location • Often use normalized histogram
• sums up to 1
8.0...03.008.004.0
![Page 37: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/37.jpg)
Global Intensity Histogram Quantization • Can quantize intensities (larger bins)
• Histogram: count number of values that fall in each bin
0 15 5 10
• Quantization • helps to improve efficiency • groups similar values together (i.e. removes fine distinction)
• may help for recognition
0 15 5 10
125 5 91
![Page 38: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/38.jpg)
Multi-Dimensional Histograms
• Basic descriptors most often multi-dimensional • color, texture, optical flow, etc.
• How to build histogram? • Have to quantize, too sparse without quantization
![Page 39: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/39.jpg)
How to Quantize Multi-Dimensional Data? 1. Joint histogram
• Need lots of data to avoid empty bins • Make bins coarse to simulate lots of
data → loose resolution
slide credit: Dave Kauchak
dim
ensio
n 2
dimension 1
dim
ensio
n 2
dimension 1 dimension 1
dim
ensio
n 2
dimension 1
dim
ensio
n 2
2. Marginal Histogram • More data per bin than joint histogram • Works best for independent features
• Loose correlation information
![Page 40: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/40.jpg)
Space Shuttle Cargo Bay
Histograms based on Irregular Partitioning
Slide Credit: Dave Kauchak
• Irregular quantization (clustering) gives meaningful bins that adapt to data • k-means clustering, etc.
• Cluster centers are called codewords • A sample is identified (assigned to) with the closest codeword • Build histogram over the codeword
• count how many samples are closest to codeword 1, codeword 2, etc.
• Need to store only the codewords
codeword 1 codeword 2
goes in the ‘bin’ for codeword 1
![Page 41: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/41.jpg)
Encoding Image I as Feature Vector • Pre-computed code-words in green • Extract 2D features from image I
c1
c2
c3
c4
c5
c6
c7
c8
codeword 1
codeword 2
codeword 3
codeword 8
3 6 4 2
histogram
• Feature vector that represents image I • can also normalize it
3...246
![Page 42: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/42.jpg)
Clustered Patches • So far clustered feature responses at each pixel • Can cluster other things • Like image patches
• overlapping or not cluster 1
cluster 2
cluster 3
![Page 43: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/43.jpg)
Clustered Patches • Take patches form many training
data images • But not from test images
• Use only a subset of training data for speed
• Usually normalize patches to be of zero mean, unit variance
• Cluster centers become codewords
![Page 44: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/44.jpg)
Fei-Fei et al. 2005
Centers of Clustered Patches (Codewords)
![Page 45: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/45.jpg)
Feature Vector for image I • To represent image I
• Extract patches, overlapping or not • Find the closest codeword for each
patch • Build histogram
…..
coun
t
codewords A. Efros
![Page 46: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/46.jpg)
Analogy to documents: Bag of Words
Of all the sensory impressions proceeding to the brain, the visual experiences are the dominant ones. Our perception of the world around us is based essentially on the messages that reach the brain from our eyes. For a long time it was thought that the retinal image was transmitted point by point to visual centers in the brain; the cerebral cortex was a movie screen, so to speak, upon which the image in the eye was projected. Through the discoveries of Hubel and Wiesel we now know that behind the origin of the visual perception in the brain there is a considerably more complicated course of events. By following the visual impulses along their path to the various cell layers of the optical cortex, Hubel and Wiesel have been able to demonstrate that the message about the image falling on the retina undergoes a step-wise analysis in a system of nerve cells stored in columns. In this system each cell has its specific function and is responsible for a specific detail in the pattern of the retinal image.
sensory, brain, visual, perception,
retinal, cerebral cortex, eye, cell, optical
nerve, image Hubel, Wiesel
China is forecasting a trade surplus of $90bn (£51bn) to $100bn this year, a threefold increase on 2004's $32bn. The Commerce Ministry said the surplus would be created by a predicted 30% jump in exports to $750bn, compared with a 18% rise in imports to $660bn. The figures are likely to further annoy the US, which has long argued that China's exports are unfairly helped by a deliberately undervalued yuan. Beijing agrees the surplus is too high, but says the yuan is only one factor. Bank of China governor Zhou Xiaochuan said the country also needed to do more to boost domestic demand so more goods stayed within the country. China increased the value of the yuan against the dollar by 2.1% in July and permitted it to trade within a narrow band, but the US wants the yuan to be allowed to trade freely. However, Beijing has made it clear that it will take its time and tread carefully before allowing the yuan to rise further in value.
China, trade, surplus, commerce,
exports, imports, US, yuan, bank, domestic,
foreign, increase, trade, value
ICCV 2005 short course, L. Fei-Fei
• Inspiration comes from text classification
![Page 47: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/47.jpg)
Bag of visual words
• codewords or visual words
• Bow histogram
codewords
Slide by Derek Hoiem
• Training images
![Page 48: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/48.jpg)
category decision
learning
build codewords codewords dictionary
image representation
Train Classifier
recognition
A. Efros
![Page 49: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/49.jpg)
Histograms: Implementation issues
Few Bins Need less data Coarser representation If too coarse, distinction is lost
Many Bins Need more data Finer representation If too fine, more distinction than necessary
• Quantization • Grids: fast but applicable only with few dimensions • Clustering: slower but can quantize data in higher dimension
• How many bins (clusters)?
Slide Credit: Derek Hoiem
![Page 50: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/50.jpg)
Problem with Global Histogram
Slide by Erik Learned-Miller
• Identical feature vectors!
![Page 51: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/51.jpg)
Problem with Global Histogram
Have equal histograms!
Slide by Erik Learned-Miller
![Page 52: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/52.jpg)
Conclusions
1. Pixel representations: overly sensitive to position
2. Global histogram representations: under-sensitive to position
Slide by Erik Learned-Miller
![Page 53: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/53.jpg)
A Compromise: A local histogram A separate (normalized) histogram for each region
Slide by Erik Learned-Miller
1 region hist. 2 region hist.
.
.
.
16 region hist.
![Page 54: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/54.jpg)
Local Intensity Histogram
. . .
. . .
![Page 55: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/55.jpg)
Local Intensity Histogram
• Intensity histogram is sensitive to lighting changes
. . .
. . .
![Page 56: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/56.jpg)
Local Edge Orientation Histogram
• Edges are not as sensitive to lighting changes • Compute histogram of edges
• typically consider only edge orientation
. . .
• How do we choose the right box size?
![Page 57: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/57.jpg)
Spatial pyramid
Slide Credit: Derek Hoiem
• Use boxes of different sizes!
![Page 58: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/58.jpg)
Spatial Pyramid
Slide Credit: Derek Hoiem
These get piled up into one feature vector
![Page 59: CS434a/541a: Pattern Recognition Prof. Olga Veksleroveksler/Courses/Fall2016/CS9840/Lecture… · Olga Veksler Lecture 4 Image Representation . Outline •How to represent an image](https://reader035.vdocument.in/reader035/viewer/2022081516/61327a81dfd10f4dd73a79cc/html5/thumbnails/59.jpg)
Other Representations
• Many image representation schemes are based on histogram of • texture • corner features • SIFT features • etc.
• There are other ways to represent an image as a feature vector