templates, image pyramids, and filter banks
DESCRIPTION
02/08/11. Templates, Image Pyramids, and Filter Banks. Computer Vision Derek Hoiem, University of Illinois. Review. Match the spatial domain image to the Fourier magnitude image. 2. 3. 1. 4. 5. B. C. E. A. D. Reminder. HW 1 due in one week. Today’s class. Template matching - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/1.jpg)
Templates, Image Pyramids, and Filter Banks
Computer VisionDerek Hoiem, University of Illinois
02/08/11
![Page 2: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/2.jpg)
Review1. Match the spatial domain image to the
Fourier magnitude image1 54
A
32
C
B
DE
![Page 3: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/3.jpg)
Reminder• HW 1 due in one week
![Page 4: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/4.jpg)
Today’s class
• Template matching
• Image Pyramids
• Filter banks and texture
• Denoising, Compression
![Page 5: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/5.jpg)
Template matching• Goal: find in image
• Main challenge: What is a good similarity or distance measure between two patches?– Correlation– Zero-mean correlation– Sum Square Difference– Normalized Cross
Correlation
![Page 6: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/6.jpg)
Matching with filters• Goal: find in image• Method 0: filter the image with eye patch
Input Filtered Image
],[],[],[,
lnkmflkgnmhlk
What went wrong?
f = imageg = filter
![Page 7: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/7.jpg)
Matching with filters• Goal: find in image• Method 1: filter the image with zero-mean eye
Input Filtered Image (scaled) Thresholded Image
)],[()],[(],[,
lnkmgflkfnmhlk
True detections
False detections
mean of f
![Page 8: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/8.jpg)
Matching with filters• Goal: find in image• Method 2: SSD
Input 1- sqrt(SSD) Thresholded Image
2
,
)],[],[(],[ lnkmflkgnmhlk
True detections
![Page 9: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/9.jpg)
Matching with filters
Can SSD be implemented with linear filters?2
,
)],[],[(],[ lnkmflkgnmhlk
![Page 10: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/10.jpg)
Matching with filters• Goal: find in image• Method 2: SSD
Input 1- sqrt(SSD)
2
,
)],[],[(],[ lnkmflkgnmhlk
What’s the potential downside of SSD?
![Page 11: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/11.jpg)
Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation
5.0
,
2,
,
2
,,
)],[()],[(
)],[)(],[(],[
lknm
lk
nmlk
flnkmfglkg
flnkmfglkgnmh
Matlab: normxcorr2(template, im)
mean image patchmean template
![Page 12: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/12.jpg)
Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation
Input Normalized X-Correlation Thresholded Image
True detections
![Page 13: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/13.jpg)
Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation
Input Normalized X-Correlation Thresholded Image
True detections
![Page 14: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/14.jpg)
Q: What is the best method to use?
A: Depends• Zero-mean filter: fastest but not a great
matcher• SSD: next fastest, sensitive to overall intensity• Normalized cross-correlation: slowest,
invariant to local average intensity and contrast
![Page 15: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/15.jpg)
Q: What if we want to find larger or smaller eyes?
A: Image Pyramid
![Page 16: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/16.jpg)
Review of Sampling
Low-Pass Filtered ImageImage
GaussianFilter Sample Low-Res
Image
![Page 17: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/17.jpg)
Gaussian pyramid
Source: Forsyth
![Page 18: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/18.jpg)
Template Matching with Image Pyramids
Input: Image, Template1. Match template at current scale
2. Downsample image
3. Repeat 1-2 until image is very small
4. Take responses above some threshold, perhaps with non-maxima suppression
![Page 19: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/19.jpg)
Coarse-to-fine Image Registration1. Compute Gaussian pyramid2. Align with coarse pyramid3. Successively align with finer
pyramids– Search smaller range
Why is this faster?
Are we guaranteed to get the same result?
![Page 20: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/20.jpg)
Laplacian filter
Gaussianunit impulse
Laplacian of Gaussian
Source: Lazebnik
![Page 21: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/21.jpg)
Laplacian pyramid
Source: Forsyth
![Page 22: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/22.jpg)
Computing Gaussian/Laplacian Pyramid
http://sepwww.stanford.edu/~morgan/texturematch/paper_html/node3.html
Can we reconstruct the original from the laplacian pyramid?
![Page 23: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/23.jpg)
Hybrid Image
![Page 24: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/24.jpg)
Hybrid Image in Laplacian Pyramid
High frequency Low frequency
![Page 25: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/25.jpg)
Image representation
• Pixels: great for spatial resolution, poor access to frequency
• Fourier transform: great for frequency, not for spatial info
• Pyramids/filter banks: balance between spatial and frequency information
![Page 26: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/26.jpg)
Major uses of image pyramids
• Compression
• Object detection– Scale search– Features
• Detecting stable interest points
• Registration– Course-to-fine
![Page 27: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/27.jpg)
Application: Representing Texture
Source: Forsyth
![Page 28: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/28.jpg)
Texture and Material
http://www-cvr.ai.uiuc.edu/ponce_grp/data/texture_database/samples/
![Page 29: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/29.jpg)
Texture and Orientation
http://www-cvr.ai.uiuc.edu/ponce_grp/data/texture_database/samples/
![Page 30: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/30.jpg)
Texture and Scale
http://www-cvr.ai.uiuc.edu/ponce_grp/data/texture_database/samples/
![Page 31: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/31.jpg)
What is texture?
Regular or stochastic patterns caused by bumps, grooves, and/or markings
![Page 32: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/32.jpg)
How can we represent texture?
• Compute responses of blobs and edges at various orientations and scales
![Page 33: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/33.jpg)
Overcomplete representation: filter banks
LM Filter Bank
Code for filter banks: www.robots.ox.ac.uk/~vgg/research/texclass/filters.html
![Page 34: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/34.jpg)
Filter banks• Process image with each filter and keep
responses (or squared/abs responses)
![Page 35: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/35.jpg)
How can we represent texture?
• Measure responses of blobs and edges at various orientations and scales
• Idea 1: Record simple statistics (e.g., mean, std.) of absolute filter responses
![Page 36: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/36.jpg)
Can you match the texture to the response?
Mean abs responses
FiltersA
B
C
1
2
3
![Page 37: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/37.jpg)
Representing texture by mean abs response
Mean abs responses
Filters
![Page 38: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/38.jpg)
Representing texture• Idea 2: take vectors of filter responses at each pixel and
cluster them, then take histograms (more on in coming weeks)
![Page 39: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/39.jpg)
How is it that a 4MP image can be compressed to a few hundred KB without a noticeable change?
Compression
![Page 40: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/40.jpg)
Lossy Image Compression (JPEG)
Block-based Discrete Cosine Transform (DCT)
Slides: Efros
![Page 41: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/41.jpg)
Using DCT in JPEG • The first coefficient B(0,0) is the DC component, the
average intensity• The top-left coeffs represent low frequencies, the
bottom right – high frequencies
![Page 42: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/42.jpg)
Image compression using DCT• Quantize
– More coarsely for high frequencies (which also tend to have smaller values)
– Many quantized high frequency values will be zero• Encode
– Can decode with inverse dct
Quantization table
Filter responses
Quantized values
![Page 43: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/43.jpg)
JPEG Compression Summary1. Convert image to YCrCb2. Subsample color by factor of 2
– People have bad resolution for color3. Split into blocks (8x8, typically), subtract 1284. For each block
a. Compute DCT coefficientsb. Coarsely quantize
• Many high frequency components will become zero
c. Encode (e.g., with Huffman coding)
http://en.wikipedia.org/wiki/YCbCrhttp://en.wikipedia.org/wiki/JPEG
![Page 44: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/44.jpg)
Lossless compression (PNG)
1. Predict that a pixel’s value based on its upper-left neighborhood
2. Store difference of predicted and actual value
3. Pkzip it (DEFLATE algorithm)
![Page 45: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/45.jpg)
Denoising
Additive Gaussian Noise
Gaussian Filter
![Page 46: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/46.jpg)
Smoothing with larger standard deviations suppresses noise, but also blurs the image
Reducing Gaussian noise
Source: S. Lazebnik
![Page 47: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/47.jpg)
Reducing salt-and-pepper noise by Gaussian smoothing
3x3 5x5 7x7
![Page 48: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/48.jpg)
Alternative idea: Median filtering• A median filter operates over a window by
selecting the median intensity in the window
• Is median filtering linear?Source: K. Grauman
![Page 49: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/49.jpg)
Median filter• What advantage does median filtering have over
Gaussian filtering?– Robustness to outliers
Source: K. Grauman
![Page 50: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/50.jpg)
Median filterSalt-and-pepper noise Median filtered
Source: M. Hebert
• MATLAB: medfilt2(image, [h w])
![Page 51: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/51.jpg)
Median vs. Gaussian filtering3x3 5x5 7x7
Gaussian
Median
![Page 52: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/52.jpg)
Other non-linear filters• Weighted median (pixels further from center count less)
• Clipped mean (average, ignoring few brightest and darkest pixels)
• Bilateral filtering (weight by spatial distance and intensity difference)
http://vision.ai.uiuc.edu/?p=1455Image:
Bilateral filtering
![Page 53: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/53.jpg)
Review of last three days
![Page 54: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/54.jpg)
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Credit: S. Seitz
],[],[],[,
lnkmglkfnmhlk
[.,.]h[.,.]f
Review: Image filtering111
111
111
],[g
![Page 55: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/55.jpg)
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
],[],[],[,
lnkmglkfnmhlk
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
![Page 56: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/56.jpg)
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
],[],[],[,
lnkmglkfnmhlk
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
![Page 57: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/57.jpg)
Filtering in spatial domain-101
-202
-101
* =
![Page 58: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/58.jpg)
Filtering in frequency domain
FFT
FFT
Inverse FFT
=
![Page 59: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/59.jpg)
Review of Last 3 Days• Linear filters for basic processing
– Edge filter (high-pass)– Gaussian filter (low-pass)
FFT of Gaussian
[-1 1]
FFT of Gradient Filter
Gaussian
![Page 60: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/60.jpg)
Review of Last 3 Days• Derivative of Gaussian
![Page 61: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/61.jpg)
Review of Last 3 Days• Applications of filters
– Template matching (SSD or Normxcorr2)• SSD can be done with linear filters, is sensitive to
overall intensity– Gaussian pyramid
• Coarse-to-fine search, multi-scale detection– Laplacian pyramid
• More compact image representation• Can be used for compositing in graphics
![Page 62: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/62.jpg)
Review of Last 3 Days• Applications of filters
– Downsampling• Need to sufficiently low-pass before downsampling
– Compression• In JPEG, coarsely quantize high frequencies
![Page 63: Templates, Image Pyramids, and Filter Banks](https://reader035.vdocument.in/reader035/viewer/2022062816/56815c3d550346895dca3913/html5/thumbnails/63.jpg)
Next class: edge and line detection