computational photography cse 590
DESCRIPTION
Computational Photography CSE 590. Tamara Berg Filtering & Pyramids. Image filtering. Image filtering: compute function of local neighborhood at each position Really important! Enhance images Denoise, resize, increase contrast, etc. Extract information from images - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/1.jpg)
Computational PhotographyCSE 590
Tamara Berg
Filtering & Pyramids
![Page 2: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/2.jpg)
Image filtering
• Image filtering: compute function of local neighborhood at each position
• Really important!– Enhance images
• Denoise, resize, increase contrast, etc.– Extract information from images
• Texture, edges, distinctive points, etc.– Detect patterns
• Template matching
![Page 3: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/3.jpg)
111
111
111
Slide credit: David Lowe (UBC)
],[g
Example: box filter
![Page 4: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/4.jpg)
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
[.,.]h[.,.]f
Image filtering111
111
111
],[g
![Page 5: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/5.jpg)
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
![Page 6: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/6.jpg)
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
![Page 7: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/7.jpg)
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
![Page 8: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/8.jpg)
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
![Page 9: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/9.jpg)
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
?
],[],[],[,
lnkmflkgnmhlk
![Page 10: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/10.jpg)
0 10 20 30 30
50
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
?
],[],[],[,
lnkmflkgnmhlk
![Page 11: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/11.jpg)
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30 30 30 20 10
0 20 40 60 60 60 40 20
0 30 60 90 90 90 60 30
0 30 50 80 80 90 60 30
0 30 50 80 80 90 60 30
0 20 30 50 50 60 40 20
10 20 30 30 30 30 20 10
10 10 10 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111111111],[g
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
![Page 12: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/12.jpg)
What does it do?• Replaces each pixel with an
average of its neighborhood
• Achieve smoothing effect (remove sharp features)
111
111
111
Slide credit: David Lowe (UBC)
],[g
Box Filter
![Page 13: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/13.jpg)
Smoothing with box filter
![Page 14: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/14.jpg)
Practice with linear filters
000
010
000
Original
?
Source: D. Lowe
![Page 15: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/15.jpg)
Practice with linear filters
000
010
000
Original Filtered (no change)
Source: D. Lowe
![Page 16: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/16.jpg)
Practice with linear filters
000
100
000
Original
?
Source: D. Lowe
![Page 17: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/17.jpg)
Practice with linear filters
000
100
000
Original Shifted leftBy 1 pixel
Source: D. Lowe
![Page 18: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/18.jpg)
Practice with linear filters
Original
111111111
000020000 - ?
Source: D. Lowe
![Page 19: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/19.jpg)
Practice with linear filters
Original
111111111
000020000 -
Sharpening filter- Accentuates differences with local
average
Source: D. Lowe
![Page 20: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/20.jpg)
Sharpening
Source: D. Lowe
![Page 21: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/21.jpg)
Characterizing edges• An edge is a place of rapid change in the image
intensity function
imageintensity function
(along horizontal scanline) first derivative
edges correspond toextrema of derivative
source: Svetlana Lazebnik
![Page 22: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/22.jpg)
Edge filters
Approximations of derivative filters:
Source: K. Grauman
Convolve filter with image to get edge map
![Page 23: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/23.jpg)
Edge filters
Approximations of derivative filters:
Source: K. GraumanRespond highly to vertical edges
![Page 24: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/24.jpg)
Edge filters
Approximations of derivative filters:
Source: K. GraumanRespond highly to horizontal edges
![Page 25: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/25.jpg)
Edges: example
source: Svetlana Lazebnik
![Page 26: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/26.jpg)
Gaussian Kernel
• Constant factor at front makes volume sum to 1 (can be ignored, as we should re-normalize weights to sum to 1 in any case)
0.003 0.013 0.022 0.013 0.0030.013 0.059 0.097 0.059 0.0130.022 0.097 0.159 0.097 0.0220.013 0.059 0.097 0.059 0.0130.003 0.013 0.022 0.013 0.003
5 x 5, = 1
Source: C. Rasmussen
![Page 27: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/27.jpg)
Example: Smoothing with a Gaussian
source: Svetlana Lazebnik
![Page 28: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/28.jpg)
Gaussian filters
• Remove “high-frequency” components from the image (low-pass filter)– Images become more smooth
• Convolution with self is another Gaussian– So can smooth with small-width kernel, repeat, and
get same result as larger-width kernel would have• Separable kernel
– Factors into product of two 1D Gaussians
Source: K. Grauman
![Page 29: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/29.jpg)
Separability of the Gaussian filter
Source: D. Lowe
The filter factorsinto a product of 1D
filters:
![Page 30: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/30.jpg)
Practical matters• What is the size of the output?• MATLAB: imfilter(g, f, shape)
– shape = ‘full’: output size is sum of sizes of f and g– shape = ‘same’: output size is same as f– shape = ‘valid’: output size is difference of sizes of f and g
f
gg
gg
f
gg
gg
f
gg
gg
full same valid
Source: S. Lazebnik
![Page 31: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/31.jpg)
Practical matters• What about near the edge?
– the filter window falls off the edge of the image– need to extrapolate– methods:
• clip filter (black)• wrap around• copy edge• reflect across edge
Source: S. Marschner
![Page 32: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/32.jpg)
Practical matters
– methods (MATLAB):• clip filter (black): imfilter(f, g, 0)• wrap around: imfilter(f, g, ‘circular’)• copy edge: imfilter(f, g, ‘replicate’)• reflect across edge: imfilter(f, g, ‘symmetric’)
Source: S. Marschner
Q?
![Page 33: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/33.jpg)
Template matching• Goal: find in image
• Main challenge: What is a good similarity or distance measure between two patches?– Correlation– Zero-mean correlation– Sum Square Difference– Normalized Cross Correlation
Side by Derek Hoiem
![Page 34: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/34.jpg)
Matching with filters• Goal: find in image• Method 0: filter the image with eye patch
Input Filtered Image
],[],[],[,
lnkmflkgnmhlk
What went wrong?
f = imageg = filter
Side by Derek Hoiem
![Page 35: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/35.jpg)
Matching with filters• Goal: find in image• Method 1: filter the image with zero-mean eye
Input Filtered Image (scaled) Thresholded Image
)],[()],[(],[,
lnkmgflkfnmhlk
True detections
False detections
mean of f
![Page 36: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/36.jpg)
Matching with filters• Goal: find in image• Method 2: SSD
Input 1- sqrt(SSD) Thresholded Image
2
,
)],[],[(],[ lnkmflkgnmhlk
True detections
![Page 37: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/37.jpg)
Matching with filters• Goal: find in image• Method 2: SSD
Input 1- sqrt(SSD)
2
,
)],[],[(],[ lnkmflkgnmhlk
What’s the potential downside of SSD?
Side by Derek Hoiem
![Page 38: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/38.jpg)
Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation
5.0
,
2,
,
2
,,
)],[()],[(
)],[)(],[(],[
lknm
lk
nmlk
flnkmfglkg
flnkmfglkgnmh
mean image patchmean template
Side by Derek Hoiem
![Page 39: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/39.jpg)
Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation
Input Normalized X-Correlation Thresholded Image
True detections
![Page 40: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/40.jpg)
Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation
Input Normalized X-Correlation Thresholded Image
True detections
![Page 41: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/41.jpg)
Q: What is the best method to use?
• A: Depends• Zero-mean filter: fastest but not a great
matcher• SSD: next fastest, sensitive to overall
intensity• Normalized cross-correlation: slowest,
invariant to local average intensity and contrast
Side by Derek Hoiem
![Page 42: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/42.jpg)
Image half-sizing
This image is too big tofit on the screen. Howcan we reduce it?
How to generate a half-sized version?
![Page 43: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/43.jpg)
Image sub-sampling
Throw away every other row and column to create a 1/2 size image
- called image sub-sampling
1/4
1/8
Slide by Steve Seitz
![Page 44: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/44.jpg)
Image sub-sampling
1/4 (2x zoom) 1/8 (4x zoom)
Aliasing! What do we do?
1/2
Slide by Steve Seitz
![Page 45: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/45.jpg)
Anti-aliasing
Get rid of all frequencies that are greater than half the new sampling frequency
– Will lose information– But it’s better than aliasing– Apply a smoothing filter
Slide by Derek Hoiem
![Page 46: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/46.jpg)
Algorithm for downsampling by factor of 2
1. Start with image(h, w)2. Apply low-pass filter
im_blur = imfilter(image, fspecial(‘gaussian’, 7, 1))
3. Sample every other pixelim_small = im_blur(1:2:end, 1:2:end);
Slide by Derek Hoiem
![Page 47: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/47.jpg)
Gaussian (lowpass) pre-filtering
G 1/4
G 1/8
Gaussian 1/2
Solution: filter the image, then subsample
Slide by Steve Seitz
![Page 48: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/48.jpg)
Subsampling with Gaussian pre-filtering
G 1/4 G 1/8Gaussian 1/2
Slide by Steve Seitz
![Page 49: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/49.jpg)
Compare with...
1/4 (2x zoom) 1/8 (4x zoom)1/2
Slide by Steve Seitz
![Page 50: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/50.jpg)
Image Pyramids
Known as a Gaussian Pyramid [Burt and Adelson, 1983]• In computer graphics, a mip map [Williams, 1983]• A precursor to wavelet transform
Slide by Steve Seitz
![Page 51: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/51.jpg)
A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose
Figure from David Forsyth
![Page 52: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/52.jpg)
Coarse-to-fine Image Registration1. Compute Gaussian pyramid2. Align with coarse pyramid3. Successively align with finer
pyramids– Search smaller range
Why is this faster?
![Page 53: Computational Photography CSE 590](https://reader036.vdocument.in/reader036/viewer/2022062316/56816959550346895de10a01/html5/thumbnails/53.jpg)
What are pyramids good for?• Improve Search
– Search over translations• Like HW1• Classic coarse-to-fine strategy
– Search over scale• Template matching• E.g. find a face at different scales
• Pre-computation– Need to access image at different blur levels– Useful for texture mapping at different resolutions (called
mip-mapping) – Useful for compositing/blending – coming soon!