![Page 1: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/1.jpg)
Generalized Mosaics
Yoav Y. Schechner,
Shree Nayar
Department of Computer Science
Columbia University
![Page 2: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/2.jpg)
Mosaics
Processing
![Page 3: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/3.jpg)
Redundant Measurements
Generalized Mosaicing: Yoav Schechner and Shree Nayar
![Page 4: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/4.jpg)
Scanning with Less Redundancy
Mantis Shrimp
TakataC
Different rows = Different optical characteristics
![Page 5: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/5.jpg)
Generalized Mosaicing
Field of View
BrightnessDynamic Range Spectrum
Depth of Field
Polarization
camera
Spatially varying
filter
Schechner, Nayar, Generalized mosaics
![Page 6: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/6.jpg)
Mosaic + High Dynamic Range
88 - 18,794
![Page 7: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/7.jpg)
Fusion of Measurements
)(Mask),(),(Raw xyxIyxk
1
22 1
k kI
I
k k
k
I
III
22
Maximum-Likelihood Solution
),( ),( )(Mask
),(RawyxIyxI
x
yxkk
k
Schechner, Nayar, Generalized mosaics
![Page 8: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/8.jpg)
8 bit camera Dynamic range as 16 bits
Generalized Mosaicing: Yoav Schechner and Shree Nayar
8 bits (almost)(pixels)
-2
-4
-6
-8 x
log M2
Log of the mask
==1 10-61 10-2
![Page 9: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/9.jpg)
M
x
average row
frames y
frameyxx ),(Raw )(Mask
),(Mask),(),(Raw yxyxIyxframe
Mask Self-CalibrationUnknown filter (vignetting)
consistencyconstraints
0Raw)(Mask Raw)(Mask1221 xx
MaskttMaskmin AA
Consistency, Smoothness A
Schechner, Nayar, Generalized mosaics
![Page 10: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/10.jpg)
Image Registration
2pair ingcorrespondeach
pixels all
2total DIFFDIFF
2total DIFF:Minimize
2pair ingcorrespondeach
pixels all
2total DIFFDIFF
Schechner, Nayar, Generalized mosaics
![Page 11: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/11.jpg)
)(Mask),(),(Raw xyxIyxk
2pair ingcorrespondeach
pixels all
2total
DIFFDIFF :Minimize
Bias towards “no motion”
Schechner, Nayar, Generalized mosaics
Image Registration
![Page 12: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/12.jpg)
)(Mask),(),(Raw xyxIyxk
),( )(Mask
),(RawyxI
x
yxk
k
Raw
Mask
Uncertainty
),( yxIk
1
22
22 1 where ,
k kk k
k
II
I
III
2
22
2
2
21
12pair pixel
III
III
DIFF
Schechner, Nayar, Generalized mosaics
Image Registration
![Page 13: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/13.jpg)
Registration: Standard “Coarse to Fine”
5 pixels
10 pix 9 pix
18 pix 19 pix
4 pixels
2 pixelsmotion of
However, we need: multiscale UNCERTAINTIES 2
22
2
2
21
12pair pixel
III
III
DIFF
Schechner, Nayar, Generalized mosaics
![Page 14: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/14.jpg)
Maximum-Likelihood Pyramid At each level L , for each pixel
LL II , estimate
,
),( ),(yx
yxIyxwI
,
),( yx
yxw
,
),( 2 yx
yxwI-1
),(
),(),(yxI
yxayxw
2
Schechner, Nayar, Generalized mosaics
![Page 15: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/15.jpg)
Max-Likelihood Pyramid
Gaussian Pyramid1
22
22
)(
1
)(
)( ,
xx xII
xI
xIII
x
)(xg
)1(I )2(I )3(I )4(I )5(I
x
xIxgI )()(
x )1(I )2(I )3(I )4(I )5(I )1(I )2(I )3(I )4(I )5(I
ML estimation
x
xIxnI )()(
)(
)()(2 xI
xgx
1][ )(
xn
1
2 )(
x
xI
![Page 16: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/16.jpg)
log I
Inte
nsit
y ra
nge
Spatial range x
Generalized Mosaic
x
log I
Inte
nsit
y ra
nge
Spatial range
frame
fram
e
Mosaic
Schechner, Nayar, Generalized mosaics
![Page 17: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/17.jpg)
Generalized Mosaicing: Yoav Schechner and Shree Nayar
spectral
y
x
Variable Spectral Filter
![Page 18: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/18.jpg)
MMuullttii SSppeeccttrraall Mosaic
500 700400 600 500 700400 600 500 700400 600
Generalized Mosaicing: Yoav Schechner and Shree Nayar
![Page 19: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/19.jpg)
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Rendering : Any Illumination
sunset fluorescent
![Page 20: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/20.jpg)
Halogeny
x
Illumination at a Glance
Fluorescenty
x
Illuminant spectrum
Extra information
![Page 21: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/21.jpg)
Spatially Varying Polarizer
Polarization Mosaicing: Yoav Schechner and Shree Nayar
![Page 22: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/22.jpg)
0
0.2
0.4
0.6
0.8
1
-30
60
90
30
0
-60
-90
o
o
o
o
o
o
polarizance
polarizing angle
o
Polarization Mosaicing: Yoav Schechner and Shree Nayar
transmittance
Spatially Varying Polarizer
![Page 23: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/23.jpg)
Raw images
Polarization Mosaicing: Yoav Schechner and Shree Nayar
![Page 24: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/24.jpg)
Polarization Mosaic
Schechner Shamir Kiryati
JOSA-A 2000
reflected structure
transmitted painting
![Page 25: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/25.jpg)
=
y
xDepth
Insert a prism behind the lens
Spatially Varying Focus
Generalized Mosaicing: Yoav Schechner and Shree Nayar
![Page 26: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/26.jpg)
![Page 27: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/27.jpg)
All FocusedAll Focused Mosaic
Generalized Mosaicing: Yoav Schechner and Shree Nayar
… and a Depth Map
![Page 28: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/28.jpg)
What else?
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Dynamic Range & Spectrum & Polarization etc.
Multi-dimensional Mosaics - Simultaneously
![Page 29: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/29.jpg)
Signal undergoing LPF
“Band limited” signal
Minimizing aliasing
Nyquist sampling rate
111111111111111
1?10111001??
Sampling Criteria
Generalized Mosaicing: Yoav Schechner and Shree Nayar
Number of samples per scene point
resolution
length Lbandwidth B
aperture D
22 )/()/(
2
BLD
M2
log11 M
1011100101
1011100???
M # samples
1 11/2 21/4 3
92561
101110????
![Page 30: Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University](https://reader034.vdocument.in/reader034/viewer/2022051621/56649ced5503460f949bae1b/html5/thumbnails/30.jpg)
MosaicGeneralized Mosaic
Spatially varying filter
Wide field of view
SpectrumI
Depth + Focus
Dynamic range88 – 18,794
Polarization