generalized mosaics yoav y. schechner, shree nayar department of computer science columbia...

Generalized Mosaics

Yoav Y. Schechner,

Shree Nayar

Department of Computer Science

Columbia University

Mosaics

Processing

Redundant Measurements

Generalized Mosaicing: Yoav Schechner and Shree Nayar

Scanning with Less Redundancy

Mantis Shrimp

TakataC

Different rows = Different optical characteristics

Generalized Mosaicing

Field of View

BrightnessDynamic Range Spectrum

Depth of Field

Polarization

camera

Spatially varying

filter

Schechner, Nayar, Generalized mosaics

Mosaic + High Dynamic Range

88 - 18,794

Fusion of Measurements

)(Mask),(),(Raw xyxIyxk

1

22 1

k kI

I

k k

k

I

III

22

Maximum-Likelihood Solution

),( ),( )(Mask

),(RawyxIyxI

x

yxkk

k

Schechner, Nayar, Generalized mosaics

8 bit camera Dynamic range as 16 bits

Generalized Mosaicing: Yoav Schechner and Shree Nayar

8 bits (almost)(pixels)

-2

-4

-6

-8 x

log M2

Log of the mask

==1 10-61 10-2

M

x

average row

frames y

frameyxx ),(Raw )(Mask

),(Mask),(),(Raw yxyxIyxframe

Mask Self-CalibrationUnknown filter (vignetting)

consistencyconstraints

0Raw)(Mask Raw)(Mask1221 xx

MaskttMaskmin AA

Consistency, Smoothness A

Schechner, Nayar, Generalized mosaics

Image Registration

2pair ingcorrespondeach

pixels all

2total DIFFDIFF

2total DIFF:Minimize

2pair ingcorrespondeach

pixels all

2total DIFFDIFF

Schechner, Nayar, Generalized mosaics

)(Mask),(),(Raw xyxIyxk

2pair ingcorrespondeach

pixels all

2total

DIFFDIFF :Minimize

Bias towards “no motion”

Schechner, Nayar, Generalized mosaics

Image Registration

)(Mask),(),(Raw xyxIyxk

),( )(Mask

),(RawyxI

x

yxk

k

Raw

Mask

Uncertainty

),( yxIk

1

22

22 1 where ,

k kk k

k

II

I

III

2

22

2

2

21

12pair pixel

III

III

DIFF

Schechner, Nayar, Generalized mosaics

Image Registration

Registration: Standard “Coarse to Fine”

5 pixels

10 pix 9 pix

18 pix 19 pix

4 pixels

2 pixelsmotion of

However, we need: multiscale UNCERTAINTIES 2

22

2

2

21

12pair pixel

III

III

DIFF

Schechner, Nayar, Generalized mosaics

Maximum-Likelihood Pyramid At each level L , for each pixel

LL II , estimate

,

),( ),(yx

yxIyxwI

,

),( yx

yxw

,

),( 2 yx

yxwI-1

),(

),(),(yxI

yxayxw

2

Schechner, Nayar, Generalized mosaics

Max-Likelihood Pyramid

Gaussian Pyramid1

22

22

)(

1

)(

)( ,

xx xII

xI

xIII

x

)(xg

)1(I )2(I )3(I )4(I )5(I

x

xIxgI )()(

x )1(I )2(I )3(I )4(I )5(I )1(I )2(I )3(I )4(I )5(I

ML estimation

x

xIxnI )()(

)(

)()(2 xI

xgx

1][ )(

xn

1

2 )(

x

xI

log I

Inte

nsit

y ra

nge

Spatial range x

Generalized Mosaic

x

log I

Inte

nsit

y ra

nge

Spatial range

frame

fram

e

Mosaic

Schechner, Nayar, Generalized mosaics

Generalized Mosaicing: Yoav Schechner and Shree Nayar

spectral

y

x

Variable Spectral Filter

MMuullttii SSppeeccttrraall Mosaic

500 700400 600 500 700400 600 500 700400 600

Generalized Mosaicing: Yoav Schechner and Shree Nayar

Generalized Mosaicing: Yoav Schechner and Shree Nayar

Rendering : Any Illumination

sunset fluorescent

Halogeny

x

Illumination at a Glance

Fluorescenty

x

Illuminant spectrum

Extra information

Spatially Varying Polarizer

Polarization Mosaicing: Yoav Schechner and Shree Nayar

0

0.2

0.4

0.6

0.8

1

-30

60

90

30

0

-60

-90

o

o

o

o

o

o

polarizance

polarizing angle

o

Polarization Mosaicing: Yoav Schechner and Shree Nayar

transmittance

Spatially Varying Polarizer

Raw images

Polarization Mosaicing: Yoav Schechner and Shree Nayar

Polarization Mosaic

Schechner Shamir Kiryati

JOSA-A 2000

reflected structure

transmitted painting

=

y

xDepth

Insert a prism behind the lens

Spatially Varying Focus

Generalized Mosaicing: Yoav Schechner and Shree Nayar

All FocusedAll Focused Mosaic

Generalized Mosaicing: Yoav Schechner and Shree Nayar

… and a Depth Map

What else?

Generalized Mosaicing: Yoav Schechner and Shree Nayar

Dynamic Range & Spectrum & Polarization etc.

Multi-dimensional Mosaics - Simultaneously

Signal undergoing LPF

“Band limited” signal

Minimizing aliasing

Nyquist sampling rate

111111111111111

1?10111001??

Sampling Criteria

Generalized Mosaicing: Yoav Schechner and Shree Nayar

Number of samples per scene point

resolution

length Lbandwidth B

aperture D

22 )/()/(

2

BLD

M2

log11 M

1011100101

1011100???

M # samples

1 11/2 21/4 3

92561

101110????

MosaicGeneralized Mosaic

Spatially varying filter

Wide field of view

SpectrumI

Depth + Focus

Dynamic range88 – 18,794

Polarization