image stitching richard szeliskiimage stitching2 panoramic image mosaics full screen panoramas...

Image Stitching

Richard Szeliski Image Stitching 2

Panoramic Image Mosaics

Full screen panoramas (cubic): http://www.panoramas.dk/ Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html

2003 New Years Eve: http://www.panoramas.dk/fullscreen3/f1.html

Richard Szeliski Image Stitching 3

Gigapixel panoramas & images

Mapping / Tourism / WWT

Medical Imaging

Richard Szeliski Image Stitching 4

Image Mosaics

+ + … +=

Goal: Stitch together several images into a seamless composite

Richard Szeliski Image Stitching 5

Today’s lecture

Image alignment and stitching• motion models• image warping• point-based alignment• complete mosaics (global alignment)• compositing and blending• ghost and parallax removal

Richard Szeliski Image Stitching 6

Readings

• Szeliski, CVAA:• Chapter 3.6: Image warping• Chapter 5.1: Feature-based alignment (in preparation)• Chapter 9.1: Motion models• Chapter 9.2: Global alignment• Chapter 9.3: Compositing

• Recognizing Panoramas, Brown & Lowe, ICCV’2003• Szeliski & Shum, SIGGRAPH'97

Motion models

Richard Szeliski Image Stitching 8

Motion models

What happens when we take two images with a camera and try to align them?

• translation?• rotation?• scale?• affine?• perspective?• VideoMosaic

Image Warping

Richard Szeliski Image Stitching 10

Image Warping

image filtering: change range of imageg(x) = h(f(x))

image warping: change domain of imageg(x) = f(h(x))

f

x

hf

x

f

x

hf

x

Richard Szeliski Image Stitching 11

Image Warping

image filtering: change range of imageg(x) = h(f(x))

image warping: change domain of imageg(x) = f(h(x))

h

h

f

f g

g

Richard Szeliski Image Stitching 12

Parametric (global) warping

Examples of parametric warps:

translation rotation aspect

affineperspective

cylindrical

2D coordinate transformations

Richard Szeliski Image Stitching 13

2D coordinate transformations

Translation

Rotation + translation

Similarity

Affine

Richard Szeliski Image Stitching 14

, ( , )x yt t x = x + t t

cos sin, ( , )

sin cos x yt t

x = Rx + t

R t

1

x

y

xa b t

s yb a t

x = Rx + t

00 01 02

10 11 12 1

xa a a

ya a a

x =

Homography

A: Projective – mapping between any two PPs with the same center of projection• rectangle should map to arbitrary quadrilateral • parallel lines aren’t• but must preserve straight lines• same as: project, rotate, reproject

called Homography

PP2

PP1

1yx

*********

wwy'wx'

H pp’ To apply a homography H

• Compute p’ = Hp (regular matrix multiply)

• Convert p’ from homogeneous to image coordinates

2D coordinate transformations

Projective transformation or Homography

Normalized homogeneous

Richard Szeliski Image Stitching 16

x = Hx

Homogeneous, only defined up to a scale

sH H

00 01 02

20 21 22

h x h y hx

h x h y h

10 11 12

20 21 22

h x h y hy

h x h y h

1

x

y

x

Richard Szeliski Image Stitching 17

2D coordinate transformations

Richard Szeliski Image Stitching 18

Image Warping

Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))?

f(x) g(x’)x x’

h(x)

Richard Szeliski Image Stitching 19

Forward Warping

Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)

f(x) g(x’)x x’

h(x)

• What if pixel lands “between” two pixels?

Richard Szeliski Image Stitching 20

Forward Warping

Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)

f(x) g(x’)x x’

h(x)

• What if pixel lands “between” two pixels?

Forward WarpingNon-integer x’• Nearest integer coordinate

• Aliasing• Jumping

• Splatting • Distribute to 4 neighbors (weighted) and normalize later • Both aliasing and blur

Cracks and holes• Filling hole with nearby neighbors• Further aliasing and blurring

21

Richard Szeliski Image Stitching 22

Inverse Warping

Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x)

f(x) g(x’)x x’

h(x)

• What if pixel comes from “between” two pixels?

Richard Szeliski Image Stitching 23

Inverse Warping

Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x)

• What if pixel comes from “between” two pixels?• Answer: resample color value from

interpolated (prefiltered) source image

f(x) g(x’)x x’

Richard Szeliski Image Stitching 24

Interpolation

Possible interpolation filters:• nearest neighbor• bilinear• bicubic (interpolating)• sinc / FIR

Needed to prevent “jaggies” and “texture crawl”

Interpolation

Bilinear

Richard Szeliski Image Stitching 25

0

1 0

u uU

u u

0 0( , )u v 1 0( , )u v

1 1( , )u v0 1( , )u v

1f 2f

3f 4f

( , )u v

f

0

1 0

v vV

v v

0 1

3 4

(1 )

(1 )

(1 )

a

b

a b

f f U f U

f f U f U

f f V f V

Bicubic

32

2

1 ( 3) ( 2) 1

( ) ( 1)( 2) 1 2

0

a x a x if x

h x a x x if x

otherwise

ɑ: the derivative at x=1

Interpolation

Window sinc functions

Richard Szeliski Image Stitching 26

Image Interpolation

Richard Szeliski Image Stitching 27

All perform high-accuracy low-pass filtering while preserve details and avoid aliasing

Bilinear for speedBicubic and windowed sinc for better visual quality

Decimation

Interpolation: increase the image resolution

Decimation (downsampling): reduce resolution

Richard Szeliski Image Stitching 28

Samples are convolved with a low-pass filter before downsampled to avoid aliasing

Decimation

Sample rate

Richard Szeliski Image Stitching 29

r

,

1( , ) ( , ) ,

k l

k kg i j f k l h i j

r r r

( , )h k l : same kernel as interpolation

Commonly use r=2

Decimation

Richard Szeliski Image Stitching 30

Frequency response

Decimation

• Linear filter gives a relatively poor response• Binomial filter cuts off a lot of frequencies but is

useful for computer vision analysis pyramid• Cubic filter: a=-1 filter has a shaper fall-off than the

a=-0.5 filter• Cosine-windowed sinc function• QMF-9 filter for wavelet denoising and aliases a fair

amount• 9/7 analysis filter form JPEG2000

Richard Szeliski Image Stitching 31

Richard Szeliski Image Stitching 32

Prefiltering

Essential for downsampling (decimation) to prevent aliasing

MIP-mapping [Williams’83]:1. build pyramid (but what decimation filter?):

– block averaging– Burt & Adelson (5-tap binomial)– 7-tap wavelet-based filter (better)

2. trilinear interpolation– bilinear within each 2 adjacent levels– linear blend between levels (determined by pixel size)

Richard Szeliski Image Stitching 33

Prefiltering

Essential for downsampling (decimation) to prevent aliasing

Other possibilities:• summed area tables• elliptically weighted Gaussians (EWA)

[Heckbert’86]

Resmaple on interpolated signal

To avoid blurring or aliasing

Richard Szeliski Image Stitching 34

Motion models (reprise)

Richard Szeliski Image Stitching 36

Motion models

Translation

2 unknowns

Affine

6 unknowns

Perspective

8 unknowns

3D rotation

3 unknowns

Planar perspective motion

• Simplest case:• Overlapping photographic prints, scanned pieces• Translation and rotation are usually adequate

Richard Szeliski Image Stitching 37

1 0 x Rx t

Four unknowns

Planar perspective motion

• Planar Homography

Richard Szeliski Image Stitching 38

00 01 02

20 21 22

h x h y hx

h x h y h

10 11 12

20 21 22

h x h y hy

h x h y h

Planar Perspective Motion

• Extract features from images• Match features using RANSAC and get inliers• Estimate a homography H given pairs of

corresponding features• Least squares fitting

Richard Szeliski Image Stitching 39

Solving for homography

Richard Szeliski Image Stitching 40

1

y

x

ihg

fed

cba

w

wy'

wx'p’ = Hp

Can set scale factor i=1. So, there are 8 unknowns.

Set up a system of linear equations:

Ah = b

where vector of unknowns h = [a,b,c,d,e,f,g,h]T

Need at least 8 eqs, but the more the better…

Solve for h. If overconstrained, solve using least-squares 2

min bAh

Richard Szeliski Image Stitching 41

Image warping with homographies

image plane in front image plane below

Richard Szeliski Image Stitching 42

Plane perspective mosaics

• 8-parameter generalization of affine motion– works for pure rotation or planar surfaces

• Limitations:– local minima – slow convergence– difficult to control interactively

Richard Szeliski Image Stitching 43

Rotational mosaics

• Directly optimize rotation and focal length

• Advantages:– ability to build full-view

panoramas– easier to control interactively– more stable and accurate

estimates

GrandCanyon panorama

Richard Szeliski Image Stitching 44

3D → 2D Perspective Projection

u

(Xc,Yc,Zc)

ucf

Rotational panoramas

Setting

Richard Szeliski Image Stitching 45

0 1t = t = 0 ( , ,1)K diag f f

With known f0, f1, solve R10

Rotational panoramas

• More stable for large-scale image stitching algorithms than homography

• How to solve the unknowns?• Update R10 with incremental rotation matrix R(w)

to current estimate R10

Richard Szeliski Image Stitching 46

Richard Szeliski Image Stitching 47

Rotations and quaternionsHow do we represent rotation matrices?

1. Axis / angle (n,θ)R = I + sinθ [n] + (1- cosθ) [n]

2 (Rodriguez Formula), with [n] = cross product matrix (see paper)

2. Unit quaternions [Shoemake SIGG’85]q = (n sinθ/2, cosθ/2) = (w,s)quaternion multiplication (division is easy)q0 q1 = (s1w0+ s0w1, s0 s1-w0·w1)

Richard Szeliski Image Stitching 48

Incremental rotation update

1. Small angle approximationΔR = I + sinθ [n] + (1- cosθ) [n]

2

≈ θ [n] = [ω] linear in ω

2. Update original R matrixR ← R ΔR

Rotational panoramasHow to solve the unknowns of R10?

Compute the incremental rotation vector w,

Richard Szeliski Image Stitching 49

Richard Szeliski Image Stitching 50

Rotational mosaic

Projection equations

1. Project from image to 3D ray

(x0,y0,z0) = (u0-uc,v0-vc,f)

2. Rotate the ray by camera motion

(x1,y1,z1) = R01 (x0,y0,z0)

3. Project back into new (source) image

(u1,v1) = (fx1/z1+uc,fy1/z1+vc)

Gap Closing

• Estimate a series of rotation matrices and focal lengths to create large panoramas

• Unfortunately, accumulated errors• Hard to estimate accurate focal length• Rarely produce a closed 360 pararoma

Richard Szeliski Image Stitching 51

Gap Closing

• Estimate a series of rotation matrices and focal lengths to create large panoramas

• Unfortunately, accumulated errors• Rarely produce a closed 360 pararoma

Richard Szeliski Image Stitching 52

Wrong focal length: f=510; correct focal length f=468

Gap Closing

• Matching the first image and the last one• Compute the gap angle • Distribute the gap angle evenly across the

whole sequence• Modify rotations by • Update focal length

• Only works for 1D panorama where the camera is continuously turning in the same direction

Richard Szeliski Image Stitching 53

g

/g imageN

' (1 / 360 )gf f

Richard Szeliski Image Stitching 56

mosaic PP

Image reprojection

The mosaic has a natural interpretation in 3D• The images are reprojected onto a common plane• The mosaic is formed on this plane

Image Mosaics (Stitching)

[Szeliski & Shum, SIGGRAPH’97]

[Szeliski, FnT CVCG, 2006]

Richard Szeliski Image Stitching 59

Image Mosaics (stitching)

Blend together several overlapping images into one seamless mosaic (composite)

+ + … +=

Richard Szeliski Image Stitching 60

Mosaics for Video CodingConvert masked images into a background sprite

for content-based coding

+ + +

=

Richard Szeliski Image Stitching 61

Establishing correspondences

1. Direct method:• Use generalization of affine motion model

[Szeliski & Shum ’97]

2. Feature-based method• Extract features, match, find consisten inliers

[Lowe ICCV’99; Schmid ICCV’98,Brown&Lowe ICCV’2003]

• Compute R from correspondences(absolute orientation)

Richard Szeliski Image Stitching 63

Stitching demo

Richard Szeliski Image Stitching 64

Panoramas

What if you want a 360 field of view?

mosaic Projection Cylinder

Richard Szeliski Image Stitching 65

Cylindrical panoramas

Steps• Reproject each image onto a cylinder• Blend • Output the resulting mosaic

Richard Szeliski Image Stitching 66

f = 180 (pixels)

Cylindrical Panoramas

Map image to cylindrical or spherical coordinates• need known focal length

Image 384x300 f = 380f = 280

Richard Szeliski Image Stitching 67

Determining the focal length

1. Initialize from homography H(see text or [SzSh’97])

2. Use camera’s EXIF tags (approx.)

3. Use a tape measure

4. Ask your instructor

1m4m

Richard Szeliski Image Stitching 68

3D → 2D Perspective Projection

u

(Xc,Yc,Zc)

ucf

Richard Szeliski Image Stitching 69

• Map 3D point (X,Y,Z) onto cylinder

Cylindrical projection

XY

Z

unit cylinder

unwrapped cylinder

• Convert to cylindrical coordinates

cylindrical image

• Convert to cylindrical image coordinates

– s defines size of the final image

Richard Szeliski Image Stitching 70

Cylindrical warping

Given focal length f and image center (xc,yc)

X

YZ

(X,Y,Z)

(sin,h,cos)

Richard Szeliski Image Stitching 71

Spherical warping

Given focal length f and image center (xc,yc)

X

YZ

(x,y,z)

(sinθcosφ,cosθcosφ,sinφ)

cos φ

φ

cos θ cos φ

sin φ

Richard Szeliski Image Stitching 73

Radial distortion

Correct for “bending” in wide field of view lenses

Richard Szeliski Image Stitching 74

Fisheye lens

Extreme “bending” in ultra-wide fields of view

Richard Szeliski Image Stitching 75

Other types of mosaics

Can mosaic onto any surface if you know the geometry• See NASA’s Visible Earth project for some stunning earth

mosaics– http://earthobservatory.nasa.gov/Newsroom/BlueMarble/

Richard Szeliski Image Stitching 76

Slit images

y-t slices of the video volume are known as slit images• take a single column of pixels from each input image

Richard Szeliski Image Stitching 77

Slit images: cyclographs

Richard Szeliski Image Stitching 78

Slit images: photofinish

Richard Szeliski Image Stitching 79

Final thought:What is a “panorama”?

Tracking a subject

Repeated (best) shots

Multiple exposures

“Infer” what photographer wants?