image stitching richard szeliskiimage stitching2 panoramic image mosaics full screen panoramas...
TRANSCRIPT
Image Stitching
Richard Szeliski Image Stitching 2
Panoramic Image Mosaics
Full screen panoramas (cubic): http://www.panoramas.dk/ Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html
2003 New Years Eve: http://www.panoramas.dk/fullscreen3/f1.html
Richard Szeliski Image Stitching 3
Gigapixel panoramas & images
Mapping / Tourism / WWT
Medical Imaging
Richard Szeliski Image Stitching 4
Image Mosaics
+ + … +=
Goal: Stitch together several images into a seamless composite
Richard Szeliski Image Stitching 5
Today’s lecture
Image alignment and stitching• motion models• image warping• point-based alignment• complete mosaics (global alignment)• compositing and blending• ghost and parallax removal
Richard Szeliski Image Stitching 6
Readings
• Szeliski, CVAA:• Chapter 3.6: Image warping• Chapter 5.1: Feature-based alignment (in preparation)• Chapter 9.1: Motion models• Chapter 9.2: Global alignment• Chapter 9.3: Compositing
• Recognizing Panoramas, Brown & Lowe, ICCV’2003• Szeliski & Shum, SIGGRAPH'97
Motion models
Richard Szeliski Image Stitching 8
Motion models
What happens when we take two images with a camera and try to align them?
• translation?• rotation?• scale?• affine?• perspective?• VideoMosaic
Image Warping
Richard Szeliski Image Stitching 10
Image Warping
image filtering: change range of imageg(x) = h(f(x))
image warping: change domain of imageg(x) = f(h(x))
f
x
hf
x
f
x
hf
x
Richard Szeliski Image Stitching 11
Image Warping
image filtering: change range of imageg(x) = h(f(x))
image warping: change domain of imageg(x) = f(h(x))
h
h
f
f g
g
Richard Szeliski Image Stitching 12
Parametric (global) warping
Examples of parametric warps:
translation rotation aspect
affineperspective
cylindrical
2D coordinate transformations
Richard Szeliski Image Stitching 13
2D coordinate transformations
Translation
Rotation + translation
Similarity
Affine
Richard Szeliski Image Stitching 14
, ( , )x yt t x = x + t t
cos sin, ( , )
sin cos x yt t
x = Rx + t
R t
1
x
y
xa b t
s yb a t
x = Rx + t
00 01 02
10 11 12 1
xa a a
ya a a
x =
Homography
A: Projective – mapping between any two PPs with the same center of projection• rectangle should map to arbitrary quadrilateral • parallel lines aren’t• but must preserve straight lines• same as: project, rotate, reproject
called Homography
PP2
PP1
1yx
*********
wwy'wx'
H pp’ To apply a homography H
• Compute p’ = Hp (regular matrix multiply)
• Convert p’ from homogeneous to image coordinates
2D coordinate transformations
Projective transformation or Homography
Normalized homogeneous
Richard Szeliski Image Stitching 16
x = Hx
Homogeneous, only defined up to a scale
sH H
00 01 02
20 21 22
h x h y hx
h x h y h
10 11 12
20 21 22
h x h y hy
h x h y h
1
x
y
x
Richard Szeliski Image Stitching 17
2D coordinate transformations
Richard Szeliski Image Stitching 18
Image Warping
Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))?
f(x) g(x’)x x’
h(x)
Richard Szeliski Image Stitching 19
Forward Warping
Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
f(x) g(x’)x x’
h(x)
• What if pixel lands “between” two pixels?
Richard Szeliski Image Stitching 20
Forward Warping
Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
f(x) g(x’)x x’
h(x)
• What if pixel lands “between” two pixels?
Forward WarpingNon-integer x’• Nearest integer coordinate
• Aliasing• Jumping
• Splatting • Distribute to 4 neighbors (weighted) and normalize later • Both aliasing and blur
Cracks and holes• Filling hole with nearby neighbors• Further aliasing and blurring
21
Richard Szeliski Image Stitching 22
Inverse Warping
Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x)
f(x) g(x’)x x’
h(x)
• What if pixel comes from “between” two pixels?
Richard Szeliski Image Stitching 23
Inverse Warping
Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x)
• What if pixel comes from “between” two pixels?• Answer: resample color value from
interpolated (prefiltered) source image
f(x) g(x’)x x’
Richard Szeliski Image Stitching 24
Interpolation
Possible interpolation filters:• nearest neighbor• bilinear• bicubic (interpolating)• sinc / FIR
Needed to prevent “jaggies” and “texture crawl”
Interpolation
Bilinear
Richard Szeliski Image Stitching 25
0
1 0
u uU
u u
0 0( , )u v 1 0( , )u v
1 1( , )u v0 1( , )u v
1f 2f
3f 4f
( , )u v
f
0
1 0
v vV
v v
0 1
3 4
(1 )
(1 )
(1 )
a
b
a b
f f U f U
f f U f U
f f V f V
Bicubic
32
2
1 ( 3) ( 2) 1
( ) ( 1)( 2) 1 2
0
a x a x if x
h x a x x if x
otherwise
ɑ: the derivative at x=1
Interpolation
Window sinc functions
Richard Szeliski Image Stitching 26
Image Interpolation
Richard Szeliski Image Stitching 27
All perform high-accuracy low-pass filtering while preserve details and avoid aliasing
Bilinear for speedBicubic and windowed sinc for better visual quality
Decimation
Interpolation: increase the image resolution
Decimation (downsampling): reduce resolution
Richard Szeliski Image Stitching 28
Samples are convolved with a low-pass filter before downsampled to avoid aliasing
Decimation
Sample rate
Richard Szeliski Image Stitching 29
r
,
1( , ) ( , ) ,
k l
k kg i j f k l h i j
r r r
( , )h k l : same kernel as interpolation
Commonly use r=2
Decimation
Richard Szeliski Image Stitching 30
Frequency response
Decimation
• Linear filter gives a relatively poor response• Binomial filter cuts off a lot of frequencies but is
useful for computer vision analysis pyramid• Cubic filter: a=-1 filter has a shaper fall-off than the
a=-0.5 filter• Cosine-windowed sinc function• QMF-9 filter for wavelet denoising and aliases a fair
amount• 9/7 analysis filter form JPEG2000
Richard Szeliski Image Stitching 31
Richard Szeliski Image Stitching 32
Prefiltering
Essential for downsampling (decimation) to prevent aliasing
MIP-mapping [Williams’83]:1. build pyramid (but what decimation filter?):
– block averaging– Burt & Adelson (5-tap binomial)– 7-tap wavelet-based filter (better)
2. trilinear interpolation– bilinear within each 2 adjacent levels– linear blend between levels (determined by pixel size)
Richard Szeliski Image Stitching 33
Prefiltering
Essential for downsampling (decimation) to prevent aliasing
Other possibilities:• summed area tables• elliptically weighted Gaussians (EWA)
[Heckbert’86]
Resmaple on interpolated signal
To avoid blurring or aliasing
Richard Szeliski Image Stitching 34
Motion models (reprise)
Richard Szeliski Image Stitching 36
Motion models
Translation
2 unknowns
Affine
6 unknowns
Perspective
8 unknowns
3D rotation
3 unknowns
Planar perspective motion
• Simplest case:• Overlapping photographic prints, scanned pieces• Translation and rotation are usually adequate
Richard Szeliski Image Stitching 37
1 0 x Rx t
Four unknowns
Planar perspective motion
• Planar Homography
Richard Szeliski Image Stitching 38
00 01 02
20 21 22
h x h y hx
h x h y h
10 11 12
20 21 22
h x h y hy
h x h y h
Planar Perspective Motion
• Extract features from images• Match features using RANSAC and get inliers• Estimate a homography H given pairs of
corresponding features• Least squares fitting
Richard Szeliski Image Stitching 39
Solving for homography
Richard Szeliski Image Stitching 40
1
y
x
ihg
fed
cba
w
wy'
wx'p’ = Hp
Can set scale factor i=1. So, there are 8 unknowns.
Set up a system of linear equations:
Ah = b
where vector of unknowns h = [a,b,c,d,e,f,g,h]T
Need at least 8 eqs, but the more the better…
Solve for h. If overconstrained, solve using least-squares 2
min bAh
Richard Szeliski Image Stitching 41
Image warping with homographies
image plane in front image plane below
Richard Szeliski Image Stitching 42
Plane perspective mosaics
• 8-parameter generalization of affine motion– works for pure rotation or planar surfaces
• Limitations:– local minima – slow convergence– difficult to control interactively
Richard Szeliski Image Stitching 43
Rotational mosaics
• Directly optimize rotation and focal length
• Advantages:– ability to build full-view
panoramas– easier to control interactively– more stable and accurate
estimates
GrandCanyon panorama
Richard Szeliski Image Stitching 44
3D → 2D Perspective Projection
u
(Xc,Yc,Zc)
ucf
Rotational panoramas
Setting
Richard Szeliski Image Stitching 45
0 1t = t = 0 ( , ,1)K diag f f
With known f0, f1, solve R10
Rotational panoramas
• More stable for large-scale image stitching algorithms than homography
• How to solve the unknowns?• Update R10 with incremental rotation matrix R(w)
to current estimate R10
Richard Szeliski Image Stitching 46
Richard Szeliski Image Stitching 47
Rotations and quaternionsHow do we represent rotation matrices?
1. Axis / angle (n,θ)R = I + sinθ [n] + (1- cosθ) [n]
2 (Rodriguez Formula), with [n] = cross product matrix (see paper)
2. Unit quaternions [Shoemake SIGG’85]q = (n sinθ/2, cosθ/2) = (w,s)quaternion multiplication (division is easy)q0 q1 = (s1w0+ s0w1, s0 s1-w0·w1)
Richard Szeliski Image Stitching 48
Incremental rotation update
1. Small angle approximationΔR = I + sinθ [n] + (1- cosθ) [n]
2
≈ θ [n] = [ω] linear in ω
2. Update original R matrixR ← R ΔR
Rotational panoramasHow to solve the unknowns of R10?
Compute the incremental rotation vector w,
Richard Szeliski Image Stitching 49
Richard Szeliski Image Stitching 50
Rotational mosaic
Projection equations
1. Project from image to 3D ray
(x0,y0,z0) = (u0-uc,v0-vc,f)
2. Rotate the ray by camera motion
(x1,y1,z1) = R01 (x0,y0,z0)
3. Project back into new (source) image
(u1,v1) = (fx1/z1+uc,fy1/z1+vc)
Gap Closing
• Estimate a series of rotation matrices and focal lengths to create large panoramas
• Unfortunately, accumulated errors• Hard to estimate accurate focal length• Rarely produce a closed 360 pararoma
Richard Szeliski Image Stitching 51
Gap Closing
• Estimate a series of rotation matrices and focal lengths to create large panoramas
• Unfortunately, accumulated errors• Rarely produce a closed 360 pararoma
Richard Szeliski Image Stitching 52
Wrong focal length: f=510; correct focal length f=468
Gap Closing
• Matching the first image and the last one• Compute the gap angle • Distribute the gap angle evenly across the
whole sequence• Modify rotations by • Update focal length
• Only works for 1D panorama where the camera is continuously turning in the same direction
Richard Szeliski Image Stitching 53
g
/g imageN
' (1 / 360 )gf f
Richard Szeliski Image Stitching 56
mosaic PP
Image reprojection
The mosaic has a natural interpretation in 3D• The images are reprojected onto a common plane• The mosaic is formed on this plane
Image Mosaics (Stitching)
[Szeliski & Shum, SIGGRAPH’97]
[Szeliski, FnT CVCG, 2006]
Richard Szeliski Image Stitching 59
Image Mosaics (stitching)
Blend together several overlapping images into one seamless mosaic (composite)
+ + … +=
Richard Szeliski Image Stitching 60
Mosaics for Video CodingConvert masked images into a background sprite
for content-based coding
+ + +
=
Richard Szeliski Image Stitching 61
Establishing correspondences
1. Direct method:• Use generalization of affine motion model
[Szeliski & Shum ’97]
2. Feature-based method• Extract features, match, find consisten inliers
[Lowe ICCV’99; Schmid ICCV’98,Brown&Lowe ICCV’2003]
• Compute R from correspondences(absolute orientation)
Richard Szeliski Image Stitching 63
Stitching demo
Richard Szeliski Image Stitching 64
Panoramas
What if you want a 360 field of view?
mosaic Projection Cylinder
Richard Szeliski Image Stitching 65
Cylindrical panoramas
Steps• Reproject each image onto a cylinder• Blend • Output the resulting mosaic
Richard Szeliski Image Stitching 66
f = 180 (pixels)
Cylindrical Panoramas
Map image to cylindrical or spherical coordinates• need known focal length
Image 384x300 f = 380f = 280
Richard Szeliski Image Stitching 67
Determining the focal length
1. Initialize from homography H(see text or [SzSh’97])
2. Use camera’s EXIF tags (approx.)
3. Use a tape measure
4. Ask your instructor
1m4m
Richard Szeliski Image Stitching 68
3D → 2D Perspective Projection
u
(Xc,Yc,Zc)
ucf
Richard Szeliski Image Stitching 69
• Map 3D point (X,Y,Z) onto cylinder
Cylindrical projection
XY
Z
unit cylinder
unwrapped cylinder
• Convert to cylindrical coordinates
cylindrical image
• Convert to cylindrical image coordinates
– s defines size of the final image
Richard Szeliski Image Stitching 70
Cylindrical warping
Given focal length f and image center (xc,yc)
X
YZ
(X,Y,Z)
(sin,h,cos)
Richard Szeliski Image Stitching 71
Spherical warping
Given focal length f and image center (xc,yc)
X
YZ
(x,y,z)
(sinθcosφ,cosθcosφ,sinφ)
cos φ
φ
cos θ cos φ
sin φ
Richard Szeliski Image Stitching 73
Radial distortion
Correct for “bending” in wide field of view lenses
Richard Szeliski Image Stitching 74
Fisheye lens
Extreme “bending” in ultra-wide fields of view
Richard Szeliski Image Stitching 75
Other types of mosaics
Can mosaic onto any surface if you know the geometry• See NASA’s Visible Earth project for some stunning earth
mosaics– http://earthobservatory.nasa.gov/Newsroom/BlueMarble/
Richard Szeliski Image Stitching 76
Slit images
y-t slices of the video volume are known as slit images• take a single column of pixels from each input image
Richard Szeliski Image Stitching 77
Slit images: cyclographs
Richard Szeliski Image Stitching 78
Slit images: photofinish
Richard Szeliski Image Stitching 79
Final thought:What is a “panorama”?
Tracking a subject
Repeated (best) shots
Multiple exposures
“Infer” what photographer wants?