CS 395/495-25: Spring 2005CS 395/495-25: Spring 2005

IBMR: Wrapping the world IBMR: Wrapping the world around your Camera:around your Camera:

Warps, Morphs, Panoramas, Warps, Morphs, Panoramas, and Mirror Spheresand Mirror Spheres

Jack TumblinJack Tumblinjet@cs.northwestern.edujet@cs.northwestern.edu

A Tour of Some Image ChangesA Tour of Some Image Changes

WARP DEMO: WARP DEMO: http://www.axlrosen.net/

MORPH DEMO: MORPH DEMO: http://www.dt.org/html/morph.html

PANORAMIC DEMO: PANORAMIC DEMO: http://www.panoguide.com/publish/software/go360.html

An Image Is…An Image Is…

2D Image:2D Image:A map of light A map of light

intensities intensities

Light + 3D Scene:Light + 3D Scene:Illumination, Illumination,

shape, movement, shape, movement, surface BRDF,… surface BRDF,…

Pos

ition

(x,y

)P

ositi

on(x

,y)A ‘Camera’:

?What are ALL the possibilities?

An Planar Projection Image Is…An Planar Projection Image Is…

Image PlaneImage PlaneI(x,y)I(x,y)

Ang

le(

Ang

le(

,, ))

Pos

ition

(x,y

)P

ositi

on(x

,y)

2D Image:2D Image:Collection of rays Collection of rays

through a point through a point

Light + 3D Scene:Light + 3D Scene:Illumination, Illumination,

shape, movement, shape, movement, surface BRDF,… surface BRDF,…

‘‘Center of Center of Projection’Projection’

(P(P33 or P or P22 Origin) Origin)

An Image Is…An Image Is…

Image PlaneImage PlaneI(x,y)I(x,y)

Pos

ition

(x,y

)P

ositi

on(x

,y)

2D Image:2D Image:Collection of light Collection of light

from 3D rays… from 3D rays…

Light + 3D Scene:Light + 3D Scene:Illumination, Illumination,

shape, movement, shape, movement, surface BRDF,… surface BRDF,…

ANY DESIRED 3D2D map!

(2D may be non-planar, ‘center of projection’

may not exist…

example:flatbed scanner

‘‘Projective Image’ : Projective Image’ : AngleAnglePosition MapPosition Map

• So far: So far: – planarplanar perspective projection perspective projection– PP22 projective transformations ( projective transformations (PP3 3 soon)soon)– how to find these transformations from images.how to find these transformations from images.

• But Planar Perspective Projection But Planar Perspective Projection is just is just ONE KINDONE KIND of mapping, and of mapping, and

• PP2 2 and Pand P3 3 linear projective transformation linear projective transformation HH is just is just ONE KINDONE KIND of image warping of image warping

H = HH = HSSHHAAHHPP

‘‘Image’ : AngleImage’ : AnglePosition MapPosition Map

• Many kinds of camera projections:Many kinds of camera projections:‘Fun-House Mirrors’‘Fun-House Mirrors’– Non-planar perspective projections:Non-planar perspective projections:

• Spherical, Conical, cylindrical, hyperbolic; Spherical, Conical, cylindrical, hyperbolic; even ‘centerless’ projections (Rademacher, SIGG 1998)even ‘centerless’ projections (Rademacher, SIGG 1998)

– How can we convert between them?How can we convert between them?

• All CamerasAll Cameras have imperfect geometry. have imperfect geometry.How can we dectect/correct the errors?How can we dectect/correct the errors?

• Warp, morph the 1Warp, morph the 1st st forms of IBMR? forms of IBMR? (1980s: I built warp/morph-based flight sim…)(1980s: I built warp/morph-based flight sim…)

Image Warping: General IdeaImage Warping: General Idea

2D2D2D continuous coordinate map, a ‘rubber sheet’2D continuous coordinate map, a ‘rubber sheet’Forward: find output address for each INPUT pt.Forward: find output address for each INPUT pt.

• One way: One way: Patch corner-by-patch corner; bilinear fill...Patch corner-by-patch corner; bilinear fill...

u = u(x,y)=u = u(x,y)=v = v(x,y)=…v = v(x,y)=…

xx

yy

uu

vv

1 y y1 y y22

1 1 x x xx22

‘‘Forward Forward Mapping’Mapping’

inputinput

outputoutput

Image Warping: General IdeaImage Warping: General Idea

2D2D2D continuous coordinate map, a ‘rubber sheet’2D continuous coordinate map, a ‘rubber sheet’Forward: find output address for each INPUT pt.Forward: find output address for each INPUT pt.

• One way: One way: Patch corner-by-patch corner; bilinear fill...Patch corner-by-patch corner; bilinear fill...

u = u(x,y)=u = u(x,y)=v = v(x,y)=…v = v(x,y)=…

xx

yy

uu

vv

1 y y1 y y22

1 1 x x xx22

‘‘Forward Forward Mapping’Mapping’

inputinput

outputoutput

?How can you find these two 3x3 matrices??How can you find these two 3x3 matrices?Hint: 4 input points, 4 output points, Hint: 4 input points, 4 output points,

Image Warping: General IdeaImage Warping: General Idea

2D2D2D continuous coordinate map, a ‘rubber sheet’2D continuous coordinate map, a ‘rubber sheet’Inverse: find input address for each OUTPUT pt.Inverse: find input address for each OUTPUT pt.

• One way: One way: Patch-corner by patch-corner; bilinear fill...Patch-corner by patch-corner; bilinear fill...

• x = x(u,v)=x = x(u,v)=y = y(u,v) = …y = y(u,v) = …

uu

vv

xx

yy

1 v v1 v v22

1 1 u u uu22

‘‘Inverse Inverse Mapping’Mapping’

inputinputoutputoutput

Image Warping: General IdeaImage Warping: General Idea

2D2D2D continuous coordinate map: ‘rubber sheet’2D continuous coordinate map: ‘rubber sheet’

PROBLEM:PROBLEM: inverse inverse requiredrequired sometimes (mess) sometimes (mess)

PROBLEM: PROBLEM: pixels aren’t continuous; sampling errorspixels aren’t continuous; sampling errors– aliasing (output pixels skip some input pixels)aliasing (output pixels skip some input pixels)– holes (input pixels skip some output pixels)holes (input pixels skip some output pixels)

OpenGL SolutionOpenGL Solution ::– use a textured mesh!use a textured mesh!

xx

yy

uu

vv

Image Warping: IBMR FormImage Warping: IBMR Form

2D2D2D continuous coordinate map: ‘rubber sheet’2D continuous coordinate map: ‘rubber sheet’

PROBLEM:PROBLEM: inverse inverse requiredrequired sometimes (mess) sometimes (mess)

IBMR Answer:IBMR Answer: H matrix is H matrix is (usually) invertible invertible

PROBLEM: PROBLEM: pixels aren’t continuous; sampling errorspixels aren’t continuous; sampling errors

IBMR Answer:IBMR Answer: Use vertices, not pixels: Use vertices, not pixels: let let OpenGL texture mapping keep the image “continuous”OpenGL texture mapping keep the image “continuous”

xx33

xx22

xx11

Further Reading on Warps:Further Reading on Warps:

• Many Misc. SIGGRAPH tutorials, many on-lineMany Misc. SIGGRAPH tutorials, many on-line

• T. Beier and S. Neely. T. Beier and S. Neely. Feature Based Image Feature Based Image MetamorphosisMetamorphosis. . Computer Graphics (Siggraph'92 Computer Graphics (Siggraph'92 Proceedings), Vol. 26 : pages 35--42, July 1992Proceedings), Vol. 26 : pages 35--42, July 1992

• George Wolberg George Wolberg Digital Image WarpingDigital Image Warping (book)©1990, Wiley & Sons(book)©1990, Wiley & Sons

• Jonas Gomes, Lucia Darsa, Bruno Costa, Luiz Velho, Jonas Gomes, Lucia Darsa, Bruno Costa, Luiz Velho, Warping and Morphing of Graphical ObjectsWarping and Morphing of Graphical Objects    ©1999 ©1999 Morgan Kaufmann Publishers Morgan Kaufmann Publishers

Image Warping: IBMR FormImage Warping: IBMR Form

• Projective transformation H in PProjective transformation H in P22 is a planar ‘warp’ is a planar ‘warp’• But many other But many other kindskinds of warps possible! of warps possible!

• P2 change object/camera position for viewing a plane P2 change object/camera position for viewing a plane • P3 changes object/camera postion for viewing a 3D shapeP3 changes object/camera postion for viewing a 3D shape• But shape changes are limited...But shape changes are limited...

Panoramas: Planar ‘Bow-Tie’Panoramas: Planar ‘Bow-Tie’

– For limited-size mosaics only (angle limits)For limited-size mosaics only (angle limits)– Find all Find all HH from correspondence in overlapped regions from correspondence in overlapped regions– Choose a (central) reference image Choose a (central) reference image (Zisserman pg. 196)(Zisserman pg. 196)

– Reproject, cross-dissolve in reference image planeReproject, cross-dissolve in reference image plane

Planar Panoramas in PPlanar Panoramas in P22 (or P(or P33))

• Choose a ‘reference’ image plane, extend itChoose a ‘reference’ image plane, extend it

• Add images: for each one, Add images: for each one, – find find H H from overlap correspondences (in from overlap correspondences (in PP22))– transform new image to reference planetransform new image to reference plane– re-sample, ‘blend’ re-sample, ‘blend’

(weighted sum) (weighted sum) to one imageto one image

xx33

xx22

xx11

xx33

xx22

xx11

HH11 HH22

HH3…3…

Panorama MakingPanorama Making

Planar:Planar:

• Start with overlapped planar proj. imagesStart with overlapped planar proj. images

• Do 4-point corresp. (or better) for alignmentDo 4-point corresp. (or better) for alignment

• Merge images by cross-dissolve:Merge images by cross-dissolve:(R,G,B)(R,G,B)outout = (R,G,B) = (R,G,B)11·· + (R,G,B) + (R,G,B)22·(1-·(1-))

where:where:

= opacity of image 1= opacity of image 1use barycentric coordsuse barycentric coords

linear interp within overlap:linear interp within overlap:

11 22

Why use PWhy use P33 at all? at all?

Make Projective Images from Non-planar objectsMake Projective Images from Non-planar objectsSeparate camera & imaged object coordinatesSeparate camera & imaged object coordinatesSeparate projective x4 from the R3 ‘z’ coord;Separate projective x4 from the R3 ‘z’ coord;

P2 P2 seemsseems to bridge 2D & 3D, but doesn’t: to bridge 2D & 3D, but doesn’t:– -z, +z axis in R3 ‘folds back on itself’ -z, +z axis in R3 ‘folds back on itself’ – -x-x33, +x, +x33 in P2 MUST be symmetric! in P2 MUST be symmetric!

“P2 is a symmetric half-space” (Zisserman) “P2 is a symmetric half-space” (Zisserman)– The only +/-z is “camera’s direction of gaze”The only +/-z is “camera’s direction of gaze”

PP33 includes full, complete 3D space; includes full, complete 3D space; – -z, +z can be asymmetric-z, +z can be asymmetric– no ‘camera direction’no ‘camera direction’

BUT ALSO: Non-Planar Panoramas . . .BUT ALSO: Non-Planar Panoramas . . .

Can’tCan’t use planar method beyond 180 use planar method beyond 180°° FOV; FOV;

• Make spherical image: ‘wrap around’ originMake spherical image: ‘wrap around’ origin

• How? Spherical coords How? Spherical coords – write 3D sphere eqn in Pwrite 3D sphere eqn in P22 coords: coords:

xx11 = cos( = cos()cos()cos() )

xx33 = -sin( = -sin()cos()cos())

xx22 = cos( = cos()) =longitude, =longitude, = latitude= latitude

– ‘‘Inverse Map’ warp:Inverse Map’ warp:• Output pixel m,n: Output pixel m,n: = 2 = 2n/nmax,n/nmax, = = n/nmax - n/nmax - /2/2

• Find x’ = HFind x’ = H-1-1·[x·[x11, x, x22, x, x33] for each image] for each image• Blend color(s) found at x’ for each imageBlend color(s) found at x’ for each image

Non-Planar PNon-Planar P22 Panoramas Panoramas

-x-x33

xx22

xx11

Example: QuickTime VR Example: QuickTime VR (Chen, Williams ’93)(Chen, Williams ’93)

1) Four Planar Images 1) Four Planar Images 1 Cylindrical Panorama: 1 Cylindrical Panorama:

IN:IN:

OUT:OUT:

Example: QuickTime VR Example: QuickTime VR (Chen, Williams ’93)(Chen, Williams ’93)

2) Windowing, Horizontal-only Reprojection:2) Windowing, Horizontal-only Reprojection:

IN:IN:

OUT:OUT:

Spherical Panoramas: Spherical Panoramas:

• Note wildly non-uniform sampling in Note wildly non-uniform sampling in ,,::

Richard Szeliski et al, Microsoft Research, 1998

Practical PPractical P22 Panoramas: ‘Box Cross’ Panoramas: ‘Box Cross’

• Spherical maps over-sample near poles;Spherical maps over-sample near poles;

• Cylindrical maps can’t see floor, ceiling spotCylindrical maps can’t see floor, ceiling spot

• Nice solution: ‘Box Cross’Nice solution: ‘Box Cross’– ‘‘unwrap’ a cube around originunwrap’ a cube around origin– 6 square planar images6 square planar images– Easy! Easy!

• for each image,for each image,• for each box side,for each box side,• find reprojection find reprojection HH• find pixels on boxfind pixels on box• rewarp as needed (cyl,sphere, etc.)rewarp as needed (cyl,sphere, etc.)

frontfront rightrightleftleft

toptop

bottombottom

backback

xx33

xx22

xx11

Example: Example: Spherical Spherical PanoramaPanorama

aa

Paul Debevec, from Paul Debevec, from SIGGRAPH2001 short SIGGRAPH2001 short

course ‘Image-Based course ‘Image-Based Lighting”Lighting”

CAREFUL! CAREFUL! EASY TO MISS EASY TO MISS SOMETHING!SOMETHING!

Commonplace Camera DistortionsCommonplace Camera Distortions

Correct many camera/lens errors in PCorrect many camera/lens errors in P22

• Place raw camera input at (x,y,1) planePlace raw camera input at (x,y,1) plane• Write equations for 3D ‘projection surface’Write equations for 3D ‘projection surface’

Example:Example: scaleable, offset sphere for spherical distortion scaleable, offset sphere for spherical distortionxx11 = a+ b*sin( = a+ b*sin()cos()cos(); x); x33 = c+ d*sin( = c+ d*sin()cos()cos(); x); x22 = e+f*cos( = e+f*cos())

• Scan projection surface to find output pixelsScan projection surface to find output pixelsExample:Example: take equal-sized steps in ( take equal-sized steps in (,,); sample input image); sample input image

xx33

xx22

xx11

Panoramic CamerasPanoramic Cameras

Panoramics without ‘stitching’:Panoramics without ‘stitching’:

• ‘‘Fisheye’ Lenses, conics,…Fisheye’ Lenses, conics,…

• Slit-scan: Slit-scan: (WideLux, Noblex, PanoScan…)(WideLux, Noblex, PanoScan…) – cyl. or spherical imagecyl. or spherical image– slow! no action shotsslow! no action shots

• Multiple Planar Cams Multiple Planar Cams – Fast, flexible, expensiveFast, flexible, expensive– can do panor. moviescan do panor. movies

• History: 1843…History: 1843…http://www.panoramicphoto.com/timeline.htmhttp://www.panoramicphoto.com/timeline.htm

Light ProbesLight Probes

• Example images (see Debevec’s site)Example images (see Debevec’s site)

Paul Debevec, SIGGRAPH2001 course “Image Based Lighting”

Try it yourself…Try it yourself…

..Paul Debevec,

SIGGRAPH2001 short course

“Image Based Lighting”

Try it yourself…Try it yourself…

..Paul Debevec,

SIGGRAPH2001 short course

“Image Based Lighting”

Better! We ordered 4 Better! We ordered 4 Chrome Spheres Chrome Spheres

for IBMR students to for IBMR students to borrow...borrow...

Mirror BallMirror BallPanorama ConversionPanorama Conversion

Makes an offset ‘virtual’ sphere camera Makes an offset ‘virtual’ sphere camera located at mirror ball center:located at mirror ball center:

• ? How can we write this in P? How can we write this in P22 and/or P and/or P33??

yymm

zzmm

xxmm

zzcc

yycc

xxcc

Camera Camera Mirror BallMirror Ball

Mirror BallMirror BallPanorama ConversionPanorama Conversion

Not Documented?!? Then figure it out…Not Documented?!? Then figure it out…– Parameterize ball as ‘rings’ of constant Parameterize ball as ‘rings’ of constant – (Ignore invisible backside of sphere)(Ignore invisible backside of sphere)– Warp 1: get ring colors from camera image Warp 1: get ring colors from camera image

yymm

zzmm

xxmm

zzcc

yycc

xxcc

CameraCamera Mirror BallMirror Ball

Mirror BallMirror BallPanorama ConversionPanorama Conversion

– Parameterize ball using ‘rings’ of constant Parameterize ball using ‘rings’ of constant – Warp 1: read color from rings’ camera imageWarp 1: read color from rings’ camera image– Find reflected ray angle Find reflected ray angle ’’ (constant on a ring)(constant on a ring)– (presume the ball diameter ~zero; then (presume the ball diameter ~zero; then ’ = 2’ = 2))

– Use ring at angle Use ring at angle ’’ on spherical projection on spherical projection

yymm

zzmm

zzcc

yycc xxcc

CameraCamera Mirror BallMirror Ball

’’

Light ProbesLight Probes

• ‘‘One-shot’ panoramic camera!One-shot’ panoramic camera!• Clever, fast, simple, cheap, flexibleClever, fast, simple, cheap, flexible• Probe position != Camera position; telephotos…Probe position != Camera position; telephotos…

– allows small probes in tight, risky spacesallows small probes in tight, risky spaces– Little/no image alignment / mosaicingLittle/no image alignment / mosaicing

• Drawbacks:Drawbacks:– Highly non-uniform samplingHighly non-uniform sampling– Camera ALWAYS in the imageCamera ALWAYS in the image

• Daydreams: a better probe? Daydreams: a better probe? – Huge: mirrored weather balloon?Huge: mirrored weather balloon?– Tiny, stochastic: bubbles in a liquid?Tiny, stochastic: bubbles in a liquid?– Dynamic shapes: whirling mirror on arm? Dynamic shapes: whirling mirror on arm? – Other shapes: Nayyar, Carlbom, Shum(MSRChina) etc.Other shapes: Nayyar, Carlbom, Shum(MSRChina) etc.

ENDEND