lecture 6: image warping and projection cs6670: computer vision noah snavely
Post on 21-Dec-2015
218 views
TRANSCRIPT
![Page 1: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/1.jpg)
Lecture 6: Image Warping and Projection
CS6670: Computer VisionNoah Snavely
![Page 2: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/2.jpg)
Readings
• Szeliski Chapter 3.5 (image warping), 9.1 (motion models)
![Page 3: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/3.jpg)
Announcements
• Project 1 assigned, due next Friday 2/18
![Page 4: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/4.jpg)
Point of observation
Figures © Stephen E. Palmer, 2002
Dimensionality Reduction Machine (3D to 2D)
3D world 2D image
What have we lost?• Angles• Distances (lengths)
Slide by A. Efros
![Page 5: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/5.jpg)
Projection properties
• Many-to-one: any points along same ray map to same point in image
• Points → points• Lines → lines (collinearity is preserved)
– But line through focal point projects to a point• Planes → planes (or half-planes)
– But plane through focal point projects to line
![Page 6: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/6.jpg)
Projection properties
• Parallel lines converge at a vanishing point– Each direction in space has its own vanishing point– But parallels parallel to the image plane remain
parallel
![Page 7: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/7.jpg)
2D to 2D warps
• Let’s start with simpler warps that map images to other images
• Examples of 2D 2D warps:
translation rotation aspect
![Page 8: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/8.jpg)
Parametric (global) warping
• Transformation T is a coordinate-changing machine:p’ = T(p)
• What does it mean that T is global?– Is the same for any point p– can be described by just a few numbers (parameters)
• Let’s consider linear xforms (can be represented by a 2D matrix):
T
p = (x,y) p’ = (x’,y’)
![Page 9: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/9.jpg)
Common linear transformations
• Uniform scaling by s:
(0,0) (0,0)
What is the inverse?
![Page 10: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/10.jpg)
Common linear transformations
• Rotation by angle θ (about the origin)
(0,0) (0,0)
What is the inverse?For rotations:
θ
![Page 11: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/11.jpg)
2x2 Matrices
• What types of transformations can be represented with a 2x2 matrix?2D mirror about Y axis?
2D mirror across line y = x?
![Page 12: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/12.jpg)
• What types of transformations can be represented with a 2x2 matrix?2D Translation?
Translation is not a linear operation on 2D coordinates
NO!
2x2 Matrices
![Page 13: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/13.jpg)
Homogeneous coordinates
Trick: add one more coordinate:
homogeneous image coordinates
Converting from homogeneous coordinates
x
y
w
(x, y, w)
w = 1 (x/w, y/w, 1)
homogeneous plane
![Page 14: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/14.jpg)
Translation
• Solution: homogeneous coordinates to the rescue
![Page 15: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/15.jpg)
Affine transformations
any transformation with last row [ 0 0 1 ] we call an affine transformation
![Page 16: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/16.jpg)
Basic affine transformations
1100
0cossin
0sincos
1
'
'
y
x
y
x
1100
10
01
1
'
'
y
x
t
t
y
x
y
x
1100
01
01
1
'
'
y
x
sh
sh
y
x
y
x
Translate
2D in-plane rotation Shear
1100
00
00
1
'
'
y
x
s
s
y
x
y
x
Scale
![Page 17: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/17.jpg)
Modeling projection
• Projection equations– Compute intersection with PP of ray from (x,y,z) to COP– Derived using similar triangles (on board)
• We get the projection by throwing out the last coordinate:
![Page 18: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/18.jpg)
Modeling projection• Is this a linear transformation?
Homogeneous coordinates to the rescue!
homogeneous image coordinates
homogeneous scene coordinates
Converting from homogeneous coordinates
• no—division by z is nonlinear
![Page 19: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/19.jpg)
Perspective Projection
Projection is a matrix multiply using homogeneous coordinates:
divide by third coordinate
This is known as perspective projection• The matrix is the projection matrix
• (Can also represent as a 4x4 matrix – OpenGL does something like this)
![Page 20: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/20.jpg)
Perspective Projection
• How does scaling the projection matrix change the transformation?
![Page 21: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/21.jpg)
Orthographic projection• Special case of perspective projection
– Distance from the COP to the PP is infinite
– Good approximation for telephoto optics– Also called “parallel projection”: (x, y, z) → (x, y)– What’s the projection matrix?
Image World
![Page 22: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/22.jpg)
Orthographic projection
![Page 23: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/23.jpg)
Perspective projection
![Page 24: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/24.jpg)
Perspective distortion
• What does a sphere project to?
Image source: F. Durand
![Page 25: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/25.jpg)
Perspective distortion
• The exterior columns appear bigger• The distortion is not due to lens flaws• Problem pointed out by Da Vinci
Slide by F. Durand
![Page 26: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/26.jpg)
Perspective distortion: People
![Page 27: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/27.jpg)
Distortion
• Radial distortion of the image– Caused by imperfect lenses– Deviations are most noticeable for rays that
pass through the edge of the lens
No distortion Pin cushion Barrel
![Page 28: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/28.jpg)
Correcting radial distortion
from Helmut Dersch
![Page 29: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/29.jpg)
Distortion
![Page 30: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/30.jpg)
Modeling distortion
• To model lens distortion– Use above projection operation instead of standard
projection matrix multiplication
Apply radial distortion
Apply focal length translate image center
Project to “normalized”
image coordinates
![Page 31: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/31.jpg)
Other types of projection
• Lots of intriguing variants…• (I’ll just mention a few fun ones)
![Page 32: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/32.jpg)
360 degree field of view…
• Basic approach– Take a photo of a parabolic mirror with an orthographic lens (Nayar)– Or buy one a lens from a variety of omnicam manufacturers…
• See http://www.cis.upenn.edu/~kostas/omni.html
![Page 33: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/33.jpg)
Tilt-shift
Titlt-shift images from Olivo Barbieriand Photoshop imitations
http://www.northlight-images.co.uk/article_pages/tilt_and_shift_ts-e.html
![Page 34: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/34.jpg)
Camera parameters
• How can we model the geometry of a camera?
“The World”
Camera
x
y
z
v
w
u
o
COP
Two important coordinate systems: 1. World coordinate system 2. Camera coordinate system
![Page 35: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/35.jpg)
Camera parameters
• To project a point (x,y,z) in world coordinates into a camera
• First transform (x,y,z) into camera coordinates• Need to know
– Camera position (in world coordinates)– Camera orientation (in world coordinates)
• The project into the image plane– Need to know camera intrinsics
![Page 36: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/36.jpg)
Projection equation
• The projection matrix models the cumulative effect of all parameters• Useful to decompose into a series of operations
ΠXx
1****
****
****
Z
Y
X
s
sy
sx
110100
0010
0001
100
'0
'0
31
1333
31
1333
x
xx
x
xxcy
cx
yfs
xfs
00
0 TIRΠ
projectionintrinsics rotation translation
identity matrix
Camera parametersA camera is described by several parameters
• Translation T of the optical center from the origin of world coords• Rotation R of the image plane• focal length f, principle point (x’c, y’c), pixel size (sx, sy)• blue parameters are called “extrinsics,” red are “intrinsics”
• The definitions of these parameters are not completely standardized– especially intrinsics—varies from one book to another
![Page 37: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/37.jpg)
Extrinsics• How do we get the camera to “canonical form”?
– (Center of projection at the origin, x-axis points right, y-axis points up, z-axis points backwards)
0
Step 1: Translate by -c
![Page 38: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/38.jpg)
Extrinsics• How do we get the camera to “canonical form”?
– (Center of projection at the origin, x-axis points right, y-axis points up, z-axis points backwards)
0
Step 1: Translate by -c
How do we represent translation as a matrix multiplication?
![Page 39: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/39.jpg)
Extrinsics• How do we get the camera to “canonical form”?
– (Center of projection at the origin, x-axis points right, y-axis points up, z-axis points backwards)
0
Step 1: Translate by -cStep 2: Rotate by R
3x3 rotation matrix
![Page 40: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/40.jpg)
Extrinsics• How do we get the camera to “canonical form”?
– (Center of projection at the origin, x-axis points right, y-axis points up, z-axis points backwards)
0
Step 1: Translate by -cStep 2: Rotate by R
![Page 41: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/41.jpg)
Perspective projection
(intrinsics)
in general,
: aspect ratio (1 unless pixels are not square)
: skew (0 unless pixels are shaped like rhombi/parallelograms)
: principal point ((0,0) unless optical axis doesn’t intersect projection plane at origin)
(upper triangular matrix)
(converts from 3D rays in camera coordinate system to pixel coordinates)
![Page 42: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/42.jpg)
Focal length• Can think of as “zoom”
• Also related to field of view
24mm 50mm
200mm 800mm
![Page 43: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/43.jpg)
Projection matrix
(t in book’s notation)
translationrotationprojection
intrinsics
![Page 44: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/44.jpg)
Projection matrix
0
=
(in homogeneous image coordinates)
![Page 45: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/45.jpg)
Questions?
• 3-minute break
![Page 46: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/46.jpg)
Image alignment
Full screen panoramas (cubic): http://www.panoramas.dk/ Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html
2003 New Years Eve: http://www.panoramas.dk/fullscreen3/f1.html
![Page 47: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/47.jpg)
Why Mosaic?
• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°
Slide from Brown & Lowe
![Page 48: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/48.jpg)
Why Mosaic?
• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°
Slide from Brown & Lowe
![Page 49: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/49.jpg)
Why Mosaic?
• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°– Panoramic Mosaic = 360 x 180°
Slide from Brown & Lowe
![Page 50: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/50.jpg)
Mosaics: stitching images together
![Page 51: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/51.jpg)
Readings
• Szeliski:– Chapter 3.5: Image warping– Chapter 5.1: Feature-based alignment– Chapter 8.1: Motion models
![Page 52: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/52.jpg)
Image alignment
Image taken from same viewpoint, just rotated.
Can we line them up?
![Page 53: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/53.jpg)
Image alignment
Why don’t these image line up exactly?
![Page 54: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/54.jpg)
What is the geometric relationship between these two images?
?
![Page 55: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/55.jpg)
What is the geometric relationship between these two images?
?
![Page 56: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/56.jpg)
Is this an affine transformation?
![Page 57: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/57.jpg)
Where do we go from here?
affine transformation
what happens when we mess with this row?
![Page 58: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/58.jpg)
Projective Transformations aka Homographies aka Planar Perspective Maps
Called a homography (or planar perspective map)
![Page 59: Lecture 6: Image Warping and Projection CS6670: Computer Vision Noah Snavely](https://reader030.vdocument.in/reader030/viewer/2022032521/56649d5f5503460f94a3ef8a/html5/thumbnails/59.jpg)
Homographies
• Example on board