Last Lecture

(optical center)

origin

principal point

P

(X,Y,Z)

p

(x,y)

Z

Y

X

yf

xf

Z

fY

fX

y

x

100

0

0

~

10

0

x

y

Camera calibration

Camera calibration

• Estimate both intrinsic and extrinsic parameters

• Mainly, two categories:1. Using objects with known geometry as

reference 2. Self calibration (structure from motion)

11100

0

0

~

1Z

Y

X

lkji

hgfe

dcba

Z

Y

X

yf

xf

y

x

c

c

tR

One app of camera pose application

• Virtual gaming– http://www.livestream.com/emtech/video?

clipId=pla_74103098-95fb-4704-99f0-d07339dc16a1

Camera calibration approaches

• Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)

MXXtRKx ~

Linear regression

Linear regression

i

i

iiiiiiiii

iiiiiiiii

v

u

m

m

m

m

m

m

m

m

m

m

m

YvYvXvZYX

ZuYuXuZYX

22

21

20

13

12

11

10

03

02

01

00

10000

00001

Linear regression

Solve for Projection Matrix M using least-square techniques

i

i

NNNNNNNNN

NNNNNNNNNv

u

m

m

m

m

m

m

m

m

m

m

m

YvYvXvZYX

ZuYuXuZYX

YvYvXvZYX

ZuYuXuZYX

22

21

20

13

12

11

10

03

02

01

00

111111111

111111111

10000

00001

10000

00001

Normal equation (Geometric Interpretation)

Given an overdetermined system

bAx

0)(T bAxA

the normal equation is that which minimizes the sum of the square differences between left and right sides

2min bAx

bx T1T AAA

Normal equation (Differential Interpretation)

2)( bAxx E

n

m

nmn

m

b

b

x

x

aa

aa

:

:

:

:

...

::

::

::

... 1

1

1

111

nxm, n equations, m variables

Normal equation

bbxbAxbAAxAx

bbbAxAxbAxAx

bAxbAx

bAxbAx

bAxx

TTTTTTT

TTTTTT

TTT

T

E2

)(

bAAxAx

TT 22 E

Carl Friedrich Gauss

Any issues with the method?

Nonlinear optimization

• A probabilistic view of least square• Feature measurement equations

• Likelihood of M given {(ui,vi)}

Optimal estimation

• Log likelihood of M given {(ui,vi)}

• It is a least square problem (but not necessarily linear least square)

• How do we minimize C?

2

222120

131211102

2

1 222120

030201002 1

1

1

1

iii

iiii

i

N

i iii

iiii

i ZmYmXm

mZmYmXmv

ZmYmXm

mZmYmXmu

Nonlinear least square methods

Least square fitting

number of data points

number of parameters

Nonlinear least square fitting

Function minimization

It is very hard to solve in general. Here, we only consider a simpler problem of finding local minimum.

Least square is related to function minimization.

Function minimization

Quadratic functions

Approximate the function with a quadratic function within a small neighborhood

Function minimization

Computing gradient and Hessian

hxx

hhx

xx

hxxx

xhhx

xx

hxxx

xhhx

xx

hx

hhx

x

hxhx

m

i

iim

i

ii

m

ii

m

i

iiii

m

i

ii

m

ii

m

i

iiii

iii

m

i

iii

m

ii

fffff

ffff

fff

ffff

fff

fff

fF

1

TT

T

11

2

1

T

2

2T

T

11

2

1

T

2

2T

T2

1

2

2

2T

T

1

2

2

1)()(

2

1

)(2

1)()(

2

1

)()(2)(2

1

2

1)(

2

1

)(2

1)(

Gradient Hessian

Computing gradient and Hessian

hxx

hhx

xx

hxxx

xhhx

xx

hxxx

xhhx

xx

hx

hhx

x

hxhx

m

i

iim

i

ii

m

ii

m

i

iiii

m

i

ii

m

ii

m

i

iiii

iii

m

i

iii

m

ii

fffff

ffff

fff

ffff

fff

fff

fF

1

TT

T

11

2

1

T

2

2T

T

11

2

1

T

2

2T

T2

1

2

2

2T

T

1

2

2

1)()(

2

1

)(2

1)()(

2

1

)()(2)(2

1

2

1)(

2

1

)(2

1)(

Gradient Hessian

Computing gradient and Hessian

hxx

hhx

xx

hxxx

xhhx

xx

hxxx

xhhx

xx

hx

hhx

x

hxhx

m

i

iim

i

ii

m

ii

m

i

iiii

m

i

ii

m

ii

m

i

iiii

iii

m

i

iii

m

ii

fffff

ffff

fff

ffff

fff

fff

fF

1

TT

T

11

2

1

T

2

2T

T

11

2

1

T

2

2T

T2

1

2

2

2T

T

1

2

2

1)()(

2

1

)(2

1)()(

2

1

)()(2)(2

1

2

1)(

2

1

)(2

1)(

Gradient Hessian

Computing gradient and Hessian

hxx

hhx

xx

hxxx

xhhx

xx

hxxx

xhhx

xx

hx

hhx

x

hxhx

m

i

iim

i

ii

m

ii

m

i

iiii

m

i

ii

m

ii

m

i

iiii

iii

m

i

iii

m

ii

fffff

ffff

fff

ffff

fff

fff

fF

1

TT

T

11

2

1

T

2

2T

T

11

2

1

T

2

2T

T2

1

2

2

2T

T

1

2

2

1)()(

2

1

)(2

1)()(

2

1

)()(2)(2

1

2

1)(

2

1

)(2

1)(

Gradient Hessian

Computing gradient and Hessian

hxx

hhx

xx

hxxx

xhhx

xx

hxxx

xhhx

xx

hx

hhx

x

hxhx

m

i

iim

i

ii

m

ii

m

i

iiii

m

i

ii

m

ii

m

i

iiii

iii

m

i

iii

m

ii

fffff

ffff

fff

ffff

fff

fff

fF

1

TT

T

11

2

1

T

2

2T

T

11

2

1

T

2

2T

T2

1

2

2

2T

T

1

2

2

1)()(

2

1

)(2

1)()(

2

1

)()(2)(2

1

2

1)(

2

1

)(2

1)(

Gradient Hessian

Searching for update h

Hhhhgxhx TT

1

2

2

1)(

2

1)(

m

iifF

Gradient Hessian

Idea 1: Steepest Descent

Hgg

gg

Hggggxhx

gh

T

T

T2

T

1

2

2)(

2

1)(

Let

m

iifF

Steepest descent method

isocontour gradient

Steepest descent method

It has good performance in the initial stage of the iterative process. Converge very slow with a linear rate.

Searching for update h

Hhhhgxhx TT

1

2

2

1)(

2

1)(

m

iifF

Gradient Hessian

Idea 2: minimizing the quadric directly

gHh

Hhghxh

1

0)(Let

F

Converge faster but needs to solve the linear system

Recap: Calibration

• Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)

MXXtRKx ~

Camera Model:

Recap: Calibration

• Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)

MXXtRKx ~

Linear Approach:

Recap: Calibration

• Directly estimate 11 unknowns in the M matrix using known 3D points (Xi,Yi,Zi) and measured feature positions (ui,vi)

MXXtRKx ~

NonLinear Approach:

2

222120

13121110

2

1 222120

03020100

11

iii

iiii

N

i iii

iiii ZmYmXm

mZmYmXmv

ZmYmXm

mZmYmXmu

Practical Issue

is hard to make and the 3D feature positions are difficult to measure!

A popular calibration tool

Multi-plane calibration

Images courtesy Jean-Yves Bouguet, Intel Corp.

Advantage• Only requires a plane• Don’t have to know positions/orientations• Good code available online!

– Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/

– Matlab version by Jean-Yves Bouget: http://www.vision.caltech.edu/bouguetj/calib_doc/index.html

– Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/

Step 1: data acquisition

Step 2: specify corner order

Step 3: corner extraction

Step 3: corner extraction

Step 4: minimize projection error

Step 4: camera calibration

Step 4: camera calibration

Step 5: refinement

Next

Image Mosaics and Panorama

• Today’s Readings– Szeliski and Shum paper

http://www.acm.org/pubs/citations/proceedings/graph/258734/p251-szeliski/

Full screen panoramas (cubic): http://www.panoramas.dk/ Mars: http://www.panoramas.dk/fullscreen3/f2_mars97.html

2003 New Years Eve: http://www.panoramas.dk/fullscreen3/f1.html

Why Mosaic?

• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°

Slide from Brown & Lowe

Why Mosaic?

• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°

Slide from Brown & Lowe

Why Mosaic?

• Are you getting the whole picture?– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°– Panoramic Mosaic = 360 x 180°

Slide from Brown & Lowe

Mosaics: stitching images together

Creating virtual wide-angle camera

Auto Stitch: the State of Art Method

• Demo

• Project 2 is a striped-down AutoStitch

How to do it?

• Basic Procedure– Take a sequence of images from the same

position• Rotate the camera about its optical center

– Compute transformation between second image and first

– Transform the second image to overlap with the first

– Blend the two together to create a mosaic– If there are more images, repeat

Geometric Interpretation of Mosaics

Image 1

Image 2

• If we capture all the 360º rays in different images, we can assemble them into a panorama.

• The basic operation is projecting an image from one plane to another

• The projective transformation is scene-INDEPENDENT

Optical Center

What is the transformation?

Translations are not enough to align the images

left on top right on top