camera calibration sebastian thrun, gary bradski, daniel russakoff stanford cs223b computer vision ...

Camera Calibration

Sebastian Thrun, Gary Bradski, Daniel RussakoffStanford CS223B Computer Vision

http://robots.stanford.edu/cs223b

(with material from David Forsyth, James Rehg and Allen Hanson)

Sebastian Thrun Stanford University CS223B Computer Vision

A Quiz

How Many Flat Calibration Targets are Needed for Calibration? 1: 2: 3: 4: 5: 10:

How Many Corner Points do we need in Total?

1: 2: 3: 4: 10: 20:

Sebastian Thrun Stanford University CS223B Computer Vision

Example Calibration Pattern

Sebastian Thrun Stanford University CS223B Computer Vision

Perspective Camera Model

WX

WY

WZ

CX

CY

Object Space

CZ

Sebastian Thrun Stanford University CS223B Computer Vision

Calibration Model (extrinsic)

?),(

W

W

W

C

C

Z

Y

X

fY

X

Sebastian Thrun Stanford University CS223B Computer Vision

Experiment 1: Parallel Board

Sebastian Thrun Stanford University CS223B Computer Vision

30cm10cm 20cm

Projective Perspective of Parallel Board

Sebastian Thrun Stanford University CS223B Computer Vision

Experiment 2: Tilted Board

Sebastian Thrun Stanford University CS223B Computer Vision

30cm10cm 20cm

500cm50cm 100cm

Projective Perspective of Tilted Board

Sebastian Thrun Stanford University CS223B Computer Vision

Calibration Model (extrinsic)

),,,,( T

Z

Y

X

fY

X

W

W

W

C

C

rotation

translation (3D)

Sebastian Thrun Stanford University CS223B Computer Vision

Calibration Model (extrinsic)

Z

Y

X

W

W

W

C

C

C

T

T

T

Z

Y

X

Z

Y

X

cossin0

sincos0

001

cos0sin

01

sin0cos

100

0cossin

0sincos

~

~

~

C

C

CC

C

Y

X

Z

f

Y

X~

~

~

Homogeneous Coordinates

Sebastian Thrun Stanford University CS223B Computer Vision

Homogeneous Coordinates

Idea: Most Operations Become Linear!

Extract Image Coordinates by Z-normalization

C

C

CC

C

Y

X

ZY

X~

~

~1

C

C

C

Z

Y

X

~

~

~

1

12

10

000,1

200,1

000,10

5.1

18

15

1

12

10

Sebastian Thrun Stanford University CS223B Computer Vision

Advantage of Homogeneous C’s

Z

Y

X

W

W

W

C

C

T

T

T

iZ

iY

iX

iY

iXi

][

][

][

cossin0

sincos0

001

cos0sin

010

sin0cos

0

0

cos

sin

sin

cos

][~

][~

][

i-th data point

parametersrotation in nonlinear but

][][Equation Quadratici ibXiAX T

Sebastian Thrun Stanford University CS223B Computer Vision

Calibration Model (intrinsic)

xs

ysPixel size

C

C

CC

C

Y

X

Z

f

Y

X~

~

~

Focal length

Image center

yx oo ,

Sebastian Thrun Stanford University CS223B Computer Vision

5.~

~

5.~

~

yC

C

y

xC

C

x

im

im

oZ

Y

s

f

oZ

X

s

f

y

x

Intrinsic Transformation

Sebastian Thrun Stanford University CS223B Computer Vision

5.

][

][

][

cossin0

sincos0

001

cos0sin

010

sin0cos

100

][

][

][

cossin0

sincos0

001

cos0sin

010

sin0cos

0cossin

5.

][

][

][

cossin0

sincos0

001

cos0sin

010

sin0cos

100

][

][

][

cossin0

sincos0

001

cos0sin

010

sin0cos

0sincos

y

Z

Y

X

W

W

W

Z

Y

X

W

W

W

y

x

Z

Y

X

W

W

W

Z

Y

X

W

W

W

x

im

im

o

T

T

T

iZ

iY

iX

T

T

T

iZ

iY

iX

s

f

o

T

T

T

iZ

iY

iX

T

T

T

iZ

iY

iX

s

f

y

x

Plugging the Model Together!

Sebastian Thrun Stanford University CS223B Computer Vision

Summary Parameters

Extrinsic

– Rotation

– Translation

Intrinsic

– Focal length

– Pixel size

– Image center coordinates

– (Distortion coefficients - see JYB’s tutorial )

,.

T

f

),( yx oo

,...1k

),( yx ss

Sebastian Thrun Stanford University CS223B Computer Vision

Q: Can We recover all Intrinsic Params?

No

Sebastian Thrun Stanford University CS223B Computer Vision

Summary Parameters, Revisited

Extrinsic

– Rotation

– Translation

Intrinsic

– Focal length

– Pixel size

– Image center coordinates

– (Distortion coefficients - see JYB’s tutorial )

,,

T

f

),( yx oo

,...1k

),( yx ssFocal length, in pixel units

Aspect ratioy

x

s

s

xs

f

Sebastian Thrun Stanford University CS223B Computer Vision

Calibration a la Trucco

Substitute

Advantage: Equations are linear in params

If over-constrained, minimize Least Mean Square fct

One possible solution:

Enforce constraint that R is rotation matrix

Lots of considerations to recover individual params…

cossin0

sincos0

001

cos0sin

01

sin0cos

100

0cossin

0sincos

333231

232221

131211

rrr

rrr

rrr

bAX

bAAAX TT 1

min)()( bAXbAX T

},,,,{ yxx

oos

fTRX

Sebastian Thrun Stanford University CS223B Computer Vision

Calibration a la Bouguet

5.

][

][

][

cossin0

sincos0

001

cos0sin

010

sin0cos

100

][

][

][

cossin0

sincos0

001

cos0sin

010

sin0cos

0cossin

5.

][

][

][

cossin0

sincos0

001

cos0sin

010

sin0cos

100

][

][

][

cossin0

sincos0

001

cos0sin

010

sin0cos

0sincos

y

Z

Y

X

W

W

W

Z

Y

X

W

W

W

y

x

Z

Y

X

W

W

W

Z

Y

X

W

W

W

x

im

im

o

T

T

T

iZ

iY

iX

T

T

T

iZ

iY

iX

s

f

o

T

T

T

iZ

iY

iX

T

T

T

iZ

iY

iX

s

f

y

x

),,,,,,,( yxxW

W

W

im

im oos

fT

Z

Y

X

gy

x

Sebastian Thrun Stanford University CS223B Computer Vision

Calibration a la Bouguet, cont’d

Calibration Examples:

),,,,,,,

]1[

]1[

]1[

(]1[

]1[yx

xW

W

W

im

im oos

fT

Z

Y

X

gy

x

),,,,,,,

][

][

][

(][

][yx

xW

W

W

im

im oos

fT

NZ

NY

NX

gNy

Nx

…),,,,,,,

]2[

]2[

]2[

(]2[

]2[yx

xW

W

W

im

im oos

fT

Z

Y

X

gy

x

Sebastian Thrun Stanford University CS223B Computer Vision

Calibration a la Bouguet, cont’d

Least Mean Square

min),,,,,,,

][

][

][

(][

][

2

iyx

xW

W

W

im

im oos

fT

iZ

iY

iX

giy

ixJ

Sebastian Thrun Stanford University CS223B Computer Vision

Calibration a la Bouguet, cont’d

Least Mean Square

Gradient descent:J

},,,,,,{ yxx

oos

fTX

0X

0X

J

)(1.0 11

kkk XX

JXX

min),,,,,,,

][

][

][

(][

][

2

iyx

xW

W

W

im

im oos

fT

iZ

iY

iX

giy

ixJ

Sebastian Thrun Stanford University CS223B Computer Vision

Trucco Versus Bouguet

Trucco: Mimization of Squared

distance in parameter space

Bouguet Minimization of Squared

distance in Image space

)}ˆ

ˆ

ˆ

ˆ()

ˆ

ˆ

ˆ

ˆ(

2

1exp{

|2|

1)

ˆ

ˆ|

ˆ

ˆ( 1

5.

edge

edge

edge

edgeT

edge

edge

edge

edge

edge

edge

edge

edge

y

x

y

x

y

x

y

x

y

x

y

xp

)ˆ

ˆ

ˆ

ˆ()

ˆ

ˆ

ˆ

ˆ(

2

1)

ˆ

ˆ|

ˆ

ˆ(log 1

edge

edge

edge

edgeT

edge

edge

edge

edge

edge

edge

edge

edge

y

x

y

x

y

x

y

xconst

y

x

y

xp

Sebastian Thrun Stanford University CS223B Computer Vision

Q: How Many Images Do We Need?

Assumption: K images with M corners each 4+6K parameters 2KM constraints 2KM 4+6K M>3 and K 2/(M-3) 2 images with 4 points, but will 1 images with 5

points work? No, since points cannot be co-planar!

Sebastian Thrun Stanford University CS223B Computer Vision

Nonlinear Distortions

Barrel and Pincushion Tangential

Sebastian Thrun Stanford University CS223B Computer Vision

Barrel and Pincushion Distortion

telewideangle

Sebastian Thrun Stanford University CS223B Computer Vision

Models of Radial Distortion

)1( 42

21 rkrk

y

x

y

x

d

d

distance from center

Sebastian Thrun Stanford University CS223B Computer Vision

Tangential Distortion

cheapglue

cheap CMOS chipcheap lense image

cheap camera

Sebastian Thrun Stanford University CS223B Computer Vision

Image Rectification