lesson 9: 3d vision - unizar.eswebdiis.unizar.es/~neira/12082/3dvision.pdf · 12082 - computer...

112082 - Computer Vision - J.D. Tardós

Lesson 9: 3D VisionLesson 9: 3D Vision1. Introduction

2. Camera Model

3. Calibration

4. Stereo Vision

5. Correspondence Search

6. Other 3D techniques


1. Introduction1. Introduction• Cameras get a 2D projection of a 3D scene• 3D reconstruction needs two or more images

– 2 cameras: binocular stereo– 3 cameras: trinocular stereo– 1 single moving camera (SFM: structure from motion)

• Needs a model of the camera geometry and optics• Main difficulty: search for correspondences

– Given en element in one image (a point, edge, region,…), find it on the other images


2. Camera Model (Tsai)2. Camera Model (Tsai)• Projection of a point on an image:

• Intrinsic camera parameters– Camera geometry and optics– Image acquisition system (for analog cameras)

• Extrinsic camera parameters– Camera 3D position and orientation

• Calibration: experimental determination of camera parameters

⎟⎟⎠

⎞⎜⎜⎝

⎛=

⎟⎟⎠

⎞⎜⎜⎝

⎛=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

=

ext

intc

w

w

w

w

cw

vu

zyx

θθ

θ

θ

u

x

xu ),( Proj

3D point coordinates

Coordinates on the image

Camera parameters


Tsai’s Camera Model (1)Tsai’s Camera Model (1)1) Transformation to camera reference

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

333231

232221

131211

rrrrrrrrr

zyx

cw

cw

cw

cw

cw

R

t

Optical Axis Optical

Center

x y

zC

x

yz W

Absolute Reference T cw

xy

f

Image Plane

uv

P (x w , y w, zw)

(x s, y s)

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

+=

c

c

c

c

cwwcwc

zyx

x

txRx

Coordinates relative to the Camera:

Position and Orientationof W relative to C:


Homogeneous TransformationsHomogeneous Transformations

• Orientation defined by 3 angles:

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

=

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎠

⎞⎜⎜⎝

⎛=

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

1110

1w

w

w

cww

w

w

cwcw

c

c

c

zyx

zyx

zyx

TtR

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

−−++−

=

==

ψθψθθψφψθφψφψθφθφψφψθφψφψθφθφ

ψθφ

ψθφψθφ

CCSCSSCCSSCCSSSCSSSCSCCSSSCCC

RPY

xRotyRotzRotRPY

),,(

),(),(),(),,(

⎟⎟⎠

⎞⎜⎜⎝

⎛ −== −

101 cw

Tcw

Tcw

cwwctRRTT

Position and Orientation of W

relative to C

Position and Orientation of C

relative to W

Rotation Order


Homogeneous TransformationsHomogeneous Transformations• Example:

Projection of W´saxis on reference C

x zy

W

AbsoluteReference

10

5

C

x

z

y

z

x

y

OpticalCenter

x

Cz

y

Rot(y,-90)

Cz

x

y

Rot(x,90)

z

C

y

x

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛−

−=−=

001100

010)90,()90,()0,( xRotyRotzRotcwR


Homogeneous TransformationsHomogeneous Transformations• Example:

W

AbsoluteReference

10

5

C

x

z

y

z

x

y

OpticalCenter

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

−−

==

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛−

−

=

−

10005010

100010100

100000015100

10010

1cwwc

cw

TT

T

Projection of C´saxis on reference W

x zy


Homogeneous TransformationsHomogeneous Transformations

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛−

−

=

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛−

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

=

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎠

⎞⎜⎜⎝

⎛=

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

167

10

100000015100

10010

110

13

1110

1w

w

w

cww

w

w

cwcw

c

c

c

zyx

zyx

zyx

TtR

• Example:

W

AbsoluteReference

10

5

C

xz

y

z

x

y

10

7

6

10 3 1

OpticalCentre


Tsai’s Camera Model (2)Tsai’s Camera Model (2)2) Theoretical projection on image sensor

c

cyst

c

cst

st

stst

zfy

zxfx

yx

=

=

⎟⎟⎠

⎞⎜⎜⎝

⎛=x

Optical Axis Optical

Center

x y

zC

x

yz W


xy

f

Image Plane

uv

P (x w , y w, zw)

(x s, y s)

Objective’s focal

length (mm)


Tsai’s Camera Model (3)Tsai’s Camera Model (3)3) Radial distortion

– Noticeable for wide lenses: f < 12 mm

222

21

21

)1(

)1(

sdsd

sdst

sdst

sd

sdsd

yxr

rkyy

rkxx

yx

+=

+=

+=

⎟⎟⎠

⎞⎜⎜⎝

⎛=x

Radial distortioncoefficient

Distortedcoordinates

Theoretical undistortedcoordinates


Tsai’s Camera Model (4)Tsai’s Camera Model (4)4) Transform to pixel coordinates

• Example for a CCD analogic camera:

y

sdy

xx

sdx

dyCv

sdxCu

vu

+=

+=

⎟⎟⎠

⎞⎜⎜⎝

⎛=u

Coordinates where the optical axis intersects the

image plane (in pixels)

Pixel size on the sensor (mm)

Horizontal correction factor(for analog cameras)

tx

sxcxx

cyycyy

NNdd

ddordd

=

== 2

Pixels per line on the sensor

Acquiring allthe lines

Acquiring half the lines

Pixels per line on memory


Tsai’s Camera Model (4)Tsai’s Camera Model (4)• Example: camera Pulnix TM-6 (analogic)

– Pixels: 752(H) x 582 (V)– Cell size: 8.4μm x 8.2 μm

a) Sampling 512(H) x 512(V)

b) Sampling 768(H) x 512(V)

5.1

102.8

1034.12512752104.8

3

33

==

×=

×=×=

−

−−

y

x

y

x

ddPAR

mmd

mmd

mmd

mmd

y

x

3

33

102.8

102.8768752104.8

−

−−

×=

×=×=square pixel

PAR = 1

PAR: Pixel Aspect Ratio


Tsai’s Camera Model (summary)Tsai’s Camera Model (summary)• Projection model:

1. Transform to camera:

2. Theoretical projection:

3. Radial distortion:

4. Transform to pixels:

• Known parameters: dx, dy

• Parameter to Calibrate:» Extrinsic: tcw, Rcw

» Intrinsic: f, k1, Cx, Cy, sx

y

sdyx

x

sdx

sdsd

sdstsdst

c

cyst

c

cst

cwwcwc

cw

dyCvs

dxCu

yxr

rkyyrkxx

zfy

zxfx

+=+=

+=

+=+=

==

+=

=

222

21

21 )1()1(

),Proy(

txRx

xu θ


Normalized RetinaNormalized Retina• Is an ideal retina (without distortion) placed at focal

distance f = 1

⎟⎟⎠

⎞⎜⎜⎝

⎛=

=

=

+==

n

nn

c

cyn

c

cn

cwwcwc

extwn

vu

zv

zxu

u

txRxxu ),(Norm_proj θ

Optical axisOptical Center

x y

zC

x

yz W


f=1

NormalizedRetina

unv n

P (x w, y w, z w )

The projection onlydepends on the extrinsic

camera parameters:

Normalized coordinates (mm)


Normalized RetinaNormalized Retina• Each pixel can be transformed from the real retina to

the normalized retina:

fv

fxu

rkyyrkxx

yxr

dCvysdCux

styn

stn

sdstsdst

sdsd

yysdx

xxsd

intn

==

+=+=

+=

−=−=

=

)1()1(

)()(

),Normal(

21

21

222

θuu


3. Calibration3. Calibration• Cameramodel:

• Put a pattern with N points, in known positions:(xw1, ... xwN)

• Take an image and compute the point coordinates in the image: (u1, ... uN)

• Obtain the set of parameters θc that minimizes the squared error:

),(Proj cw θxu =

[ ]),(Proj1

cwii

N

iθxu −∑

=

In order to compute all theparameters, the pattern must be 3D


4. Stereo Vision 4. Stereo Vision • Simplest case: parallel cameras with no distortion

• Depth depends on the disparity: x1 - x2

21

2

22

1

11

xxfbz

zbxf

zxfx

zxf

zxfx

c

c

c

c

−=

−==

==

x1 x2P1 P2

x

b

f f

ImagePlane

OpticalCenter

OpticalAxis

P

01 02

z


Stereo VisionStereo Vision• General case: real cameras in arbitrary positions

– Using the normalized retinas:

( )kikikiik

wcwk

wcwk

wcwk

c

c

c

wc

wc

wc

w

kkk

kkk

kkk

wcwwcc

rrrwhere

zyx

zyx

zyx

rrrrrrrrr

R

k

k

k

k

k

k

k

k

k

kkk

321

3

2

1

333231

232221

131211

:=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

+++

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

+⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

+=

r

xrxrxr

x

txx

( )( )

⎟⎟⎠

⎞⎜⎜⎝

⎛−−

=

⎟⎟⎠

⎞⎜⎜⎝

⎛−−

=

=+

+=+

+=+

==

wcnwc

wcnwck

knk

knkk

kwk

wcwknwcwk

wcwknwcwk

c

cn

c

cn

kkk

kkk

k

k

kkk

kkk

k

k

k

k

k

k

zvyzux

vu

zvy

zux

zy

vzx

u

b

rrrr

A

bxA

xrxr

xrxr

32

31

32

31

0


Stereo VisionStereo Vision• 3D point reconstruction from n images

– Three unknowns: xw, yw, zw

– Two equations in each image k:

– System of linear equations (over-determined):

– Least Squares Solution:

bAAAx

b

bb

A

AA

bAx

bxA

TTw

nn

w

kwk

1

11

)(

......

0

0

−=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

=+

=+


Example: parallel camerasExample: parallel cameras

• Choosing as absolute reference W = C1

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛−=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

00

100010001

000

100010001

22

11

btR

tR

wcwc

wcwc

x1 x2P1 P2

x

b

f f

ImagePlane

OpticalCenter

OpticalAxis

P

01 02

z


Example: parallel camerasExample: parallel cameras• Linear system:

• Analytical solution:

• Epipolar constraint: The corresponding points must be on the same row in both images

21

211

211

21

2

2

1

1

0

0

00

10011001

nn

nnnw

nnnw

nnw

w

w

w

n

n

n

n

vv

uubvy

uubux

uubz

bzyx

vuvu

=

−

−

−

=

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

−+⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

−−−−

=

=

=

EpipolarConstraint


5. Correspondence Search5. Correspondence Search• Give a point P, its epipolar plane is PC1C2

• Its projections P1 and P2 belong to P’s epipolar plane

• Epipolar constraint for P1:– Given C1 , C2 y P1, compute the epipolar plane– Intersect the epipolar plane with the image plane 2 to obtain P1’s

epipolar line in image 2 – The point corresponding to P1 must be on the epipolar line

• They can be several candidates P2 , P2’,...

C1 C2

EpipolarPlane

P1 P2

P1´sEpipolarLine

P’’

P’

P


Edge CorrespondencesEdge Correspondences• Detect edge points in both images• For each edge in image 1, search for an edge along

its epipolar line having:– Similar gradient magnitude– Similar gradient orientation

• Contours oriented parallel to its epipolar line cannot be matched:

P1’s epipolar line

P1

P2

P2’s epipolar line

?

:)


MatchingMatching usingusing CorrelationCorrelation• For each pixel, take a local patch and search for a

similar patch along the epipolar line on the second image


CorrelationCorrelation• Sum of Squared Differences SSD:

» Problematic if illumination changes

• Normalized Correlation:

» ρ close to 1: linear gray level relation correspondence

( )∑ −ji

bij

aij WW

,

2

( )( )

( ) ( )∑∑

∑

−−

−−=

ij

aij

aij

ij

bij

bij

ij

bij

bij

aij

aij

WWWW

WWWW

22ρ [ ]1,1−∈ρ

Each posible nxm patchon the second image

Searched nxm patch from the first image


StereoStereo usingusing CorrelationCorrelation• SRI Stereo Engine algorithm

by Kurt Konoligehttp://www.ai.sri.com/~konolige/

1. Rectification - remove distortion from images

2. Feature Extraction -Laplacian of Gaussian

3. Correlation - compute disparities by matching

4. Filtering - remove uncertain matches

5. 3D reconstruction - convert disparities to 3D points

http://www.ai.sri.com/~konolige/


StereoStereo usingusing CorrelationCorrelation• SRI Stereo http://www.ai.sri.com/~konolige/


http://www.ai.sri.com/~konolige/svs/images/facel.jpg

http://www.ai.sri.com/~konolige/svs/images/facer.jpg

http://www.ai.sri.com/~konolige/svs/images/facesst.jpg

http://www.ai.sri.com/~konolige/svs/images/face3d1.jpg




6. Other 3D techniques: 6. Other 3D techniques: Structured LightStructured Light

• Sweep the scene with a laser point or a laser plane

b

LaserPlane

LightProjector

camera

θ


Structured LightStructured Light• 3D information from triangulation

z

xC

ImagePlane

bθ

LightProjector

f

3D point(x,y,z)

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

−=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

+=

==

fyx

xfb

zyx

bxz

zyfy

zxfx

''

'cot

tan

''

θ

θ

x’


Other 3D techniques: Other 3D techniques: Range CameraRange Camera

Depth ImageIntensity Image


Other 3D techniques: Other 3D techniques: 3D Laser Scanner3D Laser Scanner

2000

3000

4000

5000

-10000

1000

500

1000

1500

2000

Otilio


2D 2D Laser Laser Scanner + Scanner + MotionMotion• Scan Alignment and 3-D Surface Modeling with a Helicopter Platform

by Sebastian Thrun, Mark Diel, and Dirk Haehnel, 2003 International Conference of Field and Service Robotics.

2D Laser Scanner

lesson 9: 3d vision - unizar.eswebdiis.unizar.es/~neira/12082/3dvision.pdf · 12082 - computer...

Documents