eecs 442 computer vision cameraswithout cameras we wouldn’t have c.v. • pinhole cameras •...

57
EECS 442 Computer vision Cameras without cameras we wouldn’t have C.V. Pinhole cameras Cameras & lenses The geometry of pinhole cameras Other camera models Reading: [FP] Chapters 1 3 [HZ] Chapter 6 Some slides in this lecture are courtesy to Profs. J. Ponce, S. Seitz, F-F Li

Upload: others

Post on 21-Sep-2020

7 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

EECS 442 – Computer vision

Cameras without cameras we wouldn’t have C.V.

• Pinhole cameras

• Cameras & lenses

• The geometry of pinhole cameras

• Other camera models

Reading: [FP] Chapters 1 – 3

[HZ] Chapter 6

Some slides in this lecture are courtesy to Profs. J. Ponce, S. Seitz, F-F Li

Page 2: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

EECS 442 – Computer vision

Cameras without cameras we wouldn’t have C.V.

• Pinhole cameras

• Cameras & lenses

• The geometry of pinhole cameras

• Other camera models

Reading: [FP] Chapters 1 – 3

[HZ] Chapter 6

Some slides in this lecture are courtesy to Profs. J. Ponce, S. Seitz, F-F Li

Page 3: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

How do we see the world?

• Let’s design a camera – Idea 1: put a piece of film in front of an object

– Do we get a reasonable image?

Page 4: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Pinhole camera

• Add a barrier to block off most of the rays

– This reduces blurring

– The opening known as the aperture

Page 5: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Pinhole perspective projection Pinhole camera f

f = focal length

c = center of the camera

c

Page 6: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Milestones:

• Leonardo da Vinci (1452-1519):

first record of camera obscura

Some history…

Page 7: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Milestones:

• Leonardo da Vinci (1452-1519):

first record of camera obscura

• Johann Zahn (1685): first

portable camera

Some history…

Page 8: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Photography (Niepce, ―La

Table Servie,‖ 1822)

Milestones:

• Leonardo da Vinci (1452-1519):

first record of camera obscura

• Johann Zahn (1685): first

portable camera

• Joseph Nicephore Niepce (1822):

first photo - birth of photography

Some history…

Page 9: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Photography (Niepce, ―La

Table Servie,‖ 1822)

Milestones:

• Leonardo da Vinci (1452-1519):

first record of camera obscura

• Johann Zahn (1685): first

portable camera

• Joseph Nicephore Niepce (1822):

first photo - birth of photography

• Daguerréotypes (1839)

• Photographic Film (Eastman, 1889)

• Cinema (Lumière Brothers, 1895)

• Color Photography (Lumière Brothers,

1908)

Some history…

Page 10: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Let’s also not forget…

Motzu

(468-376 BC)

Aristotle

(384-322 BC)

Also: Plato, Euclid

Al-Kindi (c. 801–873)

Ibn al-Haitham

(965-1040) Oldest existent

book on geometry

in China

Page 11: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

z

yf'y

z

xf'x

y

xP

z

y

x

P

Pinhole camera

Derived using similar triangles

Page 12: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

O

P = [x, z]

P’=[x’, f]

f

z

x

f

x

x

z

Pinhole camera

Page 13: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Common to draw image plane in front of

the focal point. Moving the image plane

merely scales the image.

f f

Pinhole camera

z

yf'y

z

xf'x

Page 14: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Kate lazuka ©

Pinhole camera

Is the size of the aperture important?

Page 15: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Shrinking

aperture

size

- Rays are mixed up

Adding lenses!

-Why the aperture cannot be too small?

-Less light passes through

-Diffraction effect

Page 16: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Cameras & Lenses

• A lens focuses light onto the film

Page 17: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

• A lens focuses light onto the film

– Rays passing through the center are not deviated

– All parallel rays converge to one point on a plane

located at the focal length f

focal point

f

Cameras & Lenses

Page 18: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

• A lens focuses light onto the film

– There is a specific distance at which objects are ―in

focus‖

[other points project to a ―circle of confusion‖ in the image]

―circle of

confusion‖

Cameras & Lenses

Page 19: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Snell’s law

n1 sin 1 = n2 sin 2

Cameras & Lenses

1 = incident angle

2 = refraction angle

ni = index of refraction

Page 20: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Thin Lenses

Snell’s law:

n1 sin 1 = n2 sin 2

Small angles:

n1 1 n2 2

n1 = n (lens)

n1 = 1 (air)

zo

z

y'z'y

z

x'z'x

ozf'z

)1n(2

Rf

Page 21: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

A compound lens Lenses are combined in various ways…

Source wikipedia

Page 22: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

f

f

f

Sourc

e w

ikip

edia

Dolly zooms

Compressed perception of depth

Exaggerated perception of depth

Page 23: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Issues with lenses: Chromatic Aberration

• Lens has different refractive indices for different

wavelengths: causes color fringing

)1n(2

Rf

Page 24: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Issues with lenses: Spherical aberration

• Rays farther from the

optical axis focus closer

Page 25: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

No distortion

Pin cushion

Barrel (fisheye lens)

Issues with lenses: Radial Distortion

– Deviations are most noticeable for rays that pass through

the edge of the lens

Image magnification

decreases with distance

from the optical axis

Page 26: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Cameras

• Pinhole cameras

• Cameras & lenses

• The geometry of pinhole cameras

• Other camera models

Page 27: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Pinhole perspective projection Pinhole camera f

f = focal length

c = center of the camera

c

)z

yf,

z

xf()z,y,x(

2E

3

Page 28: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Is this a linear transformation?

No — division by z is nonlinear

How to make it linear?

)z

yf,

z

xf()z,y,x(

Page 29: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Homogeneous coordinates

homogeneous image

coordinates

homogeneous scene

coordinates

• Converting from homogeneous

coordinates

Page 30: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Perspective Projection

Transformation

1

z

y

x

0100

00f0

000f

z

yf

xf

X

z

yf

z

xf

iX

XMX

M

3H

4

Page 31: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

From retina plane to images

Pixels, bottom-left coordinate systems

Page 32: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Coordinate systems

xc

yc

Page 33: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Converting to pixels

x

y

xc

yc

C=[cx, cy]

)cz

yf,c

z

xf()z,y,x( yx

1. Off set

Page 34: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Converting to pixels

)cz

ylf,c

z

xkf()z,y,x( yx

1. Off set

2. From metric to pixels

x

y

xc

yc

C=[cx, cy] Units: k,l : pixel/m

f : m : pixel

, Non-square pixels

Page 35: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Matrix form?

x

y

xc

yc

C=[cx, cy]

)cz

y,c

z

x()z,y,x( yx

Converting to pixels

Page 36: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Camera Matrix

)cz

y,c

z

x()z,y,x( yx

1

z

y

x

0100

0c0

0c0

z

zcy

zcx

X y

x

y

x

x

y

xc

yc

C=[cx, cy]

Page 37: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

XMX

X0IK

Camera Matrix

1

z

y

x

0100

0c0

0c0

X y

x

Camera

matrix K

Page 38: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Finite projective cameras

1

z

y

x

0100

0c0

0cs

X y

x

Skew parameter

x

y

xc

yc

C=[cx, cy] K has 5 degrees of freedom!

Page 39: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

World reference system

Ow

iw

kw

jw

R,T

•The mapping so far is defined within the camera

reference system

• What if an object is represented in the world

reference system

Page 40: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

3D Rotation of Points Rotation around the coordinate axes, counter-clockwise:

100

0cossin

0sincos

)(

cos0sin

010

sin0cos

)(

cossin0

sincos0

001

)(

z

y

x

R

R

R

p

x

Y’

p’

x’

y

z

Page 41: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

wXMX wXTRK

World reference system

Ow

iw

kw

jw

R,T

wXTR

X44

10

In 4D homogeneous coordinates:

Internal parameters External parameters

Page 42: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

wx XMX 4313

14w4333 XTRK

Projective cameras

Ow

iw

kw

jw

R,T

How many degrees of freedom?

5 + 3 + 3 =11!

100

c0

cs

K y

x

Page 43: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Projective cameras

Ow

iw

kw

jw

R,T

)X

X,

X

X()z,y,x(

w3

w2

w3

w1w

m

m

m

m M is defined up to scale!

Multiplying M by a scalar

won’t change the image

w13 XMX

14w4333 XTRK

3

2

1

m

m

m

M

Page 44: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Theorem (Faugeras, 1993)

][ bATKRKTRKM

3

2

1

a

a

a

A

100

0 y

x

c

cs

K

lf

;kf

Page 45: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Properties of Projection •Points project to points

•Lines project to lines

Page 46: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Properties of Projection

•Distant objects look smaller

z

y'f'y

z

x'f'x

Image plane

scene

Page 47: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Properties of Projection

•Angles are not preserved

•Parallel lines meet! Vanishing point

Page 48: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Vanishing points

• Each set of parallel lines meets at a different point

[The vanishing point for this direction]

•Sets of parallel lines on the same plane lead to collinear

vanishing points [The line is called the horizon for that plane]

Page 49: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

One-point perspective • Masaccio, Trinity,

Santa Maria

Novella, Florence,

1425-28

Credit slide S. Lazebnik

Page 50: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Cameras

• Pinhole cameras

• Cameras & lenses

• The geometry of pinhole cameras

• Other camera models

Page 51: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Weak perspective projection

yz

fy

xz

fx

yz

fy

xz

fx

0

0

''

''

''

''

Magnification m

P ~

When the relative scene depth is small compared to its distance from the camera

Page 52: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Orthographic (affine) projection

Distance from center of projection to image plane is infinite

yy

xx

yz

fy

xz

fx

'

'

''

''

Page 53: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Pros and Cons of These Models

• Weak perspective much simpler math. – Accurate when object is small and distant.

– Most useful for recognition.

• Pinhole perspective much more accurate for scenes. – Used in structure from motion.

Page 54: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Qingming Festival by the Riverside Zhang Zeduan ~900 AD

Weak perspective projection

Page 55: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

The Kangxi Emperor's Southern Inspection Tour (1691-1698) By Wang Hui

Page 56: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

58

http://www.eecs.umich.edu/~silvio/teaching/EECS442_2009/UCSD_videos/3DVisi

on.avi

HW 0.2: watch:

Marilia Maschion Producer & Director UCSD ICAM

undergraduate

Vincent Rabaud Technical Advisor UCSD CSE student

Serge Belongie Project Advisor UCSD CSE Department

Page 57: EECS 442 Computer vision Cameraswithout cameras we wouldn’t have C.V. • Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras • Other camera models Reading:

Next lecture

•How to calibrate a camera?