administration (cont.) schedule - · pdf file17.04.07 panorama and stitching 10.04.07 passover...

1

1

(Topics in) Video Processing Computer Science Semester B

Yacov [email protected]

Yossi [email protected]

Some slides were taken from: Bahadir Gunturk, Yung-Yu Chuang, Ran Eshel 2

Administration

• Pre-requisites / prior knowledge

• Regular course – not a seminar

• Course Home Page:– Lecture slides and handouts

– “What’s new”

– Homework, grades

• Exercises: – Programming in Matlab, ~3 Assignments

– Final project

3

Administration (Cont.)• Matlab software:

– Available in PC labs

– Student version

– For next week: Run Matlab “demo” and read Matlab primer until section 13.

• Grading policy: – Final Grade will be based on: Exercises (60%) , Final project (40%)– Exercises will be weighted – Exercises can be submitted in pairs

• Office Hours: by email appointment to [email protected]

4 Video Coding (guest lecture)05.06.07

Tracking / Recognition (project presentation)29.05.07

Shavuot22.05.07

Guest lecture15.05.07

High-Dynamic Range08.05.07

Super-resolution01.05.07

Independence day24.04.07

Panorama and stitching17.04.07

Passover holiday10.04.07

Passover holiday03.04.07

Registration27.03.07

Post-acquisition processing 220.03.07

Post-acquisition processing 113.03.07

Acquisition06.03.07

Introduction27.02.07

SubjectDate

Schedule

2

5

Further Reading

Multidimensional Signal, Image, and Video Processing and Coding / John .W. Woods

Digital Video Processing / Murat Tekalp

Video Processing and Communications / Yao Wang, JôrnOstermann, Ya-Qin Zhang,

Handbook of Image and Video Processing / Alan C. Bovik

6

Syllabus• Introduction

Pinhole camera modelShading models Light and colorHVS pathway

• AcquisitionCamera pipe-lineSensorsTemporal sampling (interlacing/progressive)Spatial sampling (Bayer)Noise models & distortionsCamera parameters trade-offsVideo formats

• Post-Acquisition Processing Geometrical distortion rectificationWhite balancingDe-interlacingDe-mosaicingDe-noising

7

• Image RegistrationGlobal motion registration Dense motion: optical flow

• Spatio-Temporal ProcessingMosaicing: panorama, stitching, blending Video summarizingVideo in-painting

• Enhancement & RestorationSuper-resolution: spatial/temporalHigh Dynamic Range

• Tracking (tentative)Kalman-filteringParticle-filteringMean-Shift

• RecognitionAction detectionAnomaly behavior detection

• CodingVideo Compression

8

Introduction (today)

• What is an image ?• What is a color ?

3

9

Acquisition

10

– Camera pipe-line– Sensors– Temporal sampling (interlacing/progressive)– Spatial sampling (Bayer)– Noise models & distortions– Camera parameters trade-offs– Video formats

11

Post-acquisition Processing

– Geometrical distortion rectification– White Balancing– De-interlacing– De-mosaicing– De-noising

12Image De-mosaicing

4

13

De-interlacing14Correcting radial distortion

from Helmut Dersch

15

White Balancingautomatic white balancewarmer +3

16

Image Registration

– Global motion registration– Dense motion: Optical Flow

5

17 Global motion registration 18 Optical Flow

19

Spatio-Temporal Processing

– Mosaicing: panorama, stitching, blending – Video summarizing– Video in-painting

t

x

y

20 Panorama

++

++

++

++

example: http://www.cs.washington.edu/education/courses/cse590ss/01wi/projects/project1/students/dougz/index.html

6

21Video Panorama

22 Video summarizing

23 24

Video inpainting

7

25

Enhancement and Restoration– Super-resolution: spatial/temporal– High Dynamic Range

Shutter Duration

Aperture

Under Exposure:Bad signal/noise ratio

High Aperture:Narrow depth of field

Long Shutter:Motion blur

Over Exposure:Saturated image

26

HDR

27

HDR

28

HDR

8

29

Example – Low Light

30

31

Example - Super-resoluton

32

9

33 34

35

Action detection / recognition

– Action detection– Anomaly behavior detection

36 Anomaly behavior detection

10

37

Video Coding

• Compression• Video formats

38

Video ProcessingIntroduction

39

Computer Vision

Rendering

Image/video Processing

Model3D ObjectGeometric Modeling

2D Images

The Visual Sciences

40

Image/video Processing

Computer Vision

Low Level

High Level

Image/Video Processing - Computer Vision

Acquisition, representation,compression,transmission

image enhancement

edge/feature extraction

Pattern matching

image "understanding“(Recognition, 3D)

11

41

Today’s Plan

• Light and the EM spectrum• The H.V.S. and Color Perception

42

What is an Image ?• An image is a projection of a 3D scene into a 2D

projection plane.• An image can be defined as a 2 variable function I(x,y) ,

where for each position (x,y) in the projection plane, I(x,y) defines the light intensity at this point.

43

Camera trial #1

scene film

Put a piece of film in front of an object.

source: Yung-Yu Chuang44

Pinhole camera

scene film

Add a barrier to block off most of the rays.• It reduces blurring• The pinhole is known as the aperture• The image is inverted

barrier

pinhole camera

source: Yung-Yu Chuang

12

45 46

XY

(x,y,z)

(x,y)

center of projection(pinhole)

d

d – focal length

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

−=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

1010000100001

ZYX

dwyx

The Pinhole Camera Model (where)

Z

47

The Shading Model (what)

Shading Model: Given the illumination incident at a point on a surface, what is reflected?

48

Shading Model Parameters

• The factors determining the shading effects are:

– The light source properties:• Positions, Electromagnetic Spectrum, Shape.

– The surface properties:• Position, orientation, Reflectance properties.

– The eye (camera) properties:• Position, orientation, Sensor spectrum sensitivities.

13

49

Newton’s Experiment, 1665 Cambridge.Discovering the fundamental spectral components of light.

Light and the Visible Spectrum

50

The light Spectrum

Electromagnetic Radiation - Spectrum

Gamma X rays Infrared Radar FM TV AMUltra-violet

10-12

10-8

10-4

104

1 108

electricityACShort-

wave

400 nm 500 nm 600 nm 700 nmWavelength in nanometers (nm)

Wavelength in meters (m)

Visible light

51

MonochromatorsMonochromators measure the power or energy at different wavelengths

52

The Spectral Power Distribution (SPD) of a light is a function e(λ) which defines the energy at each wavelength.

Wavelength (λ)

400 500 600 7000

0.5

1

Rel

ativ

e P

ower

Spectral Power Distribution

14

53

Examples of Spectral Power Distributions

Blue Skylight Tungsten bulb

Red monitor phosphor Monochromatic light

400 500 600 7000

0.5

1

400 500 600 7000

0.5

1

400 500 600 7000

0.5

1

400 500 600 7000

0.5

1

54

Specular reflection mirror like reflection at the surface

Diffuse (lambertian) reflection reflected randomly between color particlesreflection is equal in all directions

Incident light Specular reflection

Diffuse reflection

normal

Surface Parameters

55

Different Types of Surfaces

56

400 500 600 700

0.2

0.40.6

0.81

400 500 600 700

0.2

0.40.6

0.81

400 500 600 700

0.20.4

0.60.8

1

400 500 600 700

0.20.4

0.60.8

1

Surface Body Reflectances (albedo)

Yellow Red

Blue Gray

Wavelength (nm)

Spectral Property of Lambertian Surfaces

15

57

θ

NL

R

V

Ambient reflection: Iamb= K(λ) ea(λ)

Diffuse reflection: Idiff= K(λ) ep(λ) (N⋅L)

Specular reflection: Ispec= Ks(λ)ep (λ) (R⋅V)n

• ep ea - the ambient and point light intensities. • K , Ks ∈ [0,1] - the surface ambient / diffuse / specular reflectivity. • N - the surface normal, L - the light direction, V – viewing direction

Surface propertiesLight properties

geometry

58

θ

NL

R

V

Ambient reflection: Iamb= K(λ) ea(λ)

Diffuse reflection: Idiff= K(λ) ep(λ) (N⋅L)

Specular reflection: Ispec= Ks(λ)ep (λ) (R⋅V)n

• ep ea - the ambient and point light intensities. • K , Ks ∈ [0,1] - the surface ambient / diffuse / specular reflectivity. • N - the surface normal, L - the light direction, V – viewing direction

59

Diffusesurface

Ambientsurface

Diffuse +

Specular

60

I(λ) = Iamb+Idiff+Ispec

• The final illumination equation:

• If several light sources are placed in the scene:

I(λ)= Iamb+Σk (Ikdiff+Ik

spec)

16

61

Composition of Light Sources

62

Optic NerveFovea

Vitreous

Optic Disc

Lens

Pupil

Cornea

Ocular MuscleRetina

Humor

Iris

The Human Visual System

Cornea - קרנית Pupil - אישו ן Iris - קשתית Retina - רשתית

63 64

The Visual Pathway

Retina

Optic Nerve

Optic Chiasm

LateralGeniculateNucleus (LGN)

Visual Cortex

17

65

Eye v.s. Camera

Yaho Wang’s slides66

light

rods cones

horizontal

amacrine

bipolar

ganglion

The Human Retina

67

• Retina contains 2 types of photo-receptors– Cones:

• Day vision, can perceive color tone

– Rods: • Night vision, perceive brightness only

68

Cones:• High illumination levels (Photopic vision)• Sensitive to color (there are three cone types: L,M,S)• Produces high-resolution vision• 6-7 million cone receptors, located primarily in the central portion of the retina

Wavelength (nm)

Rel

ativ

e se

nsiti

vity

Cone Spectral Sensitivity

400 500 600 7000

0.25

0.5

0.75

1ML

SM

A side note:• Humans and some monkeys have three types of cones (trichromatic vision); most other mammals have two types of cones (dichromatic vision).• Marine mammals have one type of cone.• Most birds and fish have four types. •Lacking one or more type of cones result in color blindness.

18

69

Rods:• Low illumination levels (Scotopic vision).• Highly sensitive (respond to a single photon).• Produces lower-resolution vision• 100 million rods in each eye.• No rods in fovea.

Wavelength (nm)

Rel

ativ

e se

nsiti

vity

400 500 600 7000

0.25

0.5

0.75

1

Rod Spectral Sensitivity

70rods

S - Cones

L/M - Cones

Foveal Periphery photoreceptorsPhotoreceptor Distribution

71

Cone Receptor Mosaic(Roorda and Williams, 1999)

L-cones M-cones S-cones 72

Distribution of rod and cone photoreceptors

Degrees of Visual Angle

Rec

epto

rs p

er s

quar

e m

m

-60 -40 -20 0 20 40 60

2

6

10

14

18x 104

rodscones

Cone’s Distribution:• L-cones (Red) occur at about ~65% of the cones throughout the retina .

• M-cones (green) occur at about ~30% of the cones.

• S-cones (blue) occur at about ~2-5% of the cones (Why so few?).

fovea

19

73

The Cone Responses

Assuming Lambertian Surfaces

IlluminantSensors Surface

e(λ) – Fixed, point source illuminantk(λ) –surface’s reflectancel(λ),m(λ),s(λ) – Cone responsivities

Output

∫= )()()( λλλ kelL

∫= )()()( λλλ kemM

∫= )()()( λλλ kesS

74

Metamer - two lights that appear the same visually. They might have different SPDs(spectral power distributions).

400 500 600 7000

400

800

400 500 600 7000

100

200

Wavelength (nm)

Pow

er

The phosphors of the monitor were set to match the tungsten light.

Tungsten light Monitor emission

75

The Trichromatic Color Theory

Thomas Young (1773-1829) -A few different retinal receptors operating with different wavelength sensitivities will allow humans to perceivethe number of colors that they do.Suggested 3 receptors.

Helmholtz & Maxwell (1850) -Color matching with 3 primaries.

Trichromatic: “tri”=three “chroma”=colorcolor vision is based on three primaries (i.e., it is 3D).

76

Color Matching Experiment

+ -

+ -

+ -

test match

Primaries

• Given a set of 3 primaries, one can determine for every spectraldistribution, the intensity of the guns required to match the color of that spectral distribution.

• The 3 numbers can serve as a color representation.

( ) ( ) ( ) ( )λλλλ bBgGrRT ++≡

R(λ)

G(λ)

B(λ)

T(λ)

20

77

Color matching experiment for Monochromatic lights

400 500 600 7000

0.5

1

400 500 600 7000

0.5

1

400 500 600 7000

0.5

1

Primary Intensities

78

r(λ)

g(λ)b(λ)

400 500 600 700

0

1

2

3

Wavelength (nm)

Prim

ary

Inte

nsity

Stiles & Burch (1959) Color matching functions. Primaries are: 444.4 525.3 and 645.2

Problems: Some perceived colors cannot be generated. This is true for any choice of visible primaries.

79

• Observation - Color matching is linear:– if (S≡P) then (S+N≡P+N) – if (S≡P) then (α S≡ α P)

• Outcome 1: Any T(λ) can be matched:

• Outcome 2: CMF can be calculated for any chosen primaries U(λ), V(λ), W(λ):

( ) ( ) ( ) ( ) ( ) ( ) λλλλλλλλλ dbTbdgTgdrTr ∫∫∫ === ;;

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

bgr

ccccccccc

wvu

bwgwrw

bvgvrv

buguru

80

• The CIE (Commission Internationale d’Eclairage) defined three hypothetical lights X, Y, and Z whose matching functions are positive everywhere:

The CIE Color Standard

21

81

TristimulusLet X, Y, and Z be the tristimulus values.

A color can be specified by its trichromatic coefficients, defined as

XxX Y Z

=+ +

YyX Y Z

=+ +

ZzX Y Z

=+ +

X ratio

Y ratio

Z ratio

Two trichromatic coefficients are enough to specify a color. (x + y + z = 1)

From: Bahadir Gunturk 82

CIE Chromaticity DiagramInput light spectrum

x

y

From: Bahadir Gunturk

83


x

y



x

y


22

85


Boundary

x

y

380nm

700nm



Boundary


87

CIE Chromaticity DiagramLight composition


CIE Chromaticity DiagramLight composition

Light composition


23

89

CIE Chromaticity DiagramThe CIE chromaticity diagram is helpful to determine the range of colors that can be obtained from any given colors in the diagram.

Source: http://hyperphysics.phy-astr.gsu.edu/hbase/vision/visioncon.html#c1

Gamut: The range of colors that can be produced by the given primaries.

http://www.brucelindbloom.com/index.html?Eqn_ChromAdapt.html90

• The sRGB is a device-independent color space. It was created in 1996 by HP and Microsoft for use on monitors and printers.

• It is the most commonly used color space.

• It is defined by a transformation from the xyz color space.

The sRGB Color Standard

91

Color matching predicts matches, not appearance

92

Color Appearance

24

93

Color Appearance

94

Color Appearance

95

Color Spaces

96

RGB Color Space (additive)• Define colors with (r, g, b) amounts of red,

green, and blue

25

97

CMY Color Space (subtractive)• Cyan, magenta, and yellow are the complements of

red, green, and blue– We can use them as filters to subtract from white– The space is the same as RGB except the origin is white

instead of black

98

HSV color space• Hue - the color we see (red, green, purple).• Saturation - how pure is the color (how far the color

from gray ).• Value (brightness) - how bright is the color.

99

HSV - a more intuitive color space

Value

Saturation

Hue

100

Opponent Color Space• Observation: Color bands are highly

correlated in high spatial frequencies

∗),( yxh

26

101

A joint Histogram of rx v.s. gx

Red derivative

Gre

en d

eriv

ativ

e

100 200 300 400 500

50

100

150

200

250

300

350

400

450

500

102

A joint Histogram of gx v.s. bx

Green derivative

Blu

e de

rivat

ive

100 200 300 400 500

50

100

150

200

250

300

350

400

450

500

103

A joint Histogram of rx v.s. bx

Red derivative

Blu

e de

rivat

ive

100 200 300 400 500

50

100

150

200

250

300

350

400

450

500

104

27

105

Joint histograms of R v.s. G for a low pass images.

106

• Define a new color basis (l,c1,c2):

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

−−=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛=

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

211011111

2

1 nTwhereBGR

Tccl

l – luminanceC1- red/greenC2 – blue/yellow

A joint Histogram of rx v.s. gx

Red derivative

Gre

en d

eriv

ativ

e

100 200 300 400 500

50

100

150

200

250

300

350

400

450

500

L

c1

l – luminance valueC1 – Red-GreenC2 – Blue-Yellow

107

Comments:– l channel encodes the color luminance.– C1 and C2 encodes the chrominance. – In the chrominance channels high freq. are

attenuated.– It the luminance channel high freq. are

maintained.– The 3 opponent channels are uncorrelated in

the high freq.– Efficient for encoding

108

High freq. details Low freq. details Low freq. details

Claim: The HVS’ high spatial sensitivity in the luminance domain and low spatial sensitivity in the chrominance domains is a direct outcome of the statistical properties of color images!

28

109

Original Image

110

After blurring C1 and C2 bands

111

After blurring l band as well

112

Opponent Color Spaces

• The standard representation used in TV broadcasting• Backwards compatibility with B/W TV• Low bit rate is needed in the chrominance channels• There are various opponent representations:

– YIQ - used for NTSC color TV – YUV (also called YCbCr) - used for PAL TV and

video

• Question: why S cones are sparsely populated?

29

113

T H E E N D

administration (cont.) schedule - · pdf file17.04.07 panorama and stitching 10.04.07 passover...

Documents