introduction to image processing and computer...

Ivo Ihrke / Winter 2013

Introduction to Image Processing and Computer Vision

-- Noise, Dynamic Range and Color --

Winter 2013

Ivo Ihrke


Organizational Issues

I received your email addresses

Course announcements will be send via email

Course webpage at http://giana.mmci.uni-saarland.de/

Teaching -> Computational Optical Imaging

http://giana.mmci.uni-saarland.de/


Digital Images

Images are now numbers (corrupted by noise)

±σ ±σ ±σ ±σ ±σ ±σ

±σ ±σ ±σ ±σ ±σ ±σ

±σ ±σ ±σ ±σ ±σ ±σ

±σ ±σ ±σ ±σ ±σ ±σ

±σ ±σ ±σ ±σ ±σ ±σ

±σ ±σ ±σ ±σ ±σ ±σ


Digital

Sensor noise

Dynamic Range

Tone Curve

Recording “Medium”

Monochromatic

Optical

Distortions

Aberrations

Digital Images - Limitations


Dynamic Range


Dynamic Range

max output swing

noise in the dark

Saturation level – dark current

Dark shot noise + readout noise=

“noise in the dark” is random noise sources that

cannot be corrected with circuit tricks

Photon shot noise and read noise

dr =


Dynamic Range of Standard Sensors

ww

w.d

xom

ark

.com

13.5 EVs or f-stops = contrast 11,000:1 = color negative


Dependency of Dynamic Range on ISO

“Unity gain” is where 1 digital unit (ADU) equals 1 electron (e-)

This happens at different ISO settings for different sensors

Above that, the gain only increases the voltage before A/D conversion (possibly reducing the relative effect of some of the read noise)

“digital gain” multiplies the digital values

All gain settings beyond unity gain reducedynamic range


Dependency of Dynamic Range on ISO


Camera Response Curve


Radiometric Response Curve

Raw sensor readings are

─ usually linear (CCD),

─ for CMOS it can be non-linear (depending on amplifier type)

On-chip processing may modify the relation #photons/digital number

─ i.e. the mapping irradiance to digital value

may become non-linear

This mapping is called radiometric response curve

(linear) (log)


Log Encoding

CMOS – pixel amplifier output may be logarithmic

Example: advertisement for IMSChips HDRC-MDC04

Log intensityPix

el V

alu

e

eye HDRC


Programmable Response Curve

Example: Photonfocus MDC-1024


• the process of determining the radiometric response curve

EIg

:1

• Use a color chart with precisely known reflectances.

• Use more camera exposures to fill up the curve.

• Method assumes constant lighting on all patches

• Works best when source is far away (example sunlight).

• Inverse exists (g is monotonic and smooth for all cameras)

Irradiance = const * Reflectance

Pix

el V

alu

es

3.1%9.0%19.8%36.2%59.1%90%

0

255

0 1

g

?

?1

g

Radiometric Calibration


Response Curve - Practice

Measurement: ColorCalibrationToolbox http://giana.mmci.uni-saarland.de/website-template/software.php

Example: 29 exposures of Gretag-Macbeth color checker

(uses EXIF info - exiftool)

http://giana.mmci.uni-saarland.de/website-template/software.php


Color Calibration Toolbox

Zoom-in



Mark the patch rectangle



Make sure the patches are properly extracted



Verify response curve – the example is for jpg on the Canon 5D mark II

Make sure the samples are fit well

Response curves

(R,G,B)

Inverse response curves

samples



Check HDR image



How is the curve estimated ?

Variant of Mitsunaga and Nayar, “Radiometric Self Calibration”, CVPR 1999

Polynomial fit to data samples

Variations:

─ enforce monotonicity (derivative > 0)

Prevents “wiggling”

─ enforce passing of curve through (0,0) and (1,1)

map range to range

─ Perform a weighted fit

accounts for sample non-uniformity


Response Curve – Take Home Points

Usually linear for RAW images

Don’t rely on it – verify

Usually non-linear (gamma) for jpg or other compressed/processed formats

Estimation from random images may be unstable

Use well defined target (color checker)

Prefer continuous-curve algorithms, especially for high bit depths


Gamma Mapping/Correction

Account for properties of human vision

─ Logarithmic, similar to hearing

─ Approximated by “Gamma” curve


Gamma Mapping: Effect

Emphasizes contrast on lower end of linear intensity range for γ<1

Emphasizes contrast on higher end of linear intensity range for γ>1


Textbook

• HDR image / video encoding

• capture, display, tone reproduction

• visible difference predictors

• image based lighting, etc.


Color


Spectrum to Image

not a huge problem: humans have only three types of cones (color vision) and one type of rod (night vision)

cones 6-7 million

rods ~120 million

rods more sensitive


Color Vision

color vision by cones

significant overlap of the response functions

L = long

M = mid

S = short


Color Vision

L ~63%, M ~31%, S ~6% of cones

eye least sensitive to blue, most sensitive to yellowish-green

spectral region outside of support of the response functions cannot be perceived


Spectral response of human eye

reproducing color is tricky

color matching experiments

─ use light source with known spectral distribution

(i.e. assume uniform spectral distribution, can e.g. be achieved by normalization) filtered by a narrow band filter

─ additionally, use monochromatic sources

@ 700,546,435 nm

─ let human observers adjust apparent brightness of one of the sources to match the other

Color matching functions


Color Spaces

─ RGB matching functions

negative !


XYZ space

The CIE (1931) standard observer


How to compute a tristimulus image from a spectral representation ?

We have to integrate with the spectrum with the appropriate color matching function

dxlfxIXX ),(ˆ)()(

dxlfxIYY ),(ˆ)()(

dxlfxIZZ ),(ˆ)()(


Now to RGB

convert XYZ to RGB

Possibly to sRGB (non-linear space, gamma)

Where C = {R,G,B}


Other Color Spaces

Many linearly related spaces exist

have different separation properties

example: YCbCr (JPEG)

[Wikipedia]

Y=0 Y=0.5 Y=1.0

Cb

Cr


Sensing color

Eye has 3 types of color receptors

Therefore we need 3 different spectral sensitivities

sourc

e: K

odak

KA

F-5

101ce

dat

a sh

eet


Ways to sense color

Field-sequential color

simplest to implement

only still scenes

Proudkin-Gorskii, 1911

(Library of Congress exhibition)


Examples - Prokudin-Gorskij

Self-portrait 1915

http://upload.wikimedia.org/wikipedia/commons/b/b2/Sergei-Prokudin-Gorski-Larg.jpg

http://upload.wikimedia.org/wikipedia/commons/b/b2/Sergei-Prokudin-Gorski-Larg.jpg


Photograph 1910, Emir of Bukhara , Prokudin-Gorskii

Examples - Prokudin-Gorskij


Examples - Lew Tolstoy

1910 photograph, Sergey Prokudin-Gorskii1887 painting, Ilya Repin

http://upload.wikimedia.org/wikipedia/commons/c/c6/L.N.Tolstoy_Prokudin-Gorsky.jpg

http://upload.wikimedia.org/wikipedia/commons/c/c6/L.N.Tolstoy_Prokudin-Gorsky.jpg

http://upload.wikimedia.org/wikipedia/commons/b/bb/Ilya_Efimovich_Repin_(1844-1930)_-_Portrait_of_Leo_Tolstoy_(1887).jpg

http://upload.wikimedia.org/wikipedia/commons/b/bb/Ilya_Efimovich_Repin_(1844-1930)_-_Portrait_of_Leo_Tolstoy_(1887).jpg


Color Wheel

one color channel is captured at one shot

3 times the acquisition time

static images only


Ways to sense color – 3-Chip Camera

dichroic mirrors divide light into wavelength bands

does not remove light: excellent quality but expensive

interacts with lens design

problem with polarization

image: Theuwissen


Foveon Technology

3 layers capture RGB at the same location

takes advantage of silicon’s wavelength selectivity

light decays at different rates for different wavelengths

multilayer CMOS sensor gets3 different spectral sensitivities

don’t get to choose the curves


Ways to sense color

Color filter array

cover each sensor with an individual filter

requires just one chip but loses some spatial resolution

“demosaicing” requires tricky image processing

G R

B G

primary


take four images, moving the sensor by one pixel

(use fourth image for noise reduction)

can be used for supersampling(move by ½, ¼ pixel)

Multi-Shot (example Jenoptik C14)


Demosaicing

bilinear interpolation

sampling theory

edge-directed/pattern-based interpolation

correlation-based


Example raw image (Canon 5D markII)

Raw imageProcessed image

Sett

ings: f/

8 1

/25s I

SO

: 800,

no n

ois

e r

eduction



Raw imageProcessed image



Raw

im

age

Pro

cessed im

age

Colo

rs a

ssig

ned


Demosaicing

Original image Bilinear interpolationRon Kimmel, http://www.cs.technion.ac.il/~ron/demosaic.html


Demosaicing

Ron Kimmel, http://www.cs.technion.ac.il/~ron/demosaic.html

Bilinear interpolation Edge-weighted interpolation


Bilinear Interpolation

perform interpolation for each color channel separately

G R

B G

= + +



G R

B G

= + +

4

3432141223

RRRRR



G R

B G

= + +

4

3432141223

RRRRR

2

343233

RRR



set all non-measured values to zero then convolve

G R

B G

= + +

4/

121

242

121

,

BRF 4/

010

141

010

GF


Color – White Balancing


White Balance

daylight

flashflourescent

tungsten

Colors appear

different under

different

illumination

conditions


White Balance

Why is there a constant appearance for human observers ?

─ Human perception adapts to illumination condition

Practice: division of RGB values


White Balance

Camera built-in function

derive scale from white point

sun

incandescent

tungsten

infraredred green blue

ultra violet

wavelength


White Balance

Camera built-in function

derive scale from white point

infraredred green blue

ultra violet

wavelength


Horseshoe Diagram


White Point for Different Color Temperatures

Planckian

Locus:

- convert black body

temperature to XYZ and

put intohorseshoe diagram

L_\lambda = spectral radiance [W/m^2/m]

lambda = wavelength [m]

h = Planck’s constant [J.s]

k = Boltzmann constant [J/K]

c = speed of light [m/s]

T = temperature of black body [K]


White Balance

Human perception adapts to illumination condition

Practice: division of RGB values

Theory: achieve a “neutral” spectrum

(only works for broad band sources and

broad band reflectance)

Conversion to RGB is an integral over the divided spectrum + linear transformation + gamma


White Balance

daylight

flashflourescent

tungsten

capture the spectral

characteristics of

the light source to

assure correct

color reproduction


Textbook

• Physical principles

• color spaces, encoding

• chromatic adaption, perceptual

issues

• Display technology,

color management …


Compression


Compression

Lossless

Entropy coding based (e.g. Huffmann coding)

Popular example: zip/gzip (used in png format)

reproduces exact copy of bit signals

Lossy

Takes advantage of the fact that information is an image (removes “inperceptual” data)

Popular example: jpeg


Compression

Example: Jpeg

Selectable compression ratio

Picture: gradually varying compression ratio from left to right

[Wikipedia]


Compression

JPEG:

Convert image to YCbCr

Subsample chromachannels (Cb,Cr)

Split into 8x8 blocks

Apply discrete cosine transform

Remove small coefficients

Color channels are treated independently

8x8 DCT basis


Bibliography

Holst, G. CCD Arrays, Cameras, and Displays. SPIE Optical

Engineering Press, Bellingham, Washington, 1998.

Theuwissen, A. Solid-State Imaging with Charge-Coupled Devices. Kluwer Academic Publishers, Boston, 1995.

Curless, CSE558 lecture notes (UW, Spring 01).

El Gamal et al., EE392b lecture notes (Spring 01).

Several Kodak Application Notes at http://www.kodak.com/global/en/digital/ccd/publications/a

pplicationNotes.jhtml

Reibel et al., CCD or CMOS camera noise characterization, Eur. Phys. J. AP 21, 2003

http://www.kodak.com/global/en/digital/ccd/publications/applicationNotes.jhtml


(ICC –international color consortium)

color management system

capture the properties of all devices

─ camera and lighting

─ monitor settings

─ output properties

common interchange space

sRGB standard as a definition of RGB

ICC Profiles

input device

(e.g. camera)

input profile

profile

connection

space

output device

(e.g. printer)

output profile

display

device

(e.g.

monitor)

monitor

profile


profile connection spaces

─ CIELAB (perceptual linear)

─ linear CIEXYZ color space

can be used to create an high dynamic range image in the profile connection space

allows for a color calibrated workflow

ICC Profiles and HDR Image Generation

input device

(e.g. camera)

input profile

profile

connection

space

output device

(e.g. printer)

output profile

...


Other HDR approaches

Determine for each pixel when enough photons haven been collected.

Logarithmic timings yields floating point representation (mantissa + exponent).


PMD

measured distance in each pixel

exploit interference

─ emit light (modulated) at each pixel

─ compare reflected light to reference light

computation in a “smart” pixel


Next week

Signal Processing


What is High Dynamic Range (HDR)?

http://en.wikipedia.org/wiki/High-dynamic-range_imaging


HDR Acquisition – Exposure Brackets

[Debevec & Malik 97]

Radiance Map Tonemapped HDR Image

Exposure Sequence


Shutter Speed

F/stop (aperture)

Neutral Density (ND) Filters

Gain / ISO / Film Speed

(DOF)

(noise)

Ways to vary the exposure


Combining the image

dttxlxI ),()(

• scene constant over exposure time (or ND-filter)

radiance

),()( xltxI

• assumes linear response (radiometric calibration!)

have several measurements with different t


Combining the image

• introduce a weighting function for the pixels:

• centered at the sensor mean value,

e.g. Gaussian (image data in [0,1])

• compute final image as

2

2

2.0

)5.0)((

))((

xI

exIw

i

i

i

iii

finalxIw

txIxIw

xI))((

/)())((

)(

introduction to image processing and computer...

Documents