acquisition, representation, display, and perception of ......acquisition, representation, display,...

o

Acquisition, Representation, Display, and Perception of Image and Video Signals

Acquisition, Representation,Display, and Perceptionof Images and Video

Thomas Wiegand Digital Image Communication 1 / 47

o

Acquisition, Representation, Display, and Perception of Image and Video Signals

Outline

Fundamentals of Image FormationImage Formation with LensesDiffraction and Optical Resolution

Visual PerceptionThe Human Visual SystemColor PerceptionVisual Acuity

Representation of Digital Images and VideoSpatio-temporal SamplingColor SpacesNon-linear EncodingThe Y’CbCr Color FormatQuantization of Sample Values

Image AcquisitionImage SensorCapture of Color Images

Display of Images and Video


o

Acquisition, Representation, Display, and Perception of Image and Video Signals Representation of Images and Video

Representation ofImages and Video


o


Representation Formats

Raw data formats for exchanging pictures and videos

Output of camera

Input to video encoder

Output of video decoder

Input to display

Examples: BT.601 (SD), BT.709 (HD), BT.2020 (UHD)

Color images: Three sample arrays (one per color component)

Spatio-temporal sampling

Linear color space (chromaticity coordinates of primaries and white point)

Non-linear encoding (transfer function)

Color representation format (R’G’B’ or Y’CbCr)

Quantization (bit depth)


o


Spatio-Temporal Sampling


Discrete representation of continuous irradiance pattern on image sensor

Each image is represented by W×H sample array

cn[`,m] = ccont( ` ·∆x, m ·∆y, n ·∆t )

Spatial sampling is done by image sensor (photocells of finite size)

Video: Multiple pictures are taken per second

Commonly used frame rates:24/1.001, 24, 25, 30/1.001, 30, 50, 60/1.001, 60 Hz

Gray-level image

2D array of samples

Color image

Three color components

2D array of samples per color component


o


Spatial Sampling

Orthogonal progressive sampling

Image width W and image height H

Sample aspect ratio (SAR)

SAR =∆x

∆y

Picture aspect ratio (PAR)

PAR =W ·∆xH ·∆y

=W

H· SAR

∆𝑥

∆𝑦

𝑊 ⋅ ∆𝑥

𝐻⋅∆𝑦

𝑥

𝑦

Special case: Interlaced sampling

Top field: Even scan lines

Bottom field: Odd scan lines

Top and bottom fields are alternativelyscanned at successive time instances

top field

bottom field

top field

bottom field

top field

bottom field


o


Common Picture Formats

picture size sample aspect picture aspect(in samples) ratio (SAR) ratio (PAR)

720× 576 12:11 4:3standard 720× 480 10:11 4:3definition 720× 576 16:11 16:9

720× 480 40:33 16:9

1280× 720 1:1 16:9high

1440× 1080 4:3 16:9definition

1920× 1080 1:1 16:9

ultra-high 3840× 2160 1:1 16:9definition 7680× 4320 1:1 16:9

SD formats: Only 704 samples are displayed per scan line (overscan)


o


Spatial Resolution — Illustration

400× 300 samples 200× 150 samples

100× 75 samples 50× 38 samples


o


Representation of Color Images

Color components

Require 3 color components (trichromatic vision)

Usage of linear RGB color spaces

Requires conversion from camera-internal color space toRGB color space of the representation format[

RGB

]rep. format

= T 3×3 ·

[RGB

]camera

Conversion matrix T typically includes white balancing

Compression typically done in Y’CbCr color space (discussed later)


o


RGB Color Space — Color Gamut

Color Gamut

RGB color space is specified by chromaticity coordinates ofthe three primaries (red, green, blue) and the white point

The chosen linear RGB color space determines the representable color gamut

ITU-R ITU-RBT.709 BT.2020

xr 0.6400 0.7080red

yr 0.3300 0.2920

xg 0.3000 0.1700green

yg 0.6000 0.7970

xb 0.1500 0.1310blue

yb 0.0600 0.0460

white xw 0.3127 0.3127(D65) yw 0.3290 0.3290

D65white

BT.709 (HD)sRGB

BT.2020 (UHD)

human gamut

x

y


o


Non-Linear Encoding — Gamma Encoding

Human vision: Non-linear response to differences in luminance

Remember: Weber-Fechner law

=⇒ Certain amount of quantization noise is more visible in dark image regions

=⇒ Reduce effect by quantizing/coding non-linear components

E′ = fTC(E)

At encoder side: Approximation by power law

Y ′ = fTC(Y ) = Y γe with encoding gamma γe ≈ 1/2.2 ≈ 0.45

with Y being the relative luminance in range [0;1]

At receiver side: Invert the gamma encoding

Y = f−1TC(Y ′) = (Y ′)γd with decoding gamma γd ≈ 1/γe ≈ 2.2


o


Transfer Characteristics

linear increasing Y

linear increasing Y ′ = fTC(Y )

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

non-

linea

r en

code

d si

gnal

E'

linear component signal E

BT.709BT.2020

γe = 1/2.2CIE L*a*b*

linear encoding

Representation formats

Piecewise-defined transfer function (linear function for very small values)

E′ = fTC(E) =

{κ · E : 0 ≤ E < ba · Eγ − (a− 1) : b ≤ E ≤ 1

BT.709 / BT.2020: γ = 0.45, κ = 4.5, a ≈ 1.0993, b ≈ 0.0181

Similar to mapping function in CIELAB color space


o


YCC Color Formats

Reasons for using YCC color formats

Color television: Add color difference information to black & white television

Decorrelation of color components (see opponent processes, CIELAB)

Luminance-related signal L

Two color difference signals C1, C2 (e.g., yellow-blue & red-green)

Consider mapping LC1C2 7→ RGB 7→ XYZ[XYZ

]=

[Xr Xg Xb

Yr Yg Yb

Zr Zg Zb

]·

[R` Rc1 Rc2

G` Gc1 Gc2

B` Bc1 Bc2

]·

[LC1

C2

]

Desirable properties

Achromatic signals (x = xw and y = yw) have C1 = C2 = 0

Change in C1 and C2 do not have any impact on relative luminance Y


o


YCC Color Formats

Fulfilling desirable properties

First property: R` = G` = B`

Second property: Yr ·Rc1 + Yg ·Gc1 + Yb ·Bc1 = 0

Yr ·Rc2 + Yg ·Gc2 + Yb ·Bc2 = 0

Early researchers additionally chose Rc1 = 0 and Bc2 = 0

=⇒ These choices yield

L = s` · YC1 = sc1 · ( (Yr + Yg + Yb)EB − Y )C2 = sc2 · ( (Yr + Yg + Yb)ER − Y )

with s`, sc1, sc2 being arbitrary non-zero scaling factors

=⇒ Interpretation of components

L – Scaled version of relative luminance

C1, C2 – Difference between a primary and scaled relative luminance Y


o


The Y’CbCr Color Format

Decision in early years of television: YCC transform after gamma encodingTransformation is given by

E′Y = KR · E′R + (1−KR −KB) · E′B +KB · E′BE′Cb = (E′B − E′Y ) / (2− 2KB)E′Cr = (E′R − E′Y ) / (2− 2KR).

with

KR =Yr

Yr + Yg + Yband KB =

YbYr + Yg + Yb

BT.709: KR = 0.2126, KB = 0.0722

BT.2020: KR = 0.2627, KB = 0.0593

color image luma comp. Y ′ chroma comp. Cb chroma comp. Cr


o


Advantage of Y’CbCr Format

R′ G′ B′

Y ′ Cb Cr

Transform into Y’CbCr domain

Decorrelates RGB data (or cone responses) for typical natural images

Color components can be independently quantized / coded

Quantization noise is introduced in perceptually meaningful way


o


Chroma Subsampling

Property of human vision

Contrast sensitivity: Human beings are more sensitive to high-frequencycomponents in isochromatic than isoluminant stimuli

=⇒ Chroma components are often downsampled for saving bit rate

4:4:4 – No downsampling4:2:2 – Factor of two in horizontal direction4:2:0 – Factor of two in both horizontal and vertical direction

Chroma sample locations are specified in representation format or video bitstream

4:4:4 4:2:0 (BT.2020)4:2:2 4:2:0 (MPEG-1) 4:2:0 (MPEG-2)


o


Chroma Sampling Formats for Image and Video Coding

Y‘CbCr 4:4:4 Y‘CbCr 4:2:2 Y‘CbCr 4:2:0 R‘G‘B‘

most common format


o


Demonstration of Luma/Chroma Perception

Compare luma and chroma perception

Selective low-pass filtering for lumaor chroma components

Use low-pass filter (1,4,6,4,1)/16

Order of presentation

1 Original luma and chroma components

2 Low-pass filtered luma component but original chroma components

3 Original luma and chroma components (repeated)

4 Original luma component but low-pass filtered chroma components


o


Demonstration: Original Picture


o


Demonstration: Filtered Luma Component


o


Demonstration: Original Picture (Repeated)


o


Demonstration: Filtered Chroma Components


o


The Constant Luminance Y’CbCr Format

Conventional Y’CbCr format

Application of gamma encoding before conversion into Y’CbCr

=⇒ Changes in chroma components (due to coding) impactthe relative luminance Y of the displayed signal

Alternative in BT.2020: Constant luminance Y’CbCr format

Very similar properties as conventional Y’CbCr format

Advantage: Relative luminance Y only depends on Y ′ component

Transform is given by

E′Y C = fTC (KR · ER + (1−KR −KB) · EG +KB · EB )

E′CbC = (E′B − E′Y C) /NB

E′CrC = (E′R − E′Y C) /NR

with NB and NR given by (a and γ: parameters of transfer fucntion)

NX =

{2a (1−Kγ

X) : E′X − E′Y C ≤ 02a (1− (1−KX)γ)− 1 : E′X − E′Y C > 0


o


Quantization of Sample Values

Samples are represented as discrete-amplitude values

ITU-R Recommendations BT.601, BT.709, and BT.2020 specify

Y =[

(219 · E′Y + 16) · 2B−8]

Cb =[

(224 · E′Cb + 128) · 2B−8]

Cr =[

(224 · E′Cr + 128) · 2B−8]

where B specifies the bit depth (in bits per sample)

Typical bit depths: 8, 10, or 12 bits per sample

Footroom / headroom

Unused values of the range of B-bit integer values [0; 2B − 1]

Allow implementation of signal processing operations without clipping

Example for other usage: xvYCC color space

=⇒ Footroom/headroom is used for extending color gamut


o


Bit Depth — Illustration

8 bits per component 4 bits per component

3 bits per component 2 bits per component


o


Summary


Each color component of a picture: W ×H array of samplesSample aspect ratio, picture aspect ratio, frame rate

Linear RGB color space

Chromaticity coordinates of primaries and white pointSpecifies color gamut: Range of representable colors

Non-linear encoding

Gamma encoding approximates human brightness perceptionQuantization noise is introduced in a perceptually meaningful way

The Y’CbCr color format

Decorrelation of RGB data (and cone responses)Allows subsampling of chroma components

Quantization / bit depth

Represent samples as discrete-amplitude valuesUniform quantization specified by bit depth


o

Acquisition, Representation, Display, and Perception of Image and Video Signals Image Acquisition

Image Acquisition


o


Basic Principle of Image Acquisition

objects in

3-d world

camera

lens image sensor aperture

image processor

digital

picture

Lens

Projects the real-world sceneonto the image plane

Field of view & depth of field

Image sensor

Converts analog irradiancepattern into image samples

Image processor

Analog-to-digital conversion

Demosaicing

Gamma encoding / tone mapping

White balancing

Color space conversion

Denoising, sharpening, etc.

CompressionThomas Wiegand Digital Image Communication 29 / 47

o


Image Sensor

photocell

microlens

image sensor

color

filter

light filter

light

volta

ge

exposure

saturation voltage

satu

ratio

nex

posu

re le

vel

Image sensor in digital cameras

Array of light-sensitive photocells (photocell = sample)

Photocells employ photoelectric effect

Irradiance is converted into electric signal

Filter: Remove unwanted wavelengths

Two types of sensors: CCD & CMOS

Exposure-voltage function approximately linear (below saturation level)


o


Sensor Noise & ISO Speed

Predominant noise: Photon shot noise

Number of photon that arriveduring exposure time is random

Poisson distribution (σ2 = µ)

=⇒ SNR increases with exposure

=⇒ SNR increases with sensor size

Other noise sources

Dark current noise (charges created by thermal vibration)

Read noise (thermal noise in readout circuitry)

Reset noise (some charges remain after resetting photocells)

Fixed pattern noise (manufacturing variations)

ISO speed in digital cameras

Amplification factor before analog-to-digital conversion

Can be modified for selecting trade-off between noise and exposure timeThomas Wiegand Digital Image Communication 31 / 47

o


Capturing of Color Images

Basic approach

Filter incoming light using three (or more) different color filters

Capture filtered color components

Convert into signals for color primaries of representation format

red component

green component

blue component

capture green-

filtered image

capture blue-

filtered image

capture red-

filtered image

real image

color filters


o


Three-Sensor Systems

image sensor

(red component)

image sensor

(blue component)

image sensor

(green component)

filter coatingfilter coating

light falling

through lens

Cameras with three image sensors

Light is split into three color components using a trichroic prism assembly(coatings for which reflection/transmission depends on wavelength)

Three image sensors: One for each color component

Main advantage: High light sensitivity (all photons are used)

Disadvantage: Expensive, large, heavy


o


Sensors with Color Filter Arrays

color filter

photocell Bayer pattern

Single sensor cameras

Separate color filter on top of each photocell

Photocells have different spectral responses

Requires demosaicing (interpolation of unknown sample values)

Lower resolution / demosaicing artifacts

Bayer pattern

Most common type of color filter array

Twice as many green than red/blue samples(humans more sensitive to middle wavelengths)


o


Bayer Image Demosaicing — Illustration

demosaicing / interpolation

raw image data (as recorded by sensor)

generated color image (3 components)

gamma encoding

color balancing

Note: The processing order can differ


o


Demosaicing Artifacts

Only one color component is captured per pixel

67% of the samples have to be interpolated

Interpolation can cause visible artifacts: Moire patterns

Interpolation artifacts can be reduced by optical low-pass filter

Optical low-pass filter also reduces sharpness


o


The Image Processor

Converts obtained sensor signal into representation format

Demosaicing (for sensors with color filter arrays)

White balancing

Conversion into RGB space of representation format

Gamma encoding of linear color components

Transform into Y’CbCr format (if desired)

Final quantization of sample values

Additional processing steps

Algorithms for improving image quality

DenoisingSharpeningReduction of artifacts caused by lens aberrations

Compression (e.g. JPEG or H.264 — MPEG-4 AVC)


o


Summary

Structure of cameras

Lens, image sensor and image processor

Image sensor

Matrix of light-sensitive photocellsLinear transfer characteristic (photons to electrons)

Sensor noise

Predominant noise: Photon shot noise (Poisson distribution)Signal-to-noise ratio increases with exposure and sensor size

Capturing of color images

Three-chip sensorsSensors with color filter arrays (demosaicing)

Image processor

Conversion of captured data into representation formatDenoising, sharpening, lens correctionData compression


o

Acquisition, Representation, Display, and Perception of Image and Video Signals Display of Images and Video

Display of Images and Video


o


Cathode Ray Tube (CRT) Displays

electron

beams

shadow

mask

screen with

phosphors

electron

guns deflection system

(magnetic coils)

screen

Electron guns produce electron beams

Electrons that hit the phosphor-coated screen cause the emission of photons

Direction of electron beams is controlled by magnetic coils

Electron beam is linewise swept over the screen (50/60 times per second)

Color CRTs: Three electron guns and three types of phosphors


o


Liquid Crystal Display (LCD)

backlight

V

polarizer liquid

crystals color

filters polarizer

Liquid crystals are placed between glass plates with transparent electrodes

Backlight is linearly polarized

Polarization direction is modified by liquid crystals=⇒ Controlled by voltage (image signal) between electrodes

Second polarizer adjusts light intensities depending on polarization direction

Color filters for obtaining red, green and blue sub-pixels


o


Plasma Display

cell with red

phosphor

cell with green

phosphor

cell with blue

phosphor

+ - + - + - + - + -

+ - + -

Display consists of cells which are painted with a colored phosphor

Each cell corresponds to a sub-pixel (3 cells form a pixel)

The cells contain a nobel gas and a small amount of mercury

When voltage is applied, the nobel gas is ionized, forms a plasmaand UV photons are emitted

The UV photons hit the phosphor at the inside of the cell, which cause thephosphors to emit visible light of the corresponding color

The light intensity is controlled by the applied voltage


o


Organic Light-Emitting Diode (OLED) Displays

+ + + +

+ + -

- - -

- - - +

emission of

red light

emission of

green light

emission of

blue light

Organic light-emitting diode emits light and does not use a backlight

Composed of a layer of organic materials situated between two electrodes

Many OLEDs consist of a conductive (electrons) and an emissive (holes) layer

When voltage is applied, electrons and holes are recombined and form anexcited bound state called exciton

Photons are emitted when electron-hole pairs fall back to base state

Wavelength of emitted photons depends on band gap of material


o


Projectors

LCD projectors

Similar principle as slide projector, but slide is replaced by LCD

White light is split into red, green and blue component (mirrors or prisms)

An image for each color component (red, green, blue)is generated by passing the light through an LCD

The red, green and blue images are combined using dichronic prisms

Digital light processing (DLP) projectors

Digital micromirror device (DMD):One microscopic mirror for each pixel on a chip

Micromirrors can be rotated to send light through the lens or to a heat sink

Gray values are obtained by quickly toggling the mirrors

Color images are generated by sending the light through a rotating colorwheel or using 3 DMDs (one for each primary color)


o


Resolution, Optimal Viewing Distance and Angle of View

Visual acuity of human vision

Can resolve lines at about 1 minute of arc

Optimal viewing distance

Display of size W ×H with resolution NW ×NHViewing distance vopt (can just resolve two points)

vopt ≈(W/NW )

sin((1/60)◦)≈ 3400W

NW≈ 3400H

NH

Angle of view for optimal viewing distance

Horizontal angle of view θW for optimal viewingdistance vopt is given by

θW = 2 · arctan

(W

2 vopt

)≈ 2 · arctan

(NW6800

)Same consideration for vertical angle of view θH


o


Display Size for Optimal Viewing & Example Calculations

Display size for optimal viewing

Given viewing distance v andpicture aspect ratio a = W/H

Display diagonal Dopt

Dopt =√

(a ·H)2 +H2

≈ NH ·√a2 + 1

3400· v

picture format NW ×NH opt. viewing corr. angle of corr. display size Dopt

(picture aspect ratio) distance vopt view θW × θH for v = 2 m / v = 3 m

SDTV: 720× 576 (4:3) 5.9 ·H 12◦× 10◦ 28” / 42”

HDTV: 1920× 1080 (16:9) 3.1 ·H 32◦× 18◦ 50” / 75”

UHD-1: 3840× 2160 (16:9) 1.6 ·H 59◦× 35◦ 100” / 150”

UHD-2: 7680× 4320 (16:9) 0.8 ·H 97◦× 65◦ 200” / 300”


o


Summary

Display technologies

Cathode ray tube (CRT) displays

Liquid crystal displays (LCDs)

Plasma displays

Organic light-emitting diode (OLED) displays

Projection technologies

LCD projectors

DLP projectors

Display size and resolution, optimal viewing distance, angle of view

Human vision has a maximum acuity

Optimal viewing distance depends on display size and resolution

Angle of view for optimal viewing conditions

Display size for given viewing distance and optimal viewing


acquisition, representation, display, and perception of ......acquisition, representation, display,...

Documents