compressionfrom production format to distribution format

erik.luyten@dml.kuleuven.be

What's this ?

René Margritte: La Trahison des Images (1928-29)

Moving images !?

Temporal resolution

2 images/second 5 images/second

12,5 images/second 25 images/second

A very small image

Spatial resolution

7 x 10 = 7028 x 40 = 1120

14 x 20 = 280

112 x 160 = 1792056 x 80 = 4480

448 x 640 = 309120224 x 320 = 71680

graylevel resolution

8 bit/2567 bit/128

6 bit/ 645 bit/ 32

4 bit/ 16

3 bit/ 82 bit/ 41 bit/ 2

Creating colour: mixing light

3 x 8 bits

RGB triangle

256 red steps 256 green steps 256 blue steps

Red channel Green channel

Blue channel

Color resolution

24 bit color pallet244 KB file

8 bit color pallet28 KB file

4 bit color pallet16 KB file4 bit color palletNo “dithering”

8 bit optimized pallet28 KB file

Video signal(uncompressed)

768 pixels

576

pix

els

25 images/secondEach pixel has 3 values: (3 bytes)

• Red: 256 steps (1 byte)

• Green: 256 steps (1 byte)

• Blue: 256 steps (1 byte)

Datarate for the RGB video signal:768(pix) x 576(lines) x 3(byte) x 25(images)=32 MByte/sec(256 Mbit/sec)

Video signal (component)

720 pixels

576

pix

els

25 images/second

Each pixel has 3 values: (2 bytes)

• Luminace(Y): 256 steps (1 byte)

• Cr (R-Y): 128 steps (1/2 byte)

• Cb (B-Y): 128 steps (1/2 byte)

Datarate for the component video signal:720(pix) x 576(lines) x 2(byte) x 25(images)=20 MByte/sec(160 Mbit/sec)

Audio signalSampling Rate: 44,1 kHz (CD)Bit Depth: 16 bit (65.536 steps)Stereo: x 216 bit x 2 x 44.100 = 1.411.200 bps 1,4 Mbit/sec

Production systems normally use a sampling rate of 48 kHz

16 bit x 2 x 48.000 = 1.536.000 bps

Video distribution ?

Disk

floppy disk = 0,05 seconds

CD-rom = 20 seconds

1 Gbyte hard disk = 32 seconds

16 Gbyte array = 8 min 30 sec

Network

Ethernet 100 Mbit/sec (shared) LAN1/2.5

Ethernet 10 Mbit/sec (shared !) LAN1/25

ISDN lijn ( 2B + D) : 128Kbit/sec 1/2000

Modem connection : 56 Kbit/sec 1/4600

1 second video: 32 Mbyte (256 Mbit/sec) 1 minute video: 1,9 GByte 1 hour video: 111 GByte

Compression techniques

Intraframe (spatial compression)Compression within a single frame

Interframe (temporal compression)Compression between successive frames

LosslessNo information loss

Lossy(Important) information is lost (f.i. medical applications)

SymmetricCoding and decoding equally processor intensive

AsymmetricCoding is more processor intensive than decoding

Lower datastream ?43,2 Mbit/sec 10,4 Mbit/sec320x240, 25fps, 24 bit 160x120 pix

8 bit14,4 Mbit/sec

12,5fps21,6 Mbit/sec

Intraframe compression (in one frame)

Run Length Encoding• without compression:

one line of 100 (white) pixels 100 x 3 bytes = 300 bytes

• RLE compression 100 identical white pixels in a row 2 bytes: total amount of pixels involved 3 bytes for the colour information 2 + 3 = 5 bytes of info for 100 pixels

• Compression factor of 60:1 without information loss

100 pixels

Other Intraframe techniques

Discrete Cosine Transformation• JPEG, MPEG, DV,…• Type of Fourier Transformation • Blocks of 8 x 8 pixels and/or 16 x 16

Wavelet Compression• MPEG-4 (still image), JPEG2000• Improving image resolution as more data are received

Vector QuantizationFractal Compression

Interframe compression

Image 124

Image 125

Interframe: MPEG (Moving Picture Expert Group)

I B B P B B P B B P B B I

GOP Group Of PicturesI-pictures: (Intra pictures) reference image with complete information

for one frame (scene changes, ...)

P-pictures: (Predicted pictures) images are “predicted” using the information in earlier I or P pictures.

B-pictures: (Bi-directional predicted) pictures are recomposed from information in earlier or later I or P pictures. B-pictures are never used as reference for other images.

Group Of Pictures (GOP)

Movie Formats

File Format:Specifies how data are arranged in a file

(interleaving video and audio)

Codec:Specifies how (image) data is compressed

Architecture:Synchronisation of different media components

(video, audio, language tracks, text,…. tracks)Control for interactivity, alternate tracks,…

Movie Architectures

QuickTime

AVI

Windows Media Player

Real Media

Flash ?

MPEG-4

Media platformplatform distribution cost alternates remarks

QuickTimeMacWindows(Linux)

CD romDownload (HTTP)Streaming (RTSP)

Player = freePro = €30Server= open srce

RTSP alternatesHTTP alternatesMixed alternatesSkip prot.,instant -on

QTVR, text, midi, SMIL,..production and distribution

H.264

RealWindowsMacLinux

Streaming (RTSP)Download (HTTP)

Player = freeSuperpass=$13/mServer=payable

RTSP alternates(Surestream techn.)Dynamic

only distribution and mainly for streaming

WindowsMedia

Windows(Mac)(Linux)

Streaming ( )Download (HTTP)CD rom

Player=freeServer=Windows2003

RTSP alternatesDynamic

Production anddistributionVersion 9 on windows platform

Flash(Spark,

On2, h264)

WindowsMacLinux

CD romDownload (HTTP)Streaming

Player=freeServer=€4759

? H.264

MPEG-4WindowsMacLinux

Streaming (RTSP)Download (HTTP)CD romDVD, GSM, ….

Player=QT, Real,…Server=open srceLicense fee

scalable (in the future)

Industrie standaardfrom handheld to HD

H.264

File Formats

DV

MPEG-1

MPEG-2

MPEG-4

Codec’sPicture:

• Cinepak• Indeo (Intel Indeo)• H.261, H.263, H.264• Sorenson• M-JPEG• MPEG-1, MPEG-2, MPEG-4• DV

Sound:• IMA • Qualcomm PureVoice, Qdesign Music,… • MP3• AAC (Advanced Audio Coding)• WMA, RealAudio

MPEG-4 vs MPEG-1/2MPEG-4 differenceMPEG-1, MPEG-2

video

subtitles

background

graphics

Ren

derin

g an

dC

ompo

sitio

nC

ompr

essi

ontransport

Dec

ompr

essi

on

displayand

scaling

video

subtitles

background

graphics

transport

transport

transport

transport

com

pres

sion

Sca

ling,

Ren

derin

g an

dC

ompo

sitio

n

Dec

ompr

essi

on

display

Bitmap stream

ASCI code

Bitmap image

Vector code

MPEG-4

displayand

scaling

display

H.264

Codec’s and bitrate

MPEG-2MPEG-2

MPEG-4

56 kbit 256 Mbit1 Mbit 9 Mbit 25 Mbit200 kbit

MPEG-1

H-261

H-263

SorensonDV

uncompr.

Distribution codec’s Production codec’s

profile 0 profile 1

MPEG-4

HD distribution Studio productionPIM, GSM, ...

Web ?

Comparing codecsH-263 : 2 Mbit/secCinepack : 4,2 Mbit/sec MPEG 1: 1,8 Mbit/sec

Sorenson : 1,35 Mbit/sec MPEG-4 : 960 Kbit/sec H.264 : 502 Kbit/sec

Video to Computer• Video recording

• Interlaced, 25 frames - 50 fields per second• Restricted to 576 effective lines (PAL) and 3x4

picture aspect ratio.

• Computer image• Non-interlaced, 60 to 80 frames per second• No restrictions on picture format and aspect ratio

• De-interlacing of video material is desirable.

De-interlacing

1

2

3

311

312

313

313

314

315

623

624

625

Field 1Field 2

Delivery methods DVD, CD, USB stick,…

Internet • Progressive download (HTTP, FTP)

“file transport” type (TCP protocol)No data loss: resent lost packagesNo real-time eventsComplete file must be downloaded to allow random access. No special server needed (normal web service over http)

• Streaming media (RTP)RTSP transport (UDP protocol)Lost data (probably) will not be recoveredReal-time events possible“Random access” possibility (only needed fragments will be

downloaded)

Needs one or more(worldwide proxy delivery) streaming servers

(Progressive) downloadWeb server

http://

www.avnet.kuleuven.be/

staalkaart/2004/video04.html

<object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" codebase="http://www.apple.com/qtactivex/qtplugin.cab" width="320" height="195"><param name="loop" value="false"><param name="cache" value="true"><param name="src" value="video04.mov"><param name="autoplay" value="true"><param name="controller" value="true"><embed width="320" height="195" pluginspage="http://www.apple.com/quicktime/download/" src="video04.mov" type="video/quicktime" controller="true" autoplay="true" cache="true" loop="false"> </object>

http://

www.avnet.kuleuven.be/

staalkaart/2004/video04.mov

File : video04.mov

TCP/IP

(video) podcasting

Audio or audio/video distribution via internet

Download file to client computer

Mostly used for handheld devices

Broadcast channel ---> Feeds

Programs ---> Episodes

Podcast delivery: URL linkWeb server AVNet

Client computer

• Request XML data

• XML file + startup iTunes

• HTML with itpc link

• Request for episodes

• HTTP downloadepisodes to iTunes

• Request for web pqge

RTSP streaming (or RTMP)Web server Streaming server

http://

www.avnet.kuleuven.be/

Videotechniek/medatech.html

<object classid="clsid:02BF25D5-8C17-4B23-BC80-D3488ABDDC6B" width="384" height="304" codebase="http://www.apple.com/qtactivex/qtplugin.cab"><param name="autoplay" value="true"><param name="controller" value="true"><param name="pluginspage" value="http://www.apple.com/quicktime/download/indext.html"><param name="target" value="myself"><param name="type" value="video/quicktime"><param name="src" value="jaarverslag0304_384k.mov"><param name="HREF" value="rtsp://escher.avnet.kuleuven.be/mediatechnologie/mediatech_384k.mov"><embed src="comtechn.mov" width="384" height="304" autoplay="true" controller="true" border="0" pluginspage="http://www.apple.com/quicktime/download/indext.html" target="myself" type="video/quicktime" href="rtsp://escher.avnet.kuleuven.be/mediatechnologie/mediatech_384k.mov"></embed></object>

rtsp://escher.avnet.kuleuven.be/

mediatechnologie/mediatech_384k.mov

Datastream

RTP over UDP

Streaming - unicast

Streaming - multicast

HTTP streaming (iPhone, iPad,…) Ordinary webserver (Apache 2.2)

Segmented distribution of material. Material is chopped in 10 sec files Index file with info on different segments Distributed to client using http protocol

VOD via h.264 compressed material

Live streaming possible Latency 20-30 sec Stream segmenter

#EXTM3U#EXT-X-MEDIA-SEQUENCE:0#EXT-X-TARGETDURATION:10#EXTINF:10,http://media.example.com/segment1.ts#EXTINF:10,http://media.example.com/segment2.ts#EXTINF:10,http://media.example.com/segment3.ts#EXT-X-ENDLIST

• Apple HLS• Microsoft Smooth

Streaming• Adobe Flash HTTP

Dynamic Streaming