image/video compression september 28, 1999 lawrence a. rowe university of california, berkeley url:...
Post on 21-Dec-2015
215 views
TRANSCRIPT
Image/Video CompressionSeptember 28, 1999
Lawrence A. RoweUniversity of California, Berkeley
URL: http://www.BMRC.Berkeley.EDU/~larry
Copyright @1999, L.A. Rowe
Multimedia Systems and Applications 2
Outline
• Background• Block Transform Coding• Other Coding Algorithms• Software/Hardware CODEC’s• Pragmatic Issues
Multimedia Systems and Applications 3
Video Data Size
1920x1080 1280x720 640x480 320x240 160x1201 sec 0.19 0.08 0.03 0.01 0.001 min 11.20 4.98 1.66 0.41 0.101 hour 671.85 298.60 99.53 24.88 6.22
1000 hours 671,846.40 298,598.40 99,532.80 24,883.20 6,220.80
size of uncompressed video in gigabytes
image size of video
1280x720 (1.77) 640x480 (1.33) 320x240 160x120
Multimedia Systems and Applications 4
Video Bit Rate Calculation
width pixels (160, 320, 640, 720, 1280, 1920, …)
height pixels (120, 240, 480, 485, 720, 1080, …)
depth bits (1, 4, 8, 15, 16, 24, …)
fps frames per second (5, 15, 20, 24, 30, …)
compression factor (1, 6, 24, …)
width * height * depth * fps
compression factor= bits/sec
Multimedia Systems and Applications 5
Effects of Compression
storage for 1 hour of compressed video in megabytes
1920x1080 1280x720 640x480 320x240 160x1201:1 671,846 298,598 99,533 24,883 6,2213:1 223,949 99,533 33,178 8,294 2,0746:1 111,974 49,766 16,589 4,147 1,037
25:1 26,874 11,944 3,981 995 249100:1 6,718 2,986 995 249 62
3 bytes/pixel, 30 frames/sec
Multimedia Systems and Applications 6
Be Careful...
mpeg 200:1, jpeg 24:1
analogsource
digitalrepresentation
compressedrepresentation
vs
Multimedia Systems and Applications 7
Another View
Data Rate Size/Hour128 Kbs 60 MB384 Kbs 170 MB1.5 Mbs 680 MB3.0 Mbs 1.4 GB6.0 Mbs 2.7 GB25 Mbs 11.0 GB
Multimedia Systems and Applications 8
Perceptual Coding
• Encode source signal using lossy compressionLossless algorithms typically reduce signal by 3:1Must use lossy algorithm to get adequate compression
• Hide errors where humans will not see or hear itStudy hearing and vision system to understand how we see/hearMasking refers to one signal overwhelming/hiding another (e.g., loud siren or bright flash)Audio perception is 20-20 kHz but most sounds in low frequencies (e.g., 2 kHz to 4 kHz)Visual perception influenced by edges and low frequencies
Multimedia Systems and Applications 9
What is…
• JPEG - Joint Photographic Experts GroupStill image compression, intraframe picture technologyMJPEG is sequence of images coded with JPEG
• MPEG - Moving Picture Experts GroupMany standards MPEG1, MPEG2, and MPEG4Very sophisticated technology involving intra- and interframe picture coding and many other optimizations => high quality and cost in time/computation
• H.261/H.263/H.263+ - Video ConferencingLow to medium bit rate, quality, and computational cost Used in H.320 and H.323 video conferencing standards
Multimedia Systems and Applications 10
Coding Overview
• DigitizeSubsample to reduce data
• Intraframe compressionRemove redundancy within frame (spatial compression)
• Interframe compressionRemove redundancy between frames (temporal compression)
• Symbol codingEfficient coding of sequence of symbols
Multimedia Systems and Applications 11
Digitizing
• Modify color space24 bit RGB => 15 or 16 bit RGB24 bit RGB => YUV (8 bit Y, 4 bit U, 4 bit V)24 bit RGB => 8 bit color map
• Encode only 1 field• Reduce frame rate
Film is 24 fps so why encode 30 frames of videoIs 20 fps good enough? 18? 15? 12? 8? 4? ...Variable frame rate?
Multimedia Systems and Applications 12
Block Transform EncodingDCT
Zig-zag Quantize
Run-length Code
Huffman Code
011010001011101...
Multimedia Systems and Applications 13
Block Encoding
139 144 149 153144 151 153 156150 155 160 163159 161 162 160
original image
DCT1260 -1 -12 -5-23 -17 -6 -3-11 -9 -2 2-7 -2 0 1
DC component
AC components
Quantize
79 0 -1 0-2 -1 0 0-1 -1 0 00 0 0 0
zigzag79 0 -2 -1 -1 -1 0 0 -1 0 0 0 0 0 0 0
run-lengthcode
0 791 -20 -10 -10 -12 -10 0
Huffmancode 10011011100011...
coded bitstream < 10 bits (0.55 bits/pixel)
Multimedia Systems and Applications 14
Result of Coding/Decoding
139 144 149 153144 151 153 156150 155 160 163159 161 162 160
144 146 149 152156 150 152 154155 156 157 158160 161 161 162
original block reconstructed block
-5 -2 0 1-4 1 1 2-5 -1 3 5-1 0 1 -2
errors
Multimedia Systems and Applications 15
Examples
Uncompressed(262 KB)
Compressed (50)(22 KB, 12:1)
Compressed (1)(6 KB, 43:1)
Multimedia Systems and Applications 16
Discrete Cosine Transform
F[u,v] = 4C(u)C(v)
n2n-1 n-1
j=0 k=0
f(j,k) cos(2j+1)u
2n
(2k+1)v
2ncos
where
C(w) =
1
21
for w=0
for w=1,2,…,n-1
• Inverse is very similar• DCT better at reducing redundancy than Discrete Fourier
Transform but computationally expensive
Multimedia Systems and Applications 17
DCT vs DFT
original signal
recovered from DCT
recovered from DFT
Multimedia Systems and Applications 18
Inter-frame Compression
• Pixel difference with previous frameIf previous pixel very similar, skip it
• Send sequence of blocks rather than framesIf previous block similar, skip it or send difference
• Motion compensationSearch around block in previous frame for a better matching block and encode position and error differenceSearch in future frameAverage block in previous and future frame
Multimedia Systems and Applications 19
Background
• JPEG still image (1 bpp)Symmetric codec
• MPEGInterleaved audio and videoLow cost decoder at expense of high cost encoder (asymmetric)
• H.26xVideo conferencing standard (QCIF, CIF, 4CIF, and 16CIF)Variable bit rates and coding flexibility
Multimedia Systems and Applications 20
MPEG Standards
• MPEG1 - vhs quality (1992)CIF images, 4:2:0 sampling, 1.5 MbsFrame encoding
• MPEG2 - broadcast quality (1994)CCIR 601 images, 4:2:2 sampling, 15 MbsInterlaced and progressive scanningFrame and field encoding
Multimedia Systems and Applications 21
MPEG Technology
• Picture coding types• Motion compensation• Bit rate control• Picture order
Multimedia Systems and Applications 22
Do MPEG slides HERE
• Frame typesI, P, B, and Bi frames
• JargonGOP, macroblock, slice
• Motion vectorsPredicted and interpolated vectors
Multimedia Systems and Applications 23
Compression Formats
• Cinepack (2 bpp)Inter-frame, RGB15, software playback
• Motion JPEG (1-4 bpp)Intra-frame, DCT+Quantization, good for editing
• MPEG (0.5-2 bpp)Inter-frame, DCT+Quantization+Motion Compensation, excellent for playback
• H.261/H.263Inter-frame, DCT+Quantization+Motion Compensation, block stream, excellent for conferencing
Multimedia Systems and Applications 24
Quantitative Measures of Quality
• Mean Squared Error n-1 n-1
MSE = f(i,j) - f(i,j)
i=0 i=0
where f(i,j) is original image and f’(i,j) is image
after compression and decompression
• Perceptual MeasuresPerceptual Distortion Measure (Heeger)Picture Quality Scale (Miyahara, et.al.)Spatial/Temporal Measure (Webster, et.al.)
1
n2
Multimedia Systems and Applications 25
Discussion
• Where are the lossy steps?Quantization and subsampling before coding
• How do you choose quantization matrix?Standard proscribes matrix based on psychophysicsVary quantization by scaling Q matrix (i.e. MQUANT)Custom designed Q matrix
• Can improve compressioin by using motion compensation (MC)
• Comparison of standards…JPEG - still image uses fixed Q matrixH.261 - video conferencing uses MC and variable
quantizationMPEG - video playback uses MC and variable
quantization
Multimedia Systems and Applications 26
Discussion
• How to do very low bit rate (28.8 Kbs)?
Small image, low quality/bit rate, and better codingMPEG4 will address this problem
• Gazillions of other proposals…
Fine, but who needs themScalable proposals will be useful
• Must be prepared to deliver several formats
Multimedia Systems and Applications 27
H.261 Coding Example
1 2
543
6 7
1 2
54
3
6
1 2 3 4 5 6 7
1 2 3 4 5 6
Multimedia Systems and Applications 28
Other Techniques
• Vector QuantizationUse small codebook of valuesSlow encode, fast decode
• Fractal CodingFit curves to signalSlow encode, fast decode?
• Wavelet CodingBetter transform than DCT – incorporates both spatial and frequency in transformUsed in most research work and scalable codecs
Multimedia Systems and Applications 29
Scalable Algorithms• Some applications want to control bandwidth and
performanceSend low quality, low bit rate image on high priority channel and quality improvements on low priority channel(s)May be dictated by network bandwidth (wireless), end station (PDA), or application need (video gallery)
• Scalability parametersImage sizeFrame rateReconstructed image quality
Multimedia Systems and Applications 31
MPEG4 Standards Work
• Original goal was very low-bit rate codingCan do remarkably well at rates as low as 10 bits/sec but coding/decoding time is largeHierarchical VQ is popular because of low cost to decode, but trade-off is less coding efficiency
• Recent activity has moved towards programmable decoders
Idea is to allow code to be downloaded to the decoding chip
• Difficult trade-off between power consumption, quality and coding algorithm
Multimedia Systems and Applications 32
Understanding CODEC Performance
• Problem is motion vector search in MPEG• Low bit-rate coding (1.5 Mbs) requires
excellent coding
CODEC Compress Decompress
JPEG 314 inst/pixel 283 inst/pixel
MPEG1 1120 inst/pixel 60-100 inst/pixel
Multimedia Systems and Applications 33
MPEG Coding Performance• Decoding is easy
MPEG1 decoding in software on most platformsHardware decoders are widely available ($150/board)Windows graphics accelerators with MPEG decoding now entering market (e.g., Matrox, Diamond, …)
• Encoding is expensiveSequential software encoders are 20:1 real-timeReal-time encoders use parallel processingReal-time hardware encoders are expensive (e.g., $12K-$50K for MPEG1 and $100K-$500K for MPEG2)Hardware-assisted off-line MPEG1 encoders (3:1) used for multimedia authoring at reasonable cost ($5k)
Multimedia Systems and Applications 35
Other Boards
• MJPEGDEC J300 (Turbochannel, CL560) costs $3KParallax (Sun, HP, IBM R6K, CL560) costs $4K-$7.5KSGI Cosmo (CL560) cost (?)Radius VVS (Mac, CL560) costs $3K (?)Radius Truevision (PC, (?)) costs (?)PC boards (ASL, RealMagic MPEG Editor, IIT, …)
• Proprietary AlgorithmsIntel Smart Video Recorder Pro ($3K, I860)
• Programmable BoardsSun VideoPix (CL4K) supports MJPEG, MPEG, CELLB costs $1.5KChromatics MPACT chip board (mid ‘96)