multimedia systems
TRANSCRIPT
![Page 1: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/1.jpg)
Multimedia Systems 1CS502 Spring 2006
Multimedia Systems
CS-502 Operating Systems
![Page 2: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/2.jpg)
Multimedia Systems 2CS502 Spring 2006
Outline
• Requirements and challenges for audio and video in computer systems
• Systems for multimedia• Compression and bandwidth• Processor scheduling• File, disk, and network management
Tanenbaum, Chapter 7Silbershatz, Chapter 20
![Page 3: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/3.jpg)
Multimedia Systems 3CS502 Spring 2006
What do we mean by “multimedia”
• Audio and video within a computer system– CD’s & DVD’s– Computer hard drive
• Live broadcast & web casts– Webcams, Skype, …
• Video on demand– Pause, fast forward, reverse, etc.
• Interactive meetings– Presentations with 2-way audio– Teleconferencing
• Interactive gaming• …
![Page 4: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/4.jpg)
Multimedia Systems 4CS502 Spring 2006
Requirements
• “Smooth” audio and video– Deterioration in quality >> jerky playback– Note: human is more sensitive to jitter in audio
than to jitter in video!
• Audio/video on PC’s doing something else• Multiple concurrent streams
– Video & multimedia servers– TiVo, etc.
• Wide range of network bandwidths
![Page 5: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/5.jpg)
Multimedia Systems 5CS502 Spring 2006
System and OS Challenges
• Bandwidths and Compression
• Jitter
• Processor Scheduling
• Disk Scheduling
• Network Streaming
![Page 6: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/6.jpg)
Multimedia Systems 6CS502 Spring 2006
Some System Architectures
• Simple: – Data paths for audio/video that are separate
from computational data paths
• Modern– Fast system bus, CPU, devices
• Video server– Disk farm and multiple streams
![Page 7: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/7.jpg)
Multimedia Systems 7CS502 Spring 2006
audio stream
CD-ROM drive
System Organization (simple)
• Separate data path for audio stream• Main system bus and CPU were too busy/slow to
handle real-time audio
CPU
Memory
memory bus
Device Sound card
![Page 8: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/8.jpg)
Multimedia Systems 8CS502 Spring 2006
video stream viaISA & bridge to
graphics card
audio stream viaISA bridge to
sound card
System Organization (typical Pentium)
ISA bridge
IDE disk
Main Memory
CPULevel
2 cache
Bridge Moni-tor
Graphics card
USB
Key-board
Mouse
Ether-net
SCSI
ModemSoundcard
Printer
PCI bus
ISA bus
AGP Port
![Page 9: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/9.jpg)
Multimedia Systems 9CS502 Spring 2006
Video Server
• Multiple CPUs
• Disk farm– 1000s of disks
• Multiple high-bandwidth network links– Cable TV– Video on demand– Internet
![Page 10: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/10.jpg)
Multimedia Systems 10CS502 Spring 2006
Why Compression? – CD-quality audio
• 22,050 Hz 44,100 samples/sec• 16 bits per sample
• Two channels 176,000 bytes/sec 1.4 mbits/sec
– Okay for a modern PC
– Not okay for 56 kb/sec modem (speed) or iPod (space)!
• MP-3 0.14 mbits/sec (10:1)– Same audio quality!
– Compression ratio varies with type of music
![Page 11: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/11.jpg)
Multimedia Systems 11CS502 Spring 2006
Why Compression? – Video
• “Standard” TV frame = 640 480 pixels @ 25-30 frames/sec (fps) 9,216,000 pixels/sec = 27,648,000 bytes/sec
• HDTV = 1280 720 pixels @ 30 fps 82,944,000 bytes/sec
• Typical movie 133 minutes approx. 220 gigabytes!
• DVD holds ~ 4.7 gigabytes average of ~ 620 kilobytes/sec!
• “Standard” movie of 133 minutes requires serious compression just to fit onto DVD
![Page 12: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/12.jpg)
Multimedia Systems 12CS502 Spring 2006
Video Compression Requirements
• Compression ratio > 50:1• i.e., 220 gigabytes:4.7 gigabytes
• Visually indistinguishable from original• Even when paused
• Fast, cheap decoder• Slow encoder is okay
• VCR controls• Pause, fast forward, reverse
![Page 13: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/13.jpg)
Multimedia Systems 13CS502 Spring 2006
Video Compression Standards
• MPEG (Motion Picture Experts Group)– Based on JPEG (Joint Photographic Experts Group)– Multi-layer
• Layer 1 = system and timing information• Layer 2 = video stream• Layer 3 = audio and text streams
• Three standards– MPEG-1 – 352240 frames; < 1.5 mb/sec ( < VHS quality)
• Layer 3 = MP3 Audio standard
– MPEG-2 – standard TV & HDTV; 1.5-15 mb/sec• DVD encoding
– MPEG-4 – combined audio, video, graphics• 2D & 3D animations
![Page 14: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/14.jpg)
Multimedia Systems 14CS502 Spring 2006
JPEG compression (single frame)
1. Convert RGB into YIQ• Y = luminance (i.e., brightness) ~ black-white TV• I, Q = chrominance (similar to saturation and hue)
Reason: Human eye is more sensitive to luminance than to color (rods vs. cones)
2. Down-sample I, Q channels• i.e., average over 22 pixels to reduce resolution• lossy compression, but barely noticeable to eye
3. Partition each channel into 88 blocks• 4800 Y blocks, 1200 each I & Q blocks
![Page 15: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/15.jpg)
Multimedia Systems 15CS502 Spring 2006
JPEG (continued)
![Page 16: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/16.jpg)
Multimedia Systems 16CS502 Spring 2006
JPEG (continued)
![Page 17: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/17.jpg)
Multimedia Systems 17CS502 Spring 2006
JPEG (continued)
4. Calculate Discrete Cosine Transform (DCT) of each 88 block
• What is a Discrete Cosine Transform?
5. Divide 88 block of DCT values by 88 quantization table
• Effectively throwing away higher frequencies
6. Linearize 88 block, run-length encode, and apply a Huffman code to reduce to a small fraction of original size (in bytes)
![Page 18: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/18.jpg)
Multimedia Systems 18CS502 Spring 2006
JPEG (concluded)
7. Store or transmit 88 quantization table followed by list of compressed blocks
• Achieves 20:1 compression with good visual characteristics
• Higher compression ratios possible with visible degradation
• JPEG algorithm executed backwards to recover image
• Visually indistinguishable from original @ 20:1
• JPEG algorithm is symmetric• Same speed forwards and backwards
![Page 19: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/19.jpg)
Multimedia Systems 19CS502 Spring 2006
MPEG
• JPEG-like encoding of each frame• Takes advantage of temporal locality• I.e., each frame usually shares similarities
with previous frame encode and transmit only differences
• Sometimes an object moves relative to background find object in previous frame, calculate
difference, apply motion vector
![Page 20: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/20.jpg)
Multimedia Systems 20CS502 Spring 2006
Temporal Locality (example)
Consecutive Video Frames
![Page 21: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/21.jpg)
Multimedia Systems 21CS502 Spring 2006
MPEG organization
• Three types of frames– I-frame: Intracoded or Independent.
• Full JPEG-encoded frame• Occurs at intervals of a second or so• Also at start of every scene
– P-frame: Predictive frame• Difference from previous frame
– B-frame: Bidirectional frame• Like p-frame but difference from both previous and next frame
I B BB P B BB P B BB P B BB I B BB P B BB P
![Page 22: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/22.jpg)
Multimedia Systems 22CS502 Spring 2006
MPEG Characteristics
• Non-uniform data rate!
• Compression ratios of 50:1 – 80:1 are readily obtainable
• Asymmetric algorithm– Fast decode (like JPEG)– Encode requires image search and analysis to
get high quality differences
• Decoding chips on graphics cards available
![Page 23: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/23.jpg)
Multimedia Systems 23CS502 Spring 2006
MPEG Problem – Fast Forward/Reverse
• Cannot simply skip frames– Next desired frame might be B or P derived
from a skipped frame
• Options:– Separate fast forward and fast reverse files
• MPEG encoding of every nth frame
• See Tanenbaum §7.5.1 – video-on-demand server
– Display only I and P frames• If B frame is needed, derive from nearest I or P
![Page 24: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/24.jpg)
Multimedia Systems 24CS502 Spring 2006
“Movie” File Organization
• One MPEG-2 video stream
• Multiple audio streams• Multiple languages
• Multiple text streams• Subtitles in multiple languages
• All interleaved
![Page 25: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/25.jpg)
Multimedia Systems 25CS502 Spring 2006
Challenge
• How to get the contents of a movie file from disk or DVD drive to video screen and speakers.– Fixed frame rate (25 or 30 fps)
– Steady audio rate
– Bounded jitter
• Classical problem in real-time scheduling– Obscure niche become mainstream!
– See Silbershatz, Chapter 19
![Page 26: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/26.jpg)
Multimedia Systems 26CS502 Spring 2006
Processor Scheduling for Real-TimeRate Monotonic Scheduling (RMS)
• Assume m periodic processes– Process i requires Ci msec of processing time every Pi
msec.
– Equal processing every interval — like clockwork!
• Assume
• Let priority of process i be
• Let priority of non-real-time processes be 0
11
m
i i
i
P
C
iP
1
![Page 27: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/27.jpg)
Multimedia Systems 27CS502 Spring 2006
Rate Monotonic Scheduling (continued)
• Then using these priorities in scheduler guarantees the needed Quality of Service (QoS), provided that
)12(1
1
mm
i i
i mP
C
• Asymtotically approaches ln 2 as m • I.e., must maintain some slack in scheduling• Assumes fixed amount of processing per periodic
task– Not MPEG!
![Page 28: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/28.jpg)
Multimedia Systems 28CS502 Spring 2006
Processor Scheduling for Real-TimeEarliest Deadline First (EDF) Scheduling
• When each process i become ready, it announces deadline Di for its next task.
• Scheduler always assigns processor to process with earliest deadline.
• May pre-empt other real-time processes
![Page 29: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/29.jpg)
Multimedia Systems 29CS502 Spring 2006
Earliest Deadline First Scheduling (continued)
• No assumption of periodicity• No assumption of uniform processing times
• Theorem: If any scheduling policy can satisfy QoS requirement for a sequence of real time tasks, then EDF can also satisfy it.– Proof: If i scheduled before i+1, but Di+1 < Di,
then i and i+1 can be interchanged without affecting QoS guarantee to either one.
![Page 30: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/30.jpg)
Multimedia Systems 30CS502 Spring 2006
Earliest Deadline First Scheduling (continued)
• EDF is more complex scheduling algorithm• Priorities are dynamically calculated
• Processes must know deadlines for tasks
• EDF can make higher use of processor than RMS
• Up to 100%
• However, it is usually a good idea to build in some slack
![Page 31: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/31.jpg)
Multimedia Systems 31CS502 Spring 2006
Multimedia File & Disk Management
• Single movie or multimedia file on PC disk– Interleave audio, video, etc.
• So temporally equivalent blocks are near each other
– Attempt contiguous allocation• Avoid seeks within a frame
TextFrame
AudioFrame
![Page 32: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/32.jpg)
Multimedia Systems 32CS502 Spring 2006
File organization – Frame vs. Block
• Frame organization• Small disk blocks (4-16 Kbytes)• Frame index entries point to starting block for each frame• Frames vary in size (MPEG)• Advantage: very little storage fragmentation• Disadvantage: large frame table in RAM
• Block organization• Large disk block (256 Kbytes)• Block index entries point to first I-frame of a sequence• Multiple frames per block• Advantage: much smaller block table in RAM• Disadvantage: large storage fragmentation on disk
![Page 33: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/33.jpg)
Multimedia Systems 33CS502 Spring 2006
Frame vs. Block organization
smaller larger
![Page 34: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/34.jpg)
Multimedia Systems 34CS502 Spring 2006
File Placement on Server
• Random
• Striped
• “Organ pipe” allocation– Most popular video in center of disk– Next most popular on either side of it– Etc.– Least popular at edges of disk– Minimizes seek distance
![Page 35: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/35.jpg)
Multimedia Systems 35CS502 Spring 2006
Disk Scheduling (server)
• Scheduling disk activity is just as important as scheduling processor activity
• Advantage:– Predictability• Unlike disk activity of ordinary computing
• In server, there will be multiple disk requests in each frame interval
• One request per frame for each concurrent video stream
![Page 36: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/36.jpg)
Multimedia Systems 36CS502 Spring 2006
Disk Scheduling (continued)
• SCAN (Elevator) algorithm for each frame interval
• Sort by cylinder #
• Complete in time for start of next frame interval
• Variation – SCAN–EDF:• Sort requests by deadline
• Group similar deadlines together, apply SCAN to group
• Particularly useful for non-uniform block sizes and frame intervals
![Page 37: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/37.jpg)
Multimedia Systems 37CS502 Spring 2006
Network Streaming
• Traditional HTTP• Stateless
• Server responds to each request independently
• Real-Time Streaming Protocol (RTSP)• Client initiates a “push” request for stream
• Server provides media stream at frame rate
![Page 38: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/38.jpg)
Multimedia Systems 38CS502 Spring 2006
Push vs. Pull server
![Page 39: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/39.jpg)
Multimedia Systems 39CS502 Spring 2006
Bandwidth Negotiation
• Client (or application) provides feedback to server to adjust bandwidth
• E.g., – Windows Media Player– RealPlayer– Quicktime
![Page 40: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/40.jpg)
Multimedia Systems 40CS502 Spring 2006
Conclusion
• Multimedia computing is challenging• Possible with modern computers
• Compression is essential, especially for video
• Real-time computing techniques move into mainstream
• Processor and disk scheduling
• There is much more to this subject than fits into one class
![Page 41: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/41.jpg)
Multimedia Systems 41CS502 Spring 2006
Break
![Page 42: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/42.jpg)
Multimedia Systems 42CS502 Spring 2006
Digression on Transforms
• Fourier’s theorem:–– Every continuous periodic function can be
reduced to the sum of a series of sine waves– The Fourier transform is a representation of
that function in terms of the frequencies of those sine waves
– Original function can be recovered from its Fourier transform
• Fourier transforms occur frequently in nature!
![Page 43: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/43.jpg)
Multimedia Systems 43CS502 Spring 2006
Nyquist’s Theorem (1924)
• If a continuous function is sampled at a frequency 2f, then the function can be recovered from those samples provided that its maximum Fourier frequency is f.
![Page 44: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/44.jpg)
Multimedia Systems 44CS502 Spring 2006
Discrete Cosine Transform
• A form of the Fourier Transform
• When applied to an 88 block of samples (i.e. pixel values) yields an 88 block of spatial frequencies
• Original 88 block of samples can be recovered from its DCT.– Assuming infinite arithmetic precision
![Page 45: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/45.jpg)
Multimedia Systems 45CS502 Spring 2006
More on Nyquist• If arithmetic precision is not infinite, we get
quantization error during sampling• Recovered signal has quantization noise
• i.e., a lossy transform
![Page 46: Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062319/554f4628b4c90572088b5680/html5/thumbnails/46.jpg)
Multimedia Systems 46CS502 Spring 2006
Return to JPEG