bitm1113- multimedia systems
DESCRIPTION
Outlines Difference with classic applications Classes of multimedia applications Requirements/Constraints Problems with today’s Internet and solutions Common multimedia protocols RTP, RTCP Accessing multimedia data through a web server ConclusionTRANSCRIPT
![Page 1: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/1.jpg)
MultimediaBITM1113- Multimedia Systems
![Page 2: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/2.jpg)
Outlines Difference with classic applications Classes of multimedia applications
Requirements/Constraints Problems with today’s Internet and solutions Common multimedia protocols
RTP, RTCP Accessing multimedia data through a web
server Conclusion
![Page 3: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/3.jpg)
Difference with classic applications Highly delay-sensitive
Packets are useless if they arrive too late
Loss-tolerant (for the most part) Packet loss can be concealed
![Page 4: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/4.jpg)
Classes of multimedia Applications Streaming Stored Audio and Video Streaming Live Audio and Video Real-Time Interactive Audio and
Video Others
![Page 5: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/5.jpg)
Class: Streaming Stored Audio and Video
The multimedia content has been prerecorded and stored on a server
User may pause, rewind, forward, etc…
The time between the initial request and display start can be 1 to 10 seconds
Constraint: after display start, the playout must be continuous
![Page 6: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/6.jpg)
Class: Streaming Live Audio and Video Similar to traditional broadcast TV/radio, but
delivery on the Internet Non-interactive just view/listen
Can not pause or rewind Often combined with multicast The time between the initial request and
display start can be up to 10 seconds Constraint: like stored streaming, after
display start, the playout must be continuous
![Page 7: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/7.jpg)
Class: Real-Time Interactive Audio and Video Phone conversation/Video
conferencing Constraint: delay between initial
request and display start must be small Video: <150 ms acceptable Audio: <150 ms not perceived, <400
ms acceptable Constraint: after display start, the
playout must be continuous
![Page 8: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/8.jpg)
Class: Others Multimedia sharing applications
Download-and-then-play applications E.g. Napster, Gnutella, Freenet
Distance learning applications Coordinate video, audio and data Typically distributed on CDs
![Page 9: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/9.jpg)
Challenge TCP/UDP/IP suite provides best-effort, no
guarantees on expectation or variance of packet delay
Performance deteriorate if links are congested (transoceanic)
Most router implementations use only First-Come-First-Serve (FCFS) packet processing and transmission scheduling
![Page 10: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/10.jpg)
Problems and solutions Limited bandwidth
Solution: Compression Packet Jitter
Solution: Fixed/adaptive playout delay for Audio (example: phone over IP)
Packet loss Solution: FEC, Interleaving
![Page 11: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/11.jpg)
Problem: Limited bandwidth Intro: Digitalization Audio
x samples every second (x=frequency) The value of each sample is rounded to
a finite number of values (for example 256). This is called quantization
Video Each pixel has a color Each color has a value
![Page 12: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/12.jpg)
Problem: Limited bandwidthNeed for compression Audio
CD quality: 44100 samples per seconds with 16 bits per sample, stereo sound
44100*16*2 = 1.411 Mbps For a 3-minute song: 1.441 * 180 = 254
Mb = 31.75 MB Video
For 320*240 images with 24-bit colors 320*240*24 = 230KB/image 15 frames/sec: 15*230KB = 3.456MB 3 minutes of video: 3.456*180 = 622MB
![Page 13: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/13.jpg)
Audio compression Several techniques
GSM (13 kbps), G.729(8 kbps), G723.3(6.4 and 5.3kbps)
MPEG 1 layer 3 (also known as MP3)• Typical compress rates 96kbps, 128kbps, 160kbps• Very little sound degradation• If file is broken up, each piece is still playable• Complex (psychoacoustic masking, redundancy
reduction, and bit reservoir buffering) • 3-minute song (128kbps) : 2.8MB
![Page 14: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/14.jpg)
Image compression: JPEG Divide digitized image in 8x8 pixel blocks Pixel blocks are transformed into
frequency blocks using DCT (Discrete Cosine Transform). This is similar to FFT (Fast Fourier Transform)
The quantization phase limits the precision of the frequency coefficient.
The encoding phase packs this information in a dense fashion
![Page 15: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/15.jpg)
JPEG Compression
![Page 16: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/16.jpg)
Video compression Popular techniques
MPEG 1 for CD-ROM quality video (1.5Mbps)
MPEG 2 for high quality DVD video (3-6 Mbps)
MPEG 4 for object-oriented video compression
![Page 17: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/17.jpg)
Video Compression: MPEG MPEG uses inter-frame encoding
Exploits the similarity between consecutive frames Three frame types
I frame: independent encoding of the frame (JPEG) P frame: encodes difference relative to I-frame (predicted) B frame: encodes difference relative to interpolated frame Note that frames will have different sizes
Complex encoding, e.g. motion of pixel blocks, scene changes, … Decoding is easier then encoding
MPEG often uses fixed-rate encoding
I PB B BB B BP PI B B B B
![Page 18: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/18.jpg)
MPEG Compression (cont.)
![Page 19: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/19.jpg)
MPEG System Streams Combine MPEG video and audio streams in a
single synchronized stream Consists of a hierarchy with meta data at
every level describing the data System level contains synchronization
information Video level is organized as a stream of group of
pictures Group of pictures consists of pictures Pictures are organized in slices
![Page 20: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/20.jpg)
MPEG System Streams (cont.)
![Page 21: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/21.jpg)
MPEG System Streams (cont.)
![Page 22: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/22.jpg)
Problem: Packet Jitter Jitter: Variation in delay
Example
1
3
5 4 3 2SenderNo jitter
125 46
6
ReceiverJitter
pkt 6
pkt 5
![Page 23: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/23.jpg)
Dealing with packet jitter How does Phone over IP applications
limit the effect of jitter? A sequence number is added to each
packet A timestamp is added to each packet Playout is delayed
![Page 24: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/24.jpg)
Dealing with packet jitterFixed playout delay Fixed playout delay
![Page 25: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/25.jpg)
Dealing with packet jitterAdaptive playout delay Objective is to use a value for p-r that tracks the network delay
performance as it varies during a transfer. The following formulas are used:
di = (1-u)di-1 + u(ri – ti) u=0.01 for example i = (1-u)i-1 + u|ri-ti-di| Where ti is the timestamp of the ith packet (the time pkt i is sent) ri is the time packet i is received pi is the time packet i is played di is an estimate of the average network delay i is an estimate of the average deviation of the delay from
the estimated average delay
![Page 26: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/26.jpg)
Problem: Packet loss Loss is in a broader sense: packet
never arrives or arrives later than its scheduled playout time
Since retransmission is inappropriate for Real Time applications, FEC or Interleaving are used to reduce loss impact.
![Page 27: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/27.jpg)
Recovering from packet lossForward Error Correction Send redundant encoded chunk
every n chunks (XOR original n chunks) If 1 packet in this group lost, can
reconstruct If >1 packets lost, cannot recover
Disadvantages The smaller the group size, the larger
the overhead Playout delay increased
![Page 28: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/28.jpg)
Recovering from packet loss Piggybacking Lo-fi stream With one redundant low quality
chunk per chunk, scheme can recover from single packet losses
![Page 29: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/29.jpg)
Recovering from packet loss Interleaving Divide 20 msec of audio data into smaller units of
5 msec each and interleave Upon loss, have a set of partially filled chunks
![Page 30: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/30.jpg)
Recovering from packet lossReceiver-based Repair The simplest form: Packet repetition
Replaces lost packets with copies of the packets that arrived immediately before the loss
A more computationally intensive form: Interpolation Uses Audio before and after the loss to
interpolate a suitable packet to cover the loss
![Page 31: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/31.jpg)
Movie Time
![Page 32: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/32.jpg)
Real Time Protocol (RTP) RTP logically extends UDP
Sits between UDP and application Implemented as an application library
What does it do? Framing Multiplexing Synchronization Feedback (RTCP)
![Page 33: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/33.jpg)
RTP packet format Payload Type: 7 bits, providing 128
possible different types of encoding; eg PCM, MPEG2 video, etc.
Sequence Number: 16 bits; used to detect packet loss
![Page 34: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/34.jpg)
RTP packet format (cont) Timestamp: 32 bytes; gives the sampling
instant of the first audio/video byte in the packet; used to remove jitter introduced by the network
Synchronization Source identifier (SSRC): 32 bits; an id for the source of a stream; assigned randomly by the source
![Page 35: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/35.jpg)
Timestamp vs. Sequence No Timestamps relates packets to real
time Timestamp value sampled from a media
specific clock Sequence number relates packets to
other packets
![Page 36: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/36.jpg)
Audio silence example Consider audio data type
What do you want to send during silence?• Not sending anything
Why might this cause problems?• Other side needs to distinguish between
loss and silence Receiver uses Timestamps and
sequence No. to figure out what happened
![Page 37: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/37.jpg)
RTP Control Protocol (RTCP) Used in conjunction with RTP. Used to exchange control
information between the sender and the receiver. Three reports are defined: Receiver reception, Sender, and
Source description Reports contain statistics such as the number of packets
sent, numberof packets lost, inter-arrival jitter
Typically, limit the RTCP bandwidth to 5%. Approximately one sender report for threereceiver reports
![Page 38: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/38.jpg)
Streaming Stored Multimedia Example Audio/Video file is segmented and
sent over either TCP or UDP, public segmentation protocol: Real-Time Protocol (RTP)
User interactive control is provided, e.g. the public protocol Real Time Streaming Protocol (RTSP)
![Page 39: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/39.jpg)
Streaming Stored Multimedia Example Helper Application: displays content,
which is typically requested via a Web browser; e.g. RealPlayer; typical functions: Decompression Jitter removal Error correction: use redundant packets
to be used for reconstruction of original stream
GUI for user control
![Page 40: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/40.jpg)
Streaming from Web Servers Audio: in files sent as HTTP objects Video (interleaved audio and images in one file, or two
separate files and client synchronizes the display) sent as HTTP object(s)
A simple architecture is to have the Browser request the object(s) and after their reception pass them to the player for display - No pipelining
![Page 41: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/41.jpg)
Streaming from a Web Server (cont) Alternative: set up connection between server
and player, then download Web browser requests and receives a Meta File
(a file describing the object) instead of receiving the file itself;
Browser launches the appropriate Player and passes it the Meta File;
Player sets up a TCP connection with a streaming server Server and downloads the file
![Page 42: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/42.jpg)
Using a Streaming Server
![Page 43: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/43.jpg)
Options when using a streaming server Use UDP, and Server sends at a rate (Compression and Transmission)
appropriate for client; to reduce jitter, Player buffers initially for 2-5 seconds, then starts display
Use TCP, and sender sends at maximum possible rate under TCP; retransmit when error is encountered; Player uses a much large buffer to smooth delivery rate of TCP
![Page 44: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/44.jpg)
Real Time Streaming Protocol (RTSP) For user to control display: rewind, fast forward, pause,
resume, etc… Out-of-band protocol (uses two connections, one for
control messages (Port 554) and one for media stream) RFC 2326 permits use of either TCP or UDP for the
control messages connection, sometimes called the RTSP Channel
As before, meta file is communicated to web browser which then launches the Player; Player sets up an RTSP connection for control messages in addition to the connection for the streaming media
![Page 45: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/45.jpg)
Meta File Example <title>Twister</title> <session> <group language=en lipsync> <switch> <track type=audio e="PCMU/8000/1" src = "rtsp://audio.example.com/twister/audio.en/lofi"> <track type=audio e="DVI4/16000/2" pt="90 DVI4/8000/1" src="rtsp://audio.example.com/twister/audio.en/hifi"> </switch> <track type="video/jpeg" src="rtsp://video.example.com/twister/video"> </group> </session>
![Page 46: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/46.jpg)
RTSP Operations
C: SETUP rtsp://audio.example.com/twister/audio RTSP/1.0 Transport: rtp/udp; compression; port=3056; mode=PLAY
S: RTSP/1.0 200 1 OK Session 4231C: PLAY rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=0-S: RTSP/1.0 200 1 OK Session 4231 C: PAUSE rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231 Range: npt=37S: RTSP/1.0 200 1 OK Session 4231 C: TEARDOWN rtsp://audio.example.com/twister/audio.en/lofi RTSP/1.0 Session: 4231S: RTSP/1.0 200 1 OK Session 4231
![Page 47: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/47.jpg)
Conclusion None of the proposed solutions give
a real guarantee to the user that multimedia data will arrive on time.
Couldn’t we reserve some bandwidth for our multimedia transfer?
![Page 48: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/48.jpg)
Copyright & Piracy IssuesBITM1113 – MultimediaSystems
![Page 49: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/49.jpg)
Laws of Copyright Designed to
protect intellectual property rights and provide potential monetary rewards for inventiveness and hard work.
![Page 50: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/50.jpg)
Copyright Adequacy Material is easy
to capture.
Once material is digitized, it is easy to make changes.
![Page 51: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/51.jpg)
Obtaining Media Content Acquire rights to copyrighted material.
Use non-copyrighted material or public domain.
Create material in-house.
Contract for original material.
![Page 52: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/52.jpg)
Acquiring Rights to Copyrighted Material A simple purchase authorizes
personal use, not the right to copy.
Obtaining video rights does not give the rights to use the music or characters.
Separate licensing may be needed for cd-roms and Internet delivery.
![Page 53: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/53.jpg)
Acquiring Rights Determine
which rights are needed.
Determine who has those rights and the ability to convey them.
![Page 54: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/54.jpg)
Materials in thePublic Domain Are materials, that have no
copyright, because?:
No copyright was ever issued.
Copyright has expired.
Copyright was not renewed.
![Page 55: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/55.jpg)
Public Domain Materials There are legal considerations, when
using public domain works, such as:
Derivative works
Trademarks
Right of Publicity
![Page 56: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/56.jpg)
Derivative Works Those based on
an original work, such as translations, abridgments, adaptations, or dramatizations.
![Page 57: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/57.jpg)
Trademarks Trademarks - names, symbols, or
other devices identifying a product.
Trademark protection covers the title of a publishable book.
In the case of fiction, it is often the name of its characters.
![Page 58: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/58.jpg)
Right of Publicity A legal basis for
requiring permission and/or payment for using a person’s name, image or persona.
![Page 59: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/59.jpg)
Privacy Issues to Avoid Multimedia
developers must not:
Reveal embarrassing facts.
Place a person in a false light.
![Page 60: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/60.jpg)
Censorship Censorship -
examination of material for appropriateness of content.
Developer is “socially responsible”.
![Page 61: BITM1113- Multimedia Systems](https://reader035.vdocument.in/reader035/viewer/2022062317/5a4d1af47f8b9ab059980afd/html5/thumbnails/61.jpg)
The End