using redundancy and interleaving to ameliorate the effects of packet loss in a video stream yali...
Post on 22-Dec-2015
214 views
TRANSCRIPT
![Page 1: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/1.jpg)
Using Redundancy and Interleaving to Ameliorate the
Effects of Packet Loss in a Video Stream
Yali Zhu, Mark Claypool and Yanlin Liu
Department of Computer Science
Worcester Polytechnic Institute
CS Technical Report TR-01-03
![Page 2: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/2.jpg)
Typical Network Traffic
• Majority is text-based– File transfer, Email, Web…
• Reliability is critical
• Latency is not critical
• Transmitted Using TCP – Provide reliable service where all bytes arrive
![Page 3: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/3.jpg)
Multimedia Network Traffic
• High bandwidth– Can induce congestion packet loss
• Latency is critical
• Loss is not critical– Can tolerate some
• Transmitted using UDP– Provide unreliable service where some packets
may be lost
![Page 4: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/4.jpg)
Multimedia Over Internet
• Often suffer from delay, loss and jitter– degrading multimedia quality
• Loss has the most severe effects on quality
• Use loss recovery techniques to– Improve multimedia quality
– Keep latency low
![Page 5: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/5.jpg)
Sender Based Receiver Based
Retransmission Forward Error Correction
Interleaving Interpolation Insertion Regeneration
Repetition
Multimedia Repair Taxonomy
Repair
![Page 6: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/6.jpg)
Media Specific FEC
• Multiple copies of data
• Lower quality of secondary frames
![Page 7: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/7.jpg)
Idea of Interleaving
• Without Interleaving
WorcesterPolytechnicInstituteWorcesterPolytechnicInstitute
sterPolytechnicInstitutesterPolytechnicInstitute
• With Interleaving
otlhnuWsocItreynstcrtiteePeciotlhnuWsocItreynstcrtiteePeci
WrceserPoytecnicIstituteWrceserPoytecnicIstitute
![Page 8: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/8.jpg)
Audio Interleaving
Encode
Interleave
Transmit
Decode
![Page 9: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/9.jpg)
Goal
• Above techniques have been done primarily to audio only
• Our goal:– Apply Interleaving and Redundancy to Video
– Evaluate effects on perceptual quality
– Evaluate system overhead
![Page 10: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/10.jpg)
Groupwork
• What are the issues with applying video redundancy to video vs. audio?
• What are the issues with applying interleaving to video vs. audio?
• What would be a methodology for evaluating the benefit?
• What performance metrics should you have?
![Page 11: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/11.jpg)
MPEG Encoding Techniques
• Intra-frame encoding– Based on current frame only
• Inter-frame encoding– Based on similarity among frames
• Frame types– I-frame (Intra-coded frame)
– P-frame (Predictive-coded frame)
– B-frame (Bi-directionally predictive frames)
![Page 12: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/12.jpg)
Coding Dependency within GOP
I B B P B B P B B I
•I frames (Intra-coded frames)•P frames (Predictive-coded frames)
Require information on previous I- or P- frames•B frames (Bi-directionally predictive-coded frames)
Require information on frames before and after
![Page 13: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/13.jpg)
Loss Propagation
• Loss of one single frame result in multiple losses– Loss of P-frame
– Loss of I-frame
– B-frame loss has no propagation
Loss Propagation of Second P-frame
I B B P B B P B B I
![Page 14: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/14.jpg)
Outline
• Introduction
• Background
• ApproachApproach• System Overhead
• Perceptual Quality
• Conclusions
![Page 15: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/15.jpg)
Approach
• Apply interleaving and redundancy to video– Hypothesize will improve perceptual quality
– Measure system overhead
• Build movies (next slide)– With loss, interleaved repair, redundancy repair
• Evaluate with user study
• Measure system overhead
• Analysis
![Page 16: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/16.jpg)
Repairing a Video Stream
.mpg file
mpeg decoder
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
1 3 5 7 9 11 13 15 17 2 4 6 8 10 12 14 16 18
Interleaver
mpeg encoder
I B B P B B P B B I B B P B B P B B
mpeg decoder
Transmitted over Network
1 3 5 7 9 11 13 15 17 2 4 6 8 10 12 14 16 18
Reconstructor
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Original Stream
Interleaved Stream
Encoded Stream
Decoded Stream
ReconstructedStream
(interleave, redundancy)
(apply loss)
(apply repair)
![Page 17: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/17.jpg)
Video Redundancy
1 2 3
1 1
1 2 3 4 Encode
3 4 Decode
1 Transmit43
Repetition in the case of consecutive loss- (if I frame, bad news)
Propagation in the case of I, P frame loss- of quality or previous frame
![Page 18: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/18.jpg)
Effect of Two Frames with Different Compression Rates
Two Frames with Different Compression Rates
![Page 19: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/19.jpg)
Interleaving Approaches
• Partial-Interleaving approach– Use sub-frame as basic interleaving unit
sub-frame consists of macro blocks
– (Next slide)
• Whole-Interleaving approach– Use whole frame as basic interleaving unit
• Focus on Whole-Interleaving– (Rest of work)
![Page 20: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/20.jpg)
Partial Video Interleaving
A1 A2
A3 A4
frame A
B1 B2
B3 B4
frame B
C1 C2
C3 C4
frame C
D1 D2
D3 D4
frame D
Original Stream
A1 B1
C1 D1
frame A
A2 B2
C2 D2
frame B
A3 B3
C3 D3
frame C
A4 B4
C4 D4
frame D
Interleaved Stream
![Page 21: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/21.jpg)
Repetition and Partial Video Interleaving
(4 repair pictures here)
![Page 22: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/22.jpg)
Whole Interleaving
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Original Stream
I-frame lossI
1 3 5 7 9 11 13 15 17ReconstructedStream
frame loss duringtransmission Propagation Loss
1 1 3 3 5 5 7 7 9 9 11 11 13 13 15 15 17 17 Apply Repetition
Effects of lost I-frame with whole-interleaving /distance=2
I B B P B B P B B I B B P B B P B B Encoded Stream
1 3 5 7 9 11 13 15 17 2 4 6 8 10 12 14 16 18 Interleaved Stream
![Page 23: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/23.jpg)
Effects of Whole-Interleaving
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Stream withone I frame loss
1 2 3 4 5 6 7 8 9 9 9 9 9 9 9 9 9 9 Apply Repetition
I-frame lossI
1 3 5 7 9 11 13 15 17 ReconstructedStream
Propagation Loss
1 1 3 3 5 5 7 7 9 9 11 11 13 13 15 15 17 17 Apply Repetition
Results ofInterleaved
Streams
Results ofNon-interleaved
Streams
![Page 24: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/24.jpg)
Whole Interleaving Distance
Original Stream (GOP Size = 9)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ...
Interleaving Distance = 2
1 3 5 7 9 11 13 15 17 2 4 6 8 10 12 14 16 18 ...
Interleaving Distance = 5
1 6 11 16 21 26 31 36 41 2 7 12 17 22 27 32 37 42 …
![Page 25: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/25.jpg)
A Possible Negative Effect of Whole-Interleaving
1 3 5 7 9 11 13 15 17 2 4 6 8 10 12 14 1816
1 2 5 6 7 8 9 10 13 14 15 16 1817
Interleaved Stream
Single Losses
Reconstructed Stream
A Special Case of Single Losses in the Interleaved Stream, distance=2
![Page 26: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/26.jpg)
Outline
• Introduction
• Background
• Approach
• System OverheadSystem Overhead• Perceptual Quality
• Conclusions
![Page 27: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/27.jpg)
MPEG Quality Vs. File Size
![Page 28: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/28.jpg)
Redundancy Overhead per Frame
![Page 29: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/29.jpg)
Redundancy Overhead per Movie Type
0
5
10
15
20
Animat
ion
Sports
Sitcom
News
Video Clips
Fra
me
Siz
e (
Kb
yte
s)
Primary
Redundancy
![Page 30: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/30.jpg)
Interleaving Overhead
• About 15% bandwidth overhead
• Reason– Intra-frame encoding based on similarity among
frames
– Interleaving Decrease similarities among consecutive frames Result in bigger B- and P- frames
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Original Stream
1 3 5 7 9 11 13 15 17 2 4 6 8 10 12 14 16 18 Interleaved Stream
![Page 31: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/31.jpg)
Proposed Solution to Bandwidth Overhead
• Encode using different MPEG qualities– Original stream with MPEG quality 1
– Stream with added repair with MPEG quality 2
– File size decreases exponentially
– Video quality slightly decreasedmay be undistinguishable by users
![Page 32: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/32.jpg)
Outline
• Introduction
• Background
• Approach
• System Overhead
• Perceptual QualityPerceptual Quality• Conclusions
![Page 33: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/33.jpg)
User Study for Redundancy
• We had over 40 users watch 22 video clips
• Video are from variety of television shows
• A video clip without loss is first displayed
• Video clips are of various loss rate and loss pattern.– Loss Rate: 1 10 20 20 20
– Loss Pattern: 1 1 1 2 4
![Page 34: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/34.jpg)
Perceptual Quality for Redundancy
![Page 35: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/35.jpg)
Consecutive Loss and Redundancy
![Page 36: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/36.jpg)
User Study for Interleaving
• Parameters to be tested– MPEG Quality 1 & 2
– Loss Rate: no loss, 2%, 5%, 10%, 20%
– Movie type: hockey game & CNN newsfrequency of scene changes andintensity of object actions
– Distance Value: 2 & 5
• Totally 24 movie clips– 20 seconds / clips
![Page 37: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/37.jpg)
User Study for Interleaving (II)
• Parameters not to be tested– Frame Rate: 30 frames/sec
– Size of movie: 320 x 240 pixels
– Hardware difference All tests on one machineOne user each time
– Human interaction: one same assistant (me)
![Page 38: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/38.jpg)
Perceptual Quality for Interleaving (hockey)
20
30
40
50
60
70
80
90
100
0 5% 10 % 15% 20%
Avera
ge S
core
s
Loss Rate in % Percentage (hockey)
Interleaved hockey ClipsNon-interleaved hockey Clips
![Page 39: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/39.jpg)
Perceptual Quality for Interleaving (CNN)
20
30
40
50
60
70
80
90
100
0 5% 10% 15% 20%
Avera
ge S
core
s
Loss Rate in % Pecentage (CNN)
Interleaved CNN clipsNon-interleaved CNN clips
![Page 40: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/40.jpg)
Perceptual Quality for Interleaving – Movie Type
0
2
4
6
8
10
12
14
16
2% 5% 10% 20%
Loss Rate in Percentage
Aver
age
Incr
ease
d Po
ints
Hockey Game Clips
CNN News Clips
![Page 41: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/41.jpg)
Perceptual Quality for Interleaving – Interleaving Distance
0
10
20
30
40
50
60
70
80
5% 20%
Loss Rate in Percentage
Aver
age
Scor
e Non-interleaved Clips
Distance=5
Distance=2
![Page 42: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/42.jpg)
Perceptual Quality vs. MPEG Quality
0102030405060708090
100
0% 10%
Loss Rate in Percentage (hockey)
Ave
rage
Sco
re
quality number 1
quality number 2
![Page 43: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/43.jpg)
Conclusions
• Video Repair and Interleaving – Improves perceptual quality by 25% – 70%– Completely repairs video in the presence of single
packet loss
• Bandwidth Overhead– About 10%– Can be decreased at cost of MPEG quality
• Movie Type– Not statistically significant for quality or overhead
• Effects of Increasing Distance Value– Observed to be non-beneficial
![Page 44: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/44.jpg)
Future Work
• Frame-Packet Ratio– In our implementation = 1
– >1 or <1?
• Other Compression Standards– MPEG-2, MPEG-4, Motion-JPEG
• Combine repair techniques– Interleaving + Redundancy
• Adaptive repair
• Effects of overhead on network congestion
![Page 45: Using Redundancy and Interleaving to Ameliorate the Effects of Packet Loss in a Video Stream Yali Zhu, Mark Claypool and Yanlin Liu Department of Computer](https://reader035.vdocument.in/reader035/viewer/2022062516/56649d775503460f94a58929/html5/thumbnails/45.jpg)
Evaluation of Science?
• Category of Paper
• Science Evaluation (1-10)?
• Space devoted to Experiments?