advances in network-adaptive video streaming bernd girod j. chakareski, m. kalman, y. j. liang, e....
TRANSCRIPT
Advances in Network-adaptive Video Streaming
Bernd GirodBernd GirodJ. Chakareski, M. Kalman, Y. J. Liang, E. Setton, R. ZhangJ. Chakareski, M. Kalman, Y. J. Liang, E. Setton, R. Zhang
Information Systems LaboratoryInformation Systems LaboratoryDepartment of Electrical EngineeringDepartment of Electrical Engineering
Stanford UniversityStanford University
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 2
Streaming media: a huge success
Hundreds of thousands of streaming media servers deployed
> 1 million hours of streaming media content produced per month
Hundreds or millions streaming media players RealPlayer
– Most popular Internet application second only to Internet Explorer [Media Metrix]
– More than 400 million unique registered users– More than 200,000 new users per day– Open source code
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 3
Best-effort packet network• low bit-rate • variable throughput• variable loss• variable delay
Challenges• compression • rate scalability• error resiliency• low latency
Challenges• compression • rate scalability• error resiliency• low latency
Internet Media Streaming
Streaming client
DSL
56K modem
Media Server
Internet
wireless
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 4
Outline
What is network-adaptive video streaming?– Better delivery of video packets by considering source
coding, signal processing, and packet transport jointly– Application-layer joint source-channel coding techniques
for the Internet
This talk: review recent advances in 1. Adaptive media playout
2. Rate-distortion optimal packet scheduling
3. Network-adaptive packet dependency management
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 5
Adaptive Media Playout
Fixed deadline Flexible deadline
Idea: reduce latency and packet loss simultaneously by continuously adapting playout deadline to network conditions
Idea: reduce latency and packet loss simultaneously by continuously adapting playout deadline to network conditions
5% packet loss2 sec average receiver buffer
5% packet loss2 sec average receiver buffer
[Steinbach, Färber, Girod, ICIP 2001]
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 6
Modification of Playout Speed
Video: adaptation of display rate Audio and speech: Stretching based on time-domain interpolation
algorithm WSOLA [Verhelst et al., 1993, Liang 2001]
Output packet
1/20/1 2/3 3 4
Original packet
Pitch-period
0 21 3 4
Template
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 7
Audio Speed Adjustment
Waveform Similarity Overlap Add (WSOLA) method allows speed adjustment without changing pitch
Speech demo
Music demo
originalslower 30% faster 30%
originalslower 30% faster 30%
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 8
Reduced Pre-roll Time for Stored Streams
G=1.092, B=0.42, TG=20 sec, TB=2 sec, TRTT=220 msG=1.092, B=0.42, TG=20 sec, TB=2 sec, TRTT=220 ms
Probabilityof buffer
underflow< 1%
Probabilityof buffer
underflow< 1%
[Kalman, Steinbach, Girod, ISCAS 2002]
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 9
Rate Scalability by Playout Speed Adjustment
Server
20 kbps
50 kbps
95 kbps
25 kbps
55 kbps
100 kbps
Channel(mean throughput)
85 kbps
(29.2 dB, 53.3 kbps)
(33.1 dB, 93.6*0.9=84.2 kbps)
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 10
First Things First: Smart Prefetching
Idea: Send more important packets earlier to allow for more retransmissions
Server ClientInternetInternet
Request stream
Request stream
Rate-distortionpreamble
Rate-distortionpreamblePrefetch
times
Prefetchtimes
Video dataVideo data
Repeatrequest
RepeatrequestRepeatrequest
RepeatrequestRepeatrequest
Repeatrequest
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 11
Streaming as a Packet Scheduling Problem
Which media units should be selected for transmission, and when?
Requirements– Meet rate constraint– Meet latency constraint– Maximize reconstruction quality
Rate-distortion framework proposed, e.g., in [Podolsky, McCanne, Vetterli 2000] [Miao, Ortega 2000] [Chou, Miao 2001]
time
pre-encoded media unitstransmissionopportunities
server
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 12
Markov Decision Tree for One Packet
... N transmission opportunities before deadline
send: 1
ack: 1
0
0
0
send: 1
0
send: 1
0
ack: 1
01
01
0
0
1
1
1
0
0
0
0
tcurrent tcurrent+t tcurrent+2t
Action Observation
“Policy“ minimizing
J = D + R“Policy“ minimizing
J = D + R
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 13
Packet Delay Jitter and Loss
delay
lead-time
lossprobability
lead-time
lossprobability
loss
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 14
Source Description
Each media packet n is labeled by− Bn — size [in bits] of data unit n
− dn —distortion reduction if n is decoded
− tn — decoding deadline for n
P PI
I
B B B P P PI
I
B B B P
A
…
…
…
A …
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 15
PB
Source Description
Each media packet n is labeled by− Bn — size [in bits] of data unit n
− dn —distortion reduction if n is decoded
− tn — decoding deadline for n
P PI
I
B B P PI
I
B B B P
A
…
…
…
A …
For video: dn must be made“state-dependent” to accurately capture concealment
For video: dn must be made“state-dependent” to accurately capture concealment
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 16
R-D Optimized Streaming Performance
40 60 80 100 120 14024
25
26
27
28
29
30
31R-D OptimizedPrioritized ARQ
Foreman 120 frames 10 fps, I-P-P-… H.263+ 2 Layer SNR
scalable 20 frame GOP Copy Concealment 20 % loss forward
and back Γ-distributed delay
– κ = 10 ms– μ = 50 ms– σ = 23 ms
Pre-roll 400ms
Foreman 120 frames 10 fps, I-P-P-… H.263+ 2 Layer SNR
scalable 20 frame GOP Copy Concealment 20 % loss forward
and back Γ-distributed delay
– κ = 10 ms– μ = 50 ms– σ = 23 ms
Pre-roll 400ms
PS
NR
[dB
]
Bit-Rate [kbps]
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 17
R-D Optimized Streaming with a Proxy Server
MediaServer
Switch/Router
ProxyServer
Client
Backbone Network
Packet
Last hopReceiver-driven RaDiO streaming
Sender-drivenRaDiO streaming
Buffer packets
[Chakareski, Chou, Girod, Asilomar 2002, MMSP 2002]
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 18
RaDiO Edge: Experiment
40 60 80 100 120 140 160 18027
28
29
30
31
32
33
34
Ave
rage
Y-P
SN
R (
dB)
Bit rate (Kbps)
RaDiO EdgeSender-driven
Video: Foreman, QCIF, 130 frames Compression: H.264 3-layer temporal scalability 72…144
kbps [Liang, 2002] Backbone
– Packet loss rate: 10%
– Delay: shifted -distribution Last hop
– Packet loss rate: 1%
– Delay: shifted -distribution
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 19
Streaming with Diversity
MediaServer
Client
Channel 1
Channel 2
Channel N
MediaServer 1
Client
Channel 1
Channel 2
Channel N
MediaServer 2
MediaServer N
Packet Path Diversity Server Diversity
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 20
R-D Optimized Streaming over 2 Channels
30 40 50 60 70 80 90 10024
25
26
27
28
29
30
Bit rate (Kbps)
Ave
rage
Y-P
SN
R (
dB)
Over 2 channelsOver 1 channel
Video: Foreman, QCIF, 130 frames
Compression: H.263+ 2-layer SNR scalability 32/64
kbps 2 identical, independent 2-state
Markov channels Good/bad packet loss rates:
3%/15% -distributed delays short/long
Video: Foreman, QCIF, 130 frames
Compression: H.263+ 2-layer SNR scalability 32/64
kbps 2 identical, independent 2-state
Markov channels Good/bad packet loss rates:
3%/15% -distributed delays short/long
[Chakareski, Girod, DCC 2003]
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 21
R-D Optimized Streaming with Accelerated Retroactive Decoding (ARD)
50 100 150 200 250 300
23
24
25
26
27
28
29
30
31multiple deadlinessingle deadline
Latency: 100 ms RTT: 100 ms
3 dB
47 %
PS
NR
, in
dB
Bit-Rate, in kbps
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 22
R-D Optimized Streaming with Accelerated Retroactive Decoding (ARD)
R-D Optimized Streaming with Accelerated Retroactive Decoding (ARD)
Latency: 100ms
Multiple DeadlinesRate: 68.8 kbps
Mean PSNR: 27.0 dB
Single DeadlineRate: 89.0 kbps
Mean PSNR: 23.9 dB
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 23
Network-adaptive Packet Dependency Management
Very low latency: no retransmissions Highly robust compressed representation by network-adaptive
management of packet dependencies Utilize ACK/NACK in source coder H.263 RPS, MPEG-4 NEWPRED, H.264 multiframe prediction
Time
Transmission error
NACK
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 24
Error Resilience vs. Coding Efficiency
P1
P2
I
P5
230 frames of Foreman coded using H.26L TML8.5. Average PSNR=33.4dB
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 25
Rate-Distortion Optimal Reference Picture Selection
( )
1
1,2,... ,( ) arg min ( )
L n
v vl vll
vv v
opt v V v
D p D
J D R
v n J n
[Wiegand, Färber, Girod, 2000]
[Liang, Girod, 2002]
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 26
RD Performance of Optimal Reference Picture Selection
LTM buffer: V=5 frames
Feedback round-trip time: dfb=7 frames
Packet loss rate: p=10% Comparison with P-I scheme, where each NACK triggers insertion of I-frame
[Liang, Girod, 2002]
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 28
MaD Sequence at 10% packet lossNo Retransmissions
Optimal Reference Picture SelectionRate: 320 kbps
Mean PSNR: 39.2 dB
Adaptive I-Frame InsertionRate: 320 kbps
Mean PSNR: 38.2 dB
Bernd Girod et al.: Advances in Network-adaptive Video Streaming 29
Conclusions
Network-adaptive video streaming: jointly optimize compression, error control, packet transport, and decoding
Adaptive media playout: “real-time” more flexible than we thought
RaDiO streaming can provide virtual priority mechanisms Proxy servers with RaDiO can improve streaming performance Path Diversity can improve streaming performance RaDiO streaming with multiple deadlines for accelerated
retroactive decodingof late retransmissions (“catch-up decoding”)
Packet dependency management allows robust representations for streaming w/o retransmissions