ph.d qualifying presentation - northwestern...
TRANSCRIPT
![Page 1: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/1.jpg)
Ph.D Defense
Optimal Cross-Layer Resource Allocation for Real-Time Video Transmission over Packet Lossy Networks
Fan ZhaiDept. Electrical & Computer Engineering
Northwestern UniversityFeb. 9, 2004
![Page 2: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/2.jpg)
2
Current Streaming Video Applications
VideoconferencingDistance learning
Base Station
NetworkVideophone
Web Server
video video decoderdecoder
Network
video video encoderencoderOn-demand streaming
![Page 3: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/3.jpg)
3
Emerging Streaming Video ApplicationsAnywhere, Anytime and Anyone
Edge server
Edge server
Data Farms / Storage
Web Server
Media ServerWireless Comm Server
Cellular mobile networks: 2.5G, 3G, 4G systemsWireless LANs: IEEE 802.11, Hiper LAN 2, Bluetooth
![Page 4: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/4.jpg)
4
Video Transmission System
Transport layer
video input
Receiver side Sender side
Network
Application layer
Video encoder
Transport layer
Application layer
Video decoder
Rate control
video packet
transportpacket
videooutput
Encoder: compressionRate control: to constraint bit rate based on estimated CSI Protocol stack: RTP/UDP/IPNetwork: packet loss and delayDecoder: display video in real-time; error concealmentMax end-to-end delay (setup time): application-dependent
![Page 5: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/5.jpg)
5
Challenges and Approaches
ApproachesNetwork-adaptiveError control• Unequal error protection
Cross-layer design
ChallengesBandwidth (limited, time-varying)
• Internet: congestion• Wireless: fading, shadowingQoS (packet loss, delay)• Distortion• Strict end-to end delay constraint
Limited resources• Internet: bandwidth, buffer• Wireless: battery,bandwidth
![Page 6: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/6.jpg)
6
Cross-Layer Design
Error control ComponentsError resilient source coding
adapt source coding to channel
FEC (forward error correction)modify channel characteristics
ARQ (delay constraint)Power control
modify channel characteristics
Modulation, Rate adaptationQoS support
reserve resource to support QoS
Error concealmentrecover/conceal the error at decoder
![Page 7: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/7.jpg)
7
Related Work
Optimization on each error control component•Error resilient source coding: R. Zhang, et al, ’00•FEC: B. Hong, et al ’02•ARQ: P. A. Chou, et al, ’01•Error concealment: Y. Wang et al, ’00
Cross-layer design•Joint source-channel coding: M. Gallant, et al, IEEE CSVT’01•Joint source coding and power control: Y. Eisenberg, et al, CSVT’02•Joint source-channel coding and power control:Y.S. Chan, et al, JSAC ’02, sub•Joint source-channel coding and power control:S. Appadwedula, et al, ICIP’98
Difference in our work•The available error control components are considered in an integrated manner•We consider error resilient source coding•We consider error concealment
![Page 8: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/8.jpg)
8
Presentation Outline
Resource-distortion optimization frameworkInternet: JSCC -- Joint source-channel coding Wireless: JSCCPA -- Joint source-channel coding and power allocation DiffServ: JSCPC -- Joint source coding and packet classificationExtensions to scalable video transmissionConclusions and future work
![Page 9: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/9.jpg)
9
Resource-Distortion Optimization
End-to-End Distortion
Cost Constraint
General formulation
µ : Source coding parameter vectorν : Resource allocation parameter vectorC0 is explicitly determined by specific applicationT0 is more implicitly determined by the application• obtained from higher-level rate controller, which
is not incorporated into this work.
0
0
},{
),( ),( s.t.
)],([min
TTCC
DE
≤≤
νµνµ
νµνµ
Transmission Delay Constraint
![Page 10: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/10.jpg)
10
End-to-End DistortionConsider end-to-end distortion
Original Encoded Decoded
Distortion of the i-th pixel in the n-th frame
])~E[(]~E[2)(])~E[(]E[ 2)()()(2)(2)()()( ni
ni
ni
ni
ni
ni
ni ffffffd +−=−=
Distortion for the k-th packet that has M pixelsMdD M
in
ik /]E[]E[1
)(∑ ==
![Page 11: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/11.jpg)
11
ROPE (Recursive Optimal Per-pixel Estimate)
1st order expected value (same fashion for 2nd order)• Intra:• Inter:
In order to calculate the distortion for one frame, only need
the 1st order and 2nd order expected reconstructed pixel value
in the previous frameThis algorithm is per-pixel accurate and can fully capture error propagation.
• R. Zhang, S.L. Regunathan, and K. Rose, “Video coding with optimal inter/intra-mode switching for packet loss resilience,” IEEE J. Select. Areas Commun., June 2000.
]~E[])~E[ˆ)(1(]~E[ )1()1()()( −− ++−= nik
nj
nik
ni ffef ρρ
]~E[]~E[)1(]~E[ )1()()( −+−= nik
nik
ni fff ρρ
![Page 12: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/12.jpg)
12
Expected Distortion
Expected distortionif the packet is
received
Expected distortionif the packet is
lostProbability of LossProbability of Reception
Expected distortion for the k-th packet
[ ] ( ) [ ] [ ]kLkkRkk D E ρ D Eρ DE ,,1 +−=
E[DR,k], E[DL,k] : depends on source coding parameter µk
ρk(ε, ν): depends on CSI and resource allocation parameter
Prob of loss for Prob of loss fortransport packet source/video packet
νkε ρk
FEC parameter
![Page 13: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/13.jpg)
13
Distortion Estimation Based on Feedbacks
Expectations are taken with respect to the updated probability distribution of channel losses given the available feedback
![Page 14: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/14.jpg)
14
Transmission Delay and Cost
Transmission Delay:
∑=
=M
k T
kkkk
RPµBC
1
)()( νEnergy:
Transmissionrate
∑=
=M
k T
kk
RµB T
1
)(
Number of bits
Number of packets in a frame
Packet index
Source codingparameter
Power allocationparameter
Transmission power
![Page 15: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/15.jpg)
15
Motion-Compensated Video Encoder
![Page 16: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/16.jpg)
16
Motion-compensated Video Decoder
![Page 17: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/17.jpg)
17
Motion-compensated Video Codec
Frame n Frame n-1
P
Motion vector
I
kµSource coding parameter • Prediction mode (Intra, Inter, Skip)• Quantization stepsize
Different encoding modes result in different levels of coding efficiency and error robustness
![Page 18: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/18.jpg)
18
Channel ModelNetwork model
• Independent time-invariant packet erasure channelPacket loss
• Internet: • Before FEC : ε -- Bernoulli process• After FEC :
• Wireless:
• i.i.d fading
• Bk : Packet size
• pe,k: BER after channel coding (depending FEC, channel SNR, and fading model)
kBkek p )1(1 ,−−=ρ
),( kk vf ερ =
![Page 19: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/19.jpg)
19
Channel Model—InternetNetwork model :
• Independent time-invariant packet erasure + random delays• Receiver responds to a received/lost packet with a
positive/negative acknowledgement. • Perfect feedback channel:constant delay and no error
})({)1( τεερ >−+= kTP nk ∆
ε
ε−1
t
)(tf
minFTT ∞
Packet delay–Exponential distribution: fast decaying tail–Gamma distribution–Pareto distribution: slowly decaying (heavy) tail
Packet Loss–Bernoulli–2-state Markov (Gilbert)–High-order Markov
![Page 20: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/20.jpg)
20
Channel Model—Wireless
Packet-erasure channel, from the point of view of the applicationsChannel BER (Bit Error Rate)
⎟⎟⎠
⎞⎜⎜⎝
⎛
+−=
b
b
ENEα
α
0
121BER
)2(BER0N
EQ b=
Rayleigh fading
AWGN
Probability of Packet Losskep ,
kB: BER after channel coding
: Packet sizekB
kek p )1(1 ,−−=ρ
Wireless channel: probability of packet loss depends on source coding parameter, channel coding parameter, and power level
![Page 21: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/21.jpg)
21
Implementation IssuesPacketization: each row of blocks (GOB) is packetized into a packet
Error concealment: temporal concealment, median motion vectors
Packet k-1 (recvd)
Packet k (lost)
Expected distortion
]E[]E[)1(]E[ ,, kLkRk DDD kk ρρ +−=
]E[]E[)1(]E[)1(]E[ ,,1, 1 kZkCkkRk DDDD kkkk −+−+−= − ρρρρρ
![Page 22: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/22.jpg)
22
Internet Video Transmission
T ra n sp o r t la y e r
T ra n sp o r t la y e r
V id e o in p u t
V id e oo u tp u t
V id e o e n c o d e r
V id e o d e c o d e r
E rro r c o n c e a lm e n t
E rro r re s ilie n tS o u rc e c o d in g
I n te r n e t
A p p lic a t io n la y e r F E C
A p p lic a t io n la y e r
We jointly consider error resilient source coding, FEC, and error concealment
![Page 23: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/23.jpg)
23
Joint Source-Channel CodingShannon’s separation theorem does not strictly hold due to the delay and complexity constraints
Source rateD1
D2
R2 R1 R3
D3
Overall distortion
R4
D4
D5
R5
Error rate The total bandwidth for source and channel rates is the same for the three curves
Overall distortion = source distortion + channel distortion
Optimal bit allocation depends on the channel characteristics
![Page 24: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/24.jpg)
24
Sequential Joint Source-Channel Coding
Step 1: Bit allocation between source and channel)]([min
}{νDE
R∈ν
0/)(/))(()( .. TRBRBTts TcTs ≤+= ννµν
Step 2: Source coding based on given bit budget
0,/)()( .. sTss TRBTts ≤= µµ
)]([min}{
µDEQ∈µ
Bs : source bitsBc : channel bitsTs,0: transmission delay constraint for the source
µ : source coding parameterν : channel coding parameterT0: transmission delay constraint
![Page 25: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/25.jpg)
25
Integrated Joint Source-Channel Coding
Solve (1) Bit allocation between source and channel and (2) source coding and channel coding, in one step
0/),(),( .. TRBTts T ≤= vµvµ
)],([min},{
νµνµ
DERQ ∈∈
B: total bits used for both source and channel coding
RT : transmission rate
T0 : transmission delay constraint for this frame
F. Zhai, Y. Eisenberg, T. N. Pappas, R. Berry, A. K. Katsaggelos, “An integrated joint source-channel coding framework for video transmission over packet lossy networks,” IEEE ICIP’04, submitted.
![Page 26: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/26.jpg)
26
Experimental Results
System 1: Integrated JSCC)],([min
},{νµ
νµDE
RQ ∈∈
System 2: Fixed channel coding rate
System 3: Sequential JSCC
)]([min}{ 0µ
µ ν,DEQ∈
)]([min}{
νDER∈ν
0/)(/))(()( .. TRBRBTts TcTs ≤+= ννµν
0,/)()( .. sTss TRBTts ≤= µµ
)]([min}{
µDEQ∈µ
![Page 27: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/27.jpg)
27
Experimental ResultsAverage PSNR vs. transport packet loss probability
System 1 vs. System 2 with indicated channel ratesSystem 1 vs. System 2 and 3
(RT=480 kbps, F=15 fps, cr in the legend denotes channel rates).
![Page 28: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/28.jpg)
28
Experimental ResultsAverage PSNR vs. transmission rate
System 1 vs. System 2 with indicated channel ratesSystem 1 vs. System 2 and 3
(ε=0.15, F=15 fps, cr in the legend denotes channel rates).
![Page 29: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/29.jpg)
29
JSCC—Packetization
Packet 1
Packet M
GOB 1
GOB M
Packet N-1 (Parity)
Packet N (Parity)
1sB
2sB
3sB
MsB
1cB
2cB
Stuffing bits
Largest data packet size
Packet 1 Packet N
GOB 1
GOB M
2cB2
sB
1sB 1
cB
3cB3
sB
MsB
1l
2l
3l
Ml
1v 1c
Packet 2
m bits
Packetization scheme 1 Packetization scheme 2
∑−−
=
−−−⎟⎟⎠
⎞⎜⎜⎝
⎛ −−=
−=kN
i
iNi
b
iNkNP
1
0
1 ))1(1
1(
),1(
εεε
ερ
∑−
=
−−⎟⎟⎠
⎞⎜⎜⎝
⎛−=
=kvN
i
iNi
kb
k
iNvNP
0)1(1
),(
εε
ρ
RS(N,M) RS(N,vk)
F. Zhai, Y. Eisenberg, C. E. Luna, T. N. Pappas, R. Berry, A. K. Katsaggelos, “Packetization schemes for forward error correction in Internet video streaming,” Proc. 41st Allerton Conf. Communication, Control, and Computing, Oct. 2003.
![Page 30: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/30.jpg)
30
JSCC — Solution AlgorithmLagrangian relaxation and DP
∑∑==
+=M
kTk
M
kk RµBDEJ kk
11},{/),()],([min νλνµ
νµ
Packetization 1
⎭⎬⎫
⎩⎨⎧
= ∑∑==
M
kk
M
kk ,νJJ
1}{}{1
*
}{)(minmin)|(min µµ
µννν
Packetization 2
∑∑=
−=
−=M
kkkk
M
kk ,ν,ν,µµJJ kk
11},{1},{
)(minmin 1νµνµ
![Page 31: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/31.jpg)
31
Experimental Results
Packetization scheme 1 vs 2
R1
R2
R3
Transmission rate 360Kbps Transmission rate 480Kbps
RTP/UDP/IP header = 40bytes
![Page 32: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/32.jpg)
32
FEC vs. ARQFEC:
• Usually preferred for real-time video applications• Cannot completely avoid packet loss• Incur constant overhead even when the channel has no loss• Depend on the accurate CSI estimation
ARQ:
• Can automatically adapt to the channel loss characteristics
• Longer delay
• Useful for long end-to-end delay applications, e.g., on-demand video streaming
• Useful for short RTT situations, e.g., LAN
![Page 33: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/33.jpg)
33
JSCC—Hybrid FEC/Retransmission
T ransport layer
T ransport layer
Video input
V ideooutput
Video encoder
Video decoder
E rror concealm en t
E rror resilien tS ource cod ing
Internet
Application layer FEC ,AR Q
Application layer
We jointly consider error resilient source coding, FEC, application-layer ARQ, and error concealment
F. Zhai, Y. Eisenberg, T. N. Pappas, R. Berry, and A. K. Katsaggelos, “Rate-distortion optimized hybrid error control for real-time packetized video transmission,” IEEE ICC’04, accepted.
![Page 34: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/34.jpg)
34
Delay-Constraint Retransmission
Time
Frame 1 2 3 4 …
Frame 1 2 3 4…
Tmax
Encoder
Decoderw(4)
w(4)+T_re(1)
w(4)+T_re(1)+ T_re(2)
TF
TF
Current time
Tmax: Maximum end-to-end delay (imposed by application)
w(n): waiting time for the n-th frame in the encoder buffer
T_re(n): transmission delay for retransmitting packets in the n-th frame
TF : One frame’s time (=1/TF)
![Page 35: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/35.jpg)
35
Problem Formulation
⎪⎪⎪
⎩
⎪⎪⎪
⎨
⎧
≤++
=−+−≤+
+=
∑∑∑
∑∑
∑∑∑
== =
−−
= =
−−
==
−−
=
−
max1
)(
1 1
)()()(
max1 1
)()()(
1
)(
1
)()(
0
)(
},,{
...1 )1( s.t.
)],([)]([][min
TTTw
AjTjATTw
DEDEDE
M
k
nk
A
i
M
k
ink
ink
n
F
j
i
M
k
ink
ink
n
M
k
nk
A
i
ininA
i
in
σ
σ
γγ
µσσµ
A: # of frames that retransmission is eligible
σ(n)={σ1(n),…,σM
(n)}∈{0,1} : Retrans parameter vector for frame n
γ : FEC parameter (using RS code)
Tk(n): Transmission delay for packet k in frame n.
Tmax can be replaced by Tmax-∆T(n), where ∆T(n) is decided by a rate controller.
![Page 36: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/36.jpg)
36
Problem Formulation
⎪⎪⎩
⎪⎪⎨
⎧
≤+
+=
∑∑∑
∑∑∑
== =
−−
==
−−
=
−
)(0
1
)(
1 1
)()(
1
)(
1
)()(
0
)(
},,{
s.t.
)],([)]([][min
nM
k
nk
A
i
M
k
ink
ink
M
k
nk
A
i
ininA
i
in
TTT
DEDEDE
σ
γγ
µσσµ
We assume T0(n) is given from higher-level rate controller
Optimization with a sliding window of A+1 frames
Optimization decisions: retransmission policy for the first Aframes, and source coding and FEC for the current frameBased on the updated probabilities of packet loss, recalculate the expected distortion of all packets in the windowOptimization window shifts at the frame level
![Page 37: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/37.jpg)
37
Probability of Packet Loss for the Current FrameProbability of packet loss
)(,
)(,
)( nRETk
nFECk
nk ρρρ = Due to
retransmissionDue to FEC
The probability of loss in future retransmissions can only be estimated since the acknowledgement information and retransmission decisions are not available in the encoding of the current frame
mnRETk ερ =)(
,
2)1(~
RTTAm
+=
Approximation formula
The estimate of the total number of retransmissions
![Page 38: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/38.jpg)
38
Estimating m~
Average PSNR vs. m~
QCIF Foreman sequence at F=15 fps, RT=480 kbps and A=4
![Page 39: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/39.jpg)
39
Estimating m~
Average PSNR vs. m~
QCIF Akiyo sequence at F=15 fps, RT=360 kbps and A=4
![Page 40: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/40.jpg)
40
Probability of Packet Loss for the Past FrameProbability of packet loss
)(,
)(,
)( nRETk
nUPDk
nk ρρρ = Due to
retransmissionUpdated loss probability based on feedback
ρ (n)k,UPD can be accurately calculated
ρ (n)k,RET can be calculated using the similar way as before
![Page 41: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/41.jpg)
41
Probability of Packet Loss for the Past FrameAssume protected by RS(N, M),
• L: # packets lost• V: # retransmitted packets• J=L+M-N
)()(,
inkin
RETk
−
=− σερIf V<J
If V=J
If V>J⎩⎨⎧
=−−=
=−
−−
1 if )1(10 if
)(
)()(
, ink
J
inkin
RETk σεσε
ρ
⎪⎪⎩
⎪⎪⎨
⎧
=−⎟⎟⎠
⎞⎜⎜⎝
⎛
=−⎟⎟⎠
⎞⎜⎜⎝
⎛
=−
+−=−
−+−=
−
−
∑
∑
1 if )1(
0 if )1(
)(1j
)(1j
)(,
ink
V
JVjVj
ink
V
JVjVj
inRETk
jV
jV
Vj
σεε
σεερ
![Page 42: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/42.jpg)
42
Solution Algorithm
Lagrangian Relaxation
⎭⎬⎫
⎩⎨⎧
++
+=
∑∑∑
∑∑∑
== =
−−
==
−−
=
−
M
k
nk
A
i
M
k
ink
ink
M
k
nk
A
i
ininA
i
in
TT
DEDEJ
1
)(
1 1
)()(
1
)(
1
)()(
0
)(
},,{
)],([)]([min
σλ
γγ
µσσµ
Minimization of the Lagrangian
⎭⎬⎫
⎩⎨⎧
+= ∑∑∑==
−−
=
−M
k
nk
A
i
ininA
i
in JJJ1
)(
}{}{1
)()(
}{0
)(
},,{),(minmin)](minmin γ
γγµσ
µσσµ
Retransmission Exhaustive
searchFEC
Exhaustive search
Source codingDP
![Page 43: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/43.jpg)
43
Simulation – Hybrid FEC/Retransmission
NFNR– Neither FEC Nor Retransmission
)]([min 00}{σµ
µ,,DE γ
Pure Retransmission
)]([min 0}{σ,µ
σµγ,DE
,
Pure FEC)],,([min 0}{
σµµ
γγ
DE,
HFSR (Hybrid FEC and Selective Retransmission))],,([min
},{σµ
σµγ
γDE
,
![Page 44: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/44.jpg)
44
Sensitivity to RTTAverage PSNR vs. RTT
ε=0.02 ε=0.2
RT=480 kbps, F=15 fps , QCIF Foreman
![Page 45: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/45.jpg)
45
Sensitivity to Packet Loss RateAverage PSNR vs. probability of transport packet loss ε
RT=480 kbps, F=15 fps , QCIF ForemanRTT=TF RTT=3TF
![Page 46: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/46.jpg)
46
Sensitivity to Transmission RateAverage PSNR vs. channel transmission rate RT
RTT=TF RTT=3TF
ε=0.2, F=15 fps , QCIF Foreman
![Page 47: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/47.jpg)
47
JSCC– Conclusions
Retransmission is suitable for short network RTT, low probability of packet loss, and low transmission rate.
FEC is more suitable otherwise.
The proposed hybrid FEC/selective retransmission scheme outperforms both (average gain: 0.7 dB in PSNR).
In our simulations, we assume the CSI is accurately estimated, which favors FEC, because ARQ does not require accurate CSI.
![Page 48: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/48.jpg)
48
Wireless Video Transmission
T r a n s p o r t l a y e r
T r a n s p o r t l a y e r
V id e o in p u t
V id e oo u t p u t
V i d e o e n c o d e r
V id e o d e c o d e r
E r r o r c o n c e a lm e n t
P o w e r c o n t r o l
E r r o r r e s i l i e n tS o u r c e c o d in g
W i r e l e s sN e t w o r k s
A p p l i c a t i o n l a y e r
F E C
A p p l i c a t i o n l a y e r
P h y s i c a l l a y e r
P h y s i c a l l a y e r
L i n k l a y e r L i n k l a y e r F E C
JSCCPA– Jointly Source-Channel Coding and Power Allocation
F. Zhai, Y. Eisenberg, T. N. Pappas, R. Berry, A. K. Katsaggelos, “Joint source-channel coding and power allocation for energy efficient wireless video communications,” Allerton Conf. Communication, Control, and Computing, Oct. 2003.
![Page 49: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/49.jpg)
49
Hybrid Wireless Network
kBbkkkk p )1(1),,( −−=ηνµβ
At the IP level, the hybrid network can be modeled as the combination of two independent packet erasure channels: the wired part with loss rate α and the wireless part with loss rate β
Overall probability of loss for a transport packet
Probability of loss for a transport packet in the wireless channel
We need Product FEC to combat different types of channel error
kk βααε )1( −+=
Base Station
Wiredchannel
αβ
![Page 50: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/50.jpg)
50
Transport-Layer Packetization
Packet 1
Packet M
GOB 1
GOB M
Packet N-1 (Parity)
Packet N (Parity)
1sB
2sB
3sB
MsB
1cB
2cB
Stuffing bits
Largest data packet size R
S co
ding
One row corresponds to one source packetInter-packet FECRS coding RS(N,M)—transport-layer FECTransport-layer FEC parameter: γ
![Page 51: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/51.jpg)
51
Link-Layer Packetization
Packet 1
P acket M
GO B 1
GO B M
2,cB2,sB
1,sB 1,cB
MsB ,
3,sB
McB ,
Parity packet
Parity packet
RCPC coding
1, −NsB
NsB ,
1, −NcB
NcB ,
One row corresponds to one source/transport packetIntra-packet FECRCPC coding—link-layer FECLink-layer FEC parameter vector },...,{ 1 Nνν=ν
![Page 52: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/52.jpg)
52
Problem Formulation – Hybrid Network
0
)(
1
0
)(
1
},,,{
/
/ s.t.
)],,,([min
TRBT
CRPBC
DE
N
kTk
N
kTkk
≤=
≤=
∑
∑
=
=
γ
γ
γγ ηνµ
ηνµ
µ ={µ1,…,µM} : Source coding parameter vectorγ ∈{(N1,M),.., (Nq,M)} : Transport-layer FEC parameterν ={ν1,…, νN} : Link-layer FEC parameter vectorη ={η1,…, ηN} : Power level parameter vector
![Page 53: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/53.jpg)
53
Solution AlgorithmLagrangian relaxation
)()(][),,,,,( 020121 TTCCDEL −+−+= λλλλγ ηνµ
• Have proposed an iterative search algorithm to solve the above problem (Allerton’03)
Minimization of the Lagrangian
),,,,,(min),( 21},,,{21 λλγλλγ
ηνµηνµ
Lg =
![Page 54: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/54.jpg)
54
Solution AlgorithmLagrangian relaxation
)()(][),,,,,( 020121 TTCCDEL −+−+= λλλλγ ηνµ
Minimization of Lagrangian
),,,,,(min),( 21},,,{21 λλγλλγ
ηνµηνµ
Lg =
Dual Problem),(max 21}0,0{ 21
λλλλ
g≥≥
Goal: find the convex-hull solutionThe duel problem is concavecomplementary slackness applies
![Page 55: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/55.jpg)
55
Algorithm– Lagrangian relaxation
case (a) case (b)
λ1
λ 2
C=C0
T=T0
λ 1
λ 2
T=T0
C=C 0
λ1
λ2
C=C0
T=T0
λ1
λ2
T=T0
C=C0
case (c) case (d)
![Page 56: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/56.jpg)
56
Algorithm– Lagrangian Relaxation
Step 1 (case a, d): Let 2λ =0, find *1λ to satisfy 0
*1 ))0,(( CHC ≤λ .
If 0*1 ))0,(( THT ≤λ , )0,( *
1λH corresponds to the optimal solution. Otherwise,
Step 2: (case b, d): Let 1λ =0, find *2λ to satisfy 0
*2 )),0(( THT ≤λ .
If 0*1 ))0,(( CHC ≤λ , ),0( *
1λH corresponds to the optimal solution. Otherwise, Step 3 (case c):
i. Let 01 =lλ , *
11 λλ =r , 02 =bλ , *
22 λλ =t . ii. 2/)( 111
rlm λλλ += , find *2λ within [ b
2λ , t2λ ] to satisfy 0
*21 )),(( THT m ≤λλ .
iii. If 0*21 )),(( CHC m >λλ , then let ml
11 λλ = , *22 λλ =t , and go to step 3ii.
Otherwise, iv. If ελλ −< 0
*21 )),(( CHC m (ε is a relatively small number), then let
mr11 λλ = , *
22 λλ =b , and go to step 3ii. Otherwise, v. The optimal solution corresponds to ),( *
21 λλmH .
![Page 57: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/57.jpg)
57
Calculation of Probability of Packet Loss
∑ ∑ ∏ ∏∑+−= ∈ ∈ ∈+−= ⎟
⎟
⎠
⎞
⎜⎜
⎝
⎛−==
N
MNj kNIQ Qi Qlli
N
MNjbk
jtj
tj
tj
jNP1 ),(
)(
1)()1( ),(),,,( εεγρ
γ
γ
ηνµ
the t-th subset with jelements of Q(N)⎟⎟
⎠
⎞⎜⎜⎝
⎛==
jN
tNjNQtj ,...,1,,...,1 ),(
}||),(|)({),( jQNQkNQQkNI tj
tj
tjj =∈∈=
Loss probability of a source packet depends on the parameters chosen for all the other packets in the frame
• εk differs from packet to packet
• Inter-packet dependency introduced by inter-packet FEC
![Page 58: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/58.jpg)
58
Calculation of Probability of Packet Loss
∑ ∑ ∏ ∏
∑
+−= ∈ ∈ ∈
+−=
⎟⎟
⎠
⎞
⎜⎜
⎝
⎛−=
=
N
MNj kNIQ Qi Qlli
N
MNjbk
jtj
tj
tj
jNP
1 ),(
)(
1)(
)1(
),(),,,(
εε
γργ
γ
ηνµ
Probability that the k-th packet is correctly decoded by the RCPC decoder and the total number transport packets that are not correctly received from the group of N packets is j
),( jNPb
},3,2,1{)(3},3,2{)(3},3,1{)(3},2,1{)(3
{3}(3){2},(3) ,{1}(3) },3,2,1{)3(13
32
22
12
31
21
11
====
====
QQQQ
QQQQ
![Page 59: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/59.jpg)
59
Minimization of the Lagrangian– Hybrid
Iterative descent algorithm (alternating variables)
),...,,,,...,(minarg )()(1
)(1
)(1
)()(
tN
tii
ti
t
x
ti xxxxxLx
t +−=
Step 1: Set x(0) corresponds to any initial state
Step 2: let tn=(t mod N) (round-robin style)
if i≠tn, let ; otherwise, for i=tn)1()( −= n
in
i xx
Step 3: If not convergence, go back to Step 2
},...,,{ 21 Nxxxx = Parameter selected for the N packets
at step t,...1,0for },...,,{ )()(2
)(1
)( == txxxx tN
ttt
![Page 60: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/60.jpg)
60
Experimental Setup
“Real Time” Applications• Short-delay applications, where ARQ is not applicable
• Delay constraint equals one frame timeChannel Model
• Flat Rayleigh fading channel with AWGN• i.i.d. channel fading• RTT = 2TF , which preclude retransmission
![Page 61: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/61.jpg)
61
Experimental SetupPerformance of RCPC (in BER) over a
Rayleigh fading channel with interleaving
SNR(dB) 2 6 10 14 18
Cr = 1/2Cr = 4/7Cr = 2/3Cr = 4/5
1.4*10-3
1.1*10-1
3.2*10-1
4.2*10-1
2.2*10-5
5.3*10-4
7.4*10-3
4.0*10-2
2.1*10-6
4.1*10-5
1.7*10-4
6.6*10-4
2.4*10-7
1.2*10-5
3.5*10-5
1.1*10-4
6.4*10-8
3.8*10-6
1.2*10-5
3.6*10-5
Generator polynomial (133, 171), mother code rate ½, and puncturing rate 4Soft Viterbi decoding in conjunction with BPSK
![Page 62: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/62.jpg)
62
PFEC vs. Link-Layer FEC
Approach 1 – Link Layer FEC (LFEC)
)],,,([min 00}{ηνµ
νµγDE
,
Approach 2 – Product FEC (PFEC)
)],,,([min 0},{ηνµ
νµγ
γDE
,
F. Zhai, Y. Eisenberg, T.N. Pappas, R. Berry, A..K. Katsaggelos, “Rate-distortion optimized product code forward error correction for video transmission over IP-based wireless networks,” IEEE ICASSP’04, accepted.
![Page 63: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/63.jpg)
63
PFEC vs. LFEC
Average PSNR vs. channel SNRAverage PSNR vs. α
QCIF Foreman, 30 fps
![Page 64: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/64.jpg)
64
UEP vs. EEP
Approach 1 – EEP-PFEC
)],,,([min 00},{ηνµ
µγ
γDE
Approach 2 – UEP-PFEC
)],,,([min 0},{ηνµ
νµγ
γDE
,
![Page 65: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/65.jpg)
65
UEP vs. EEP
PSNR vs. channel SNR, α=0.1
QCIF Foreman, 30 fps
![Page 66: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/66.jpg)
66
Problem Formulations—Wireless Channel
01
01
},,{
/
/ s.t.
)],,([min
TRBT
CRPBC
DE
M
kTk
M
kTkk
≤=
≤=
∑
∑
=
=
ηνµηνµ
![Page 67: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/67.jpg)
67
Solution Algorithm–– Wireless
Lagrangian relaxation
∑∑ ==++==
M
k kkkM
k k TCDEJL1 211
][),,( λληνµ
Dynamic Programming
),,,,,( 111 kkkkkkkk JJ ηνµηνµ −−−=
![Page 68: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/68.jpg)
68
Simulation 1– Wireless Channel
Goal: To show the benefit by adjusting power levels
System 1 -- Optimal error-resilient source coding
)],,([min 00}{ηνµ
µDE
System 2 -- Joint source coding and power allocation
)],,([min 0},{ηνµ
ηµDE
![Page 69: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/69.jpg)
69
Simulation 1
PSNR vs. transmission rate
(reference SNR = 12 dB)PSNR vs. channel SNR
(Transmission rate = 360 kbps)QCIF Foreman, 30 fps
![Page 70: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/70.jpg)
70
Simulation 1
Allocation of power level (1,2,3,4,5) in percentage in JSCPA system
Ref SNR(dB)
6 12 20
Cr = 1/2Cr = 4/7Cr = 2/3Cr = 4/5
(2.4,18.5,73.9,5.1,0)(18,0,14.3,66.1,1.6)
(40,0,0,13,47)(45.8,0,0,0,54.2)
(12.6,32.4,34,19.6,1.4)(2.3,29.9,56.4,11,0.3)(0.7,14,66,18.7,0.6)
(2,4,41.8,47.3,5)
(62.3,0,12.9,0.24.8)(10,35,39.2,13.4,2.3)
(11.6,10.8,69.1,6.9,1.6)(8.2,31.5,43.8,15.3,1.3)
The reference power level 3Each value here denotes the percentage of packets using the corresponding transmission power level.The transmission power is proportional to the power level parameter
![Page 71: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/71.jpg)
71
Simulation 2
Goal: To show the benefit by adjusting channel rates
System 2 -- Joint source coding and power allocation
)],,([min 0},{ηνµ
ηµDE
System 3 -- JSCC and power allocation (JSCCPA)
)],,([min},,{
ηνµηνµ
DE
![Page 72: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/72.jpg)
72
Simulation 2
PSNR vs. channel SNR(transmission rate = 360 kbps)
PSNR vs. transmission rate(reference SNR = 12 dB)
QCIF Foreman, 30 fps
![Page 73: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/73.jpg)
73
Simulation 2
Channel Coding rates in percentages in JSCCPA system
Ref SNR(dB)
6 8 10 12 14 16 18 20
6.761.331.30.7
4.735.057.72.6
1.65.6
69.623.2
1.017.873.97.3
19.669.610.8
0
Cr = 1/2Cr = 4/7Cr = 2/3Cr = 4/5
96.23.800
67.731.90.40
41.257.31.50
The better channel, the less channel coding protection
![Page 74: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/74.jpg)
74
JSCCPA– Conclusions
Optimal cross-layer resource allocation to provide UEP for wireless video transmission
• Error resilient source coding• Transport-layer FEC• Link-layer FEC• Physical-layer power allocation• Error concealment
Transport-layer FEC (inter-packet FEC) is not necessary if the wired link has no error.Adapting either power or link-layer FEC (instead both) may be adequate to achieve the near-optimal result in some situations.
![Page 75: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/75.jpg)
75
DiffServ Video Transmission
T ra nsport layer
T ransport la yer
V ideo inpu t
V ideoou tpu t
V ideo encoder
V ideo dec oder
E rror con cea lm en t
E rror res ilien tS ou rce cod in g
D iffS e rvN etw ork
A pplica tion layer
A pplica tion la yer
Q oS S u p p ort
JSCPC– Jointly Source Coding and Packet Classification
F. Zhai, C. E. Luna, Y. Eisenberg, T. N. Pappas, R. Berry, A. K. Katsaggelos, “Joint source coding and packet classification for real-time video transmission over differentiated services networks,” IEEE Trans. Multimedia, to appear, 2004
![Page 76: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/76.jpg)
76
Differentiated Services Networks
Allocating resources discriminatorily to aggregated traffic flowsMultiple service classes: different class has different end-to-end statistical behaviorSender is charged for each transmitted bit based on its service class
![Page 77: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/77.jpg)
77
Delay Components
ddbnebe TTTTTT ∆+∆+∆+∆+∆=Constant end-to-end delay:
Encoderbuffer
decoder buffer
videoinput
videooutput
videoencoder
videodecoder
∆Tn ∆Tdb ∆Td ∆Teb ∆Te
networks
kTp (M-k+1)TpTmax=T-(M+1)Tp
Frame level
Packet levelAn example(M=6, k=3)
max)1()()()( TTMTkTkTkT pdbneb =+−=++ ∆∆∆
max)()()( 0)( :delay excessive avoid to TkTkTkTkT nebdb ≤+=⇒≥ ∆∆∆∆
![Page 78: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/78.jpg)
78
Delay Components—cont.
Packet departures
k-1 k
k-1
w(k-1)
k Bk-1(µk-1)/Rk-1(πk-1)
Tp w(k)
Time
Packet arrivals
Bk-2(µk-2)/Rk-2(πk-2)
k-2
k
kk
RB
kwkTeb
)()()( :delaybuffer encoder
µ+=∆
k
kk
RB
kwTkTTk eb
)()()()( :delaynetwork allowablemax maxmax
µτ −−=∆−=
pTR
Bkwkw
k
kk−+−=
−
−−
1
11 )()1()( : timewaiting
µ
![Page 79: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/79.jpg)
79
JSCPC– Solution Algorithm
Lagrangian relaxation 1
∑=
++=M
kkkk TCDL
12121},{
}{),,,(min λλλλµπµπ
Lagrangian relaxation 2
∑=
+=M
kkk CDL
1},{},{min),(min λλ
πµπµπ,µ
01
.. TTtsM
kk ≤∑
=
Dependency of the Lagrangian
∑=
=M
kkJL
121},{),,,(min λλµπ
µπ
),,,,...,,( 1111 kkkkkk JJ µπµπµπ −−=
![Page 80: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/80.jpg)
80
JSCPC— Solution Algorithm contd.
q1
c3
q2
q1
c3
q2
q1
c2
q2
q1
c1
q2
q1
c1
q2
q1
c2
q2
q1
c2
q2
q1
c1
q2
q1
c3
q2
k-1 k k+1
![Page 81: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/81.jpg)
81
JSCPC— Solution Algorithm contd.
),,,,...,,( 1111 kkkkkk JJ µπµπµπ −−=
]E[]E[)1(]E[)1(]E[ ,,, 11 kZkCkRk DDDD kkkkk −− +−+−= ρρρρρ
))(,,,,( 11 kwJJ kkkkkk πµπµ −−=
Do DP with respect to “system state”, w(k), instead of to choice of source coding parameters and service classes
)}()({)1( kkTP nk τεερ >∆−+=
peb TkwTR
BkwTkTTk
k
kk++−=−−=∆−= )1(
)()()()( maxmaxmax
µτ
pTR
Bkwkw
k
kk−+−=
−
−−
1
11 )()1()(
µ
![Page 82: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/82.jpg)
82
JSCPC— Solution Algorithm contd.
0011
)( )( : )()( TMsTkTjTksM
k
k
j
≤⇒≤= ∑∑==
p
k
jp
p
TkkswTkjTw
TkTkwkw
)1()1()1()1()()1(
)1()1()(1
1
−−−+=−−+=
−−+−=
∑−
=
DP solution
∑∑==
∈−−=
M
k
M
kkwkukwJkJ kkkk
11))}(()({))(,,,,()(min 11 πµπµ
U
)})1(,min()()()(
0:)({))(( 0max pp kTwTTTkwRB
kukwkk
kk−+≤−+≤×∈=
πµ
QΠU
MkTTkwkw p ,...,1for )1()1()( 0 =≤−+−
![Page 83: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/83.jpg)
83
JSCPC— Solution Algorithm contd.
w3
w4
Packet 2 Packet 3
w1
w2
w3
w4
w1
w2
after pruning
Intra
Inter
Intra
Inter
Intra
Inter
Inter
Intra
w2
Packet 1
initial condition
w3
w4
Packet k-1 Packet k Packet k+1
w1
w2
w3
w4
w1
w2
w3
w4
w1
w2
after pruning
Intra
Inter
Intra
Inter
Intra
Intra
Inter
Inter
Intra
Inter
Inter
Inter
Inter
Intra
Intra
Intra
![Page 84: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/84.jpg)
84
JSCPC — Experimental Results
Minimum distortion approach Minimum cost approach
QCIF Foreman, F=30 fps, reference class = 3
![Page 85: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/85.jpg)
85
JSCPC — Experimental Results
minimum distortion approach
minimum cost approach
![Page 86: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/86.jpg)
86
Distribution of packet classification
minimum distortion approach
minimum cost approach
![Page 87: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/87.jpg)
87
Internet Scalable VideoJSCC based on H.263+ SNR scalability codec
I
P
P
EI
EP
EP
Base layer
Enhancement layer:
Problem Formulation
)],([)],([][min )()()()()()(
},,,{ )()()()(
eeebbb DEDEDEeebb
νµνµνµνµ
+=
)(0
)( s.t. bb TT ≤
0)()( TTT eb ≤+
F. Zhai, R. Berry, T. N. Pappas, and A. K. Katsaggeos, “Rate-distortion optimized error control scheme for scalable video streaming over the Internet,” Proc. IEEE ICME, July 2003
![Page 88: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/88.jpg)
88
Wireless Scalable VideoOptimal power allocation based on MPEG-4 FGS codec
I
B
Base layer
FGS layer:
P
B
P
Rmax . . . . . . .
Rmin
FGST
layer:
{ }][min )()(
}{ )(
eb
PED
ei
∆−
0 s.t. EE tot =
Problem Formulation
Assume base layer is pre-encoded and always received correctlyC. Costa, Y. Eisenberg, F. Zhai, and A. K. Katsaggelos, “Energy efficient wireless transmission of MPEG-4 Fine Granular Scalable video,” IEEE ICC’04, 2004, accepted.
![Page 89: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/89.jpg)
89
ConclusionsObjective:
• Optimal error control to achieve the best video qualityMethod:
• Optimal cross-layer resource allocationBase:
• Resource-distortion optimization frameworkScope:
• End-system design for video communicationHave studied following applications:
• JSCC—Internet video (Allerton’03, ICC’04, ICIP’04, TIM’04)• JSCCPA—wireless video (Allerton’03, ICASSP’04)• JSCPC– DiffServ video (ICIP’03, TMM’04)• Scalable video (ICME’03, ICC’04)
![Page 90: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/90.jpg)
90
Future Work
Rate control• Incorporate rate control into the optimization framework
Channel model• Take into account the correlation of packet loss, e.g.,
Gilbert model Cross-layer Design
• Consider modifying the current network protocols: Link-layer ARQ, UDPLite,
![Page 91: Ph.D qualifying presentation - Northwestern Universityusers.eecs.northwestern.edu/~fzhai/publications/PhD_Defense.pdf · Presentation Outline ¾Resource-distortion optimization framework](https://reader030.vdocument.in/reader030/viewer/2022041022/5ed27f84ad716a665e6d908a/html5/thumbnails/91.jpg)
91
Q&A
Thank You!