![Page 1: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/1.jpg)
http://medianetlab.ee.ucla.edu
A Unified Framework for Delay-Sensitive
Communications
Fangwen Fu
Advisor: Prof. Mihaela van der Schaar
![Page 2: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/2.jpg)
2
Motivation
Sensor networks Wireless video phone
• Delay sensitive multimedia applications are booming over a variety of time-varying networks (e.g. sensor networks, WiMax, Wireless LAN, etc.)
• Existing dynamic distributed network environments cannot provideadequate support for delay-sensitive multimedia applications
• This problem has been investigated for a decade, but we still do not have efficient solutions for it.
VOIP Video conference
RG
Access Point
DVD Recorder
HDTV
SDTVPDA
Bridge
Internet
Internet Access
Content Server
RG
Access Point
DVD Recorder
HDTV
SDTVPDA
Bridge
Internet
Internet Access
Content Server
In-home streaming
![Page 3: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/3.jpg)
3
Challenges
Transmitter Receiver
Transmitter Receiver
Channel state
Data arrival
• Challenge 1: Unknown time-varying environments
– Time-varying data arrivals and channel conditions
– Lack of statistic knowledge of dynamics
• Challenge 2: Heterogeneity in the data to transmit (e.g. media data)
– Different delay deadlines, importance, and dependencies
• Challenge 3: Coupling in multi-user transmission
– Mutual impact due to dynamically sharing of the same network resources (e.g. bandwidth, transmission opportunities) by multiple users
![Page 4: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/4.jpg)
4
Existing solutions-1
• Minimize average delay for homogeneous traffic in point-to-point communications
• Information theory [Shannon and beyond] –Challenge 1
– Water-filling algorithms
– Maximize the throughput without delay constraints
• Control theory – Challenge 1
– Markov decision process (MDP) formulation [Berry 2002, Borkar 2007, Krishnamurthy 2006]
• Statistic knowledge of the underlying dynamics is required
– Online learning [Krishnamurthy 2007, Borkar 2008]
• Slow convergence and large memory requirement
– Stability-constrained optimization for single-user transmission [Tassiulas 1992,2006, Neely 2006, Kumar 1995, Stolyar 2003]
• Queue is stable, but delay performance is suboptimal (for low delay applications)
Transmitter Receiver
![Page 5: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/5.jpg)
5
Existing solutions-2
• Maximize quality of delay-sensitive applications with heterogeneous
traffic
• Multimedia communication theory –Challenge 2
– Cross-layer optimization [van der Schaar 2001, 2003, 2005, Katsaggelos 2002]
• Observes and then optimizes (i.e. myopic optimization)
– Rate distortion optimization (RaDiO) [Chou, 2001, Frossard 2006, Girod 2006, Ortega
2009]
• Explicitly considers importance, delay deadlines and dependencies of packets
• Linear transmission cost (e.g. not suitable for energy-constrained transmission)
• No learning ability in unknown environments
– Both solutions only explore the heterogeneity in the media data, but do not explore the
network dynamics (e.g. time-varying channel conditions) and resource constraints.
Transmitter Receiver
![Page 6: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/6.jpg)
6
Existing solutions-3
• Multi-user transmission by sharing network resources
• Network optimization theory
– Network utility maximization [Chiang 2007, Katsaggelos 2008] –Challenge 3
• Uses static utility function without considering the network dynamics
• No delay guarantee
• No learning ability in unknown environments
– Stability-constrained optimization for multi-user transmission [Tassiulas 1992,
2006, Neely 2006, 2007, Kumar 1995, Stolyar 2003] - Challenges 1 and 3
• Queue is stable, but delay performance is suboptimal (for low-delay applications)
• Does not consider heterogeneous media data
Transmitter Receiver
Transmitter Receiver
![Page 7: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/7.jpg)
7
A unified foresighted optimization
framework
Current time slot Next time slot
u(s, y) V (f(s, y, w))Ewmaxy{ }+
Current utility State-value function
Challenges Solutions
dynamic systems Foresighted optimization framework
Unknown dynamics Online learning
Learning efficiency
Heterogeneity
Multi-user coupling
State: s
Action: y
State:
Dynamics: w
Separation principles
Queue length
Channel condition
Heterogeneity
s′ = f(s, y, w)
![Page 8: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/8.jpg)
8
Key accomplishments
Reduce the delay by
70% (at low delay region)
Stability constrained optimization
[Neely 2006]
Energy-efficient data
transmission*
Improve up to 5dB in
video quality
Rate-distortion optimization [Chou
2001]
Wireless video
transmission
Improve 1~3dB in video
quality
Network utility maximization [Chiang
2007]
Multi-user video
transmission
ImprovementsPrevious state-of-art methods
*minimize the average delay
![Page 9: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/9.jpg)
9
Roadmap
• Separation principle 1 (improving learning efficiency)
– Post-decision state-based formulation
– Structure-aware online learning with adaptive approximation
• Separation principle 2 (Separating the foresighted decision for
heterogeneous media data transmission)
– Context-based state
– Priority-based scheduling
• Separation principle 3 (decomposing multi-user coupling )
– Multi-user Markov decision process formulation
– Post-decision state value function decomposition
![Page 10: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/10.jpg)
10
Roadmap
• Separation principle 1 (improving learning efficiency)
– Post-decision state-based formulation
– Structure-aware online learning with adaptive approximation
• Separation principle 2 (Separating the foresighted decision for
heterogeneous media data transmission)
– Context-based state
– Priority-based scheduling
• Separation principle 3 (decomposing multi-user coupling )
– Multi-user Markov decision process formulation
– Post-decision state value function decomposition
![Page 11: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/11.jpg)
11
Energy-efficient data transmission
Transmitter Receiverat
xt ytht
• Point-to-point time-slotted communication system
• System variables
– Backlog (queue length):
– Channel state: Finite state Markov chain (e.g. Rayleigh fading)
– Data arrival process:
• Decision at each time slot
– Amount of data to transmit (transmission rate):
– Energy consumption:
xt
ht
yt, 0 ≤ yt ≤ xt
ρ(ht, yt), convex in yt, e.g. ρt(ht, yt) = σ2 (2yt−1)ht
.
at: i.i.d.
What is the optimal (queueing) delay and energy trade-off?
![Page 12: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/12.jpg)
12
Foresighted optimization formulation
• Foresighted optimization (MDP) formulation
– State:
– Action:
– Policy:
– Utility function:
• Objective (optimize the trade-off between delay and energy consumption)
– State value function:
• Bellman’s equations
– Policy iteration
(xt, ht)ytπ : (xt, ht)→ yt
25 30 35 40 450
2
4
6
8
energy consumption
avera
ge
dela
y
Constant rate
Optimal trade-off
Constant energy
α ∈ [0, 1) is discount factor.maxπE
∞∑
t=0
αt {u(xt, ht, π(xt, ht))}
u(xt, ht, yt) = −(xt − yt + λρ(ht, yt)).
V (xt, ht) = maxπE
∞∑
k=t
α(k−t) {u(xk, hk, π(xk, hk))}
V (x, h) = maxπ{u(x, h, π(x, h)) + αEa,h′|hV (x− π(x, h) + a, h
′)}
![Page 13: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/13.jpg)
13
Challenges for solving the Bellman’s
equations
• Lack of statistical knowledge of the underlying dynamics
– Unknown traffic characteristics
– Unknown channel (network) dynamics
• Coupling between the maximization and expectation
• Curses of dimensionality
– Large state space
• Intractable due to large memory and heavy computation requirements
Bellman’s equation:
V (x, h) = maxπ{u(x, h, π(x, h)) + αEa,h′|hV (x− π(x, h) + a, h
′)}
![Page 14: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/14.jpg)
14
Conventional online learning methods
• Decision and dynamics
• Foresighted optimization
• Online learning
– Learn Q-function (Q-learning):
(xt, ht) (xt+1, ht+1)
…Exogenous dynamicsDecision ytat, ht+1
Normal state Normal state
V (xt, ht) V (xt+1, ht+1)
State-value functionState-value function
V (x, h) = max0≤y≤x
{u(x, h, y) + αEa,h′|hV (x− y + a, h′)}
Q(x, h, y)
Q(x, h, y)
Low convergence, high space complexity
![Page 15: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/15.jpg)
15
Our approach- separation via post-decision
state
(xt, ht) (xt+1, ht+1)
V (xt, ht) V (xt+1, ht+1)U(xt, ht)
Post-decisionstate-value function
…
State-value functionState-value function
Post-decision state separates foresighted decision from dynamics.
U(x, h) = Ea,h′|hV (x+ a, h′)V (x, h) = maxy{u(x, h, y) + αU(x− y, h)}
Exogenous dynamicsDecision ytat, ht+1
Foresighted decision
Expectation over dynamics
Foresighted decision Expectation over dynamics
Normal stateNormal state(xt − yt, ht)
Post-decision state
![Page 16: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/16.jpg)
16
Post-decision state-based online learning
• Online learning
e.g. βt = 1/t
Vt(x, ht) = maxy∈Y
{u(x, ht, y) + αUt−1(x− y, ht)}
Ut(x, ht−1) = (1− βt)Ut−1(x, ht−1) + βtVt(x, ht)
Foresighted decision
Online update
U(x, h) = Ea,h′|hV (x+ a, h′)
V (x, h) = maxy{u(x, h, y) + αU(x− y, h)}
Expectation is independent of backlog x → batch update (fast convergence).
Theorem:
Online adaptation converges to the optimal solution when t→∞
Time-average
Batch update incurs high complexity. �
![Page 17: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/17.jpg)
17
Structural properties of optimal solution
• Structural properties of optimal solution
– Assumption: is jointly concave and supermodular* in
.
U(x, h) = Ea,h′|hV (x+ a, h′)
V (x, h) = maxy{u(x, h, y) + αU(x− y, h)}
Foresighted decision
Expectation over dynamics
is monotonic in xπ(x, h) is concave in xU(x, h)
u(x, h, y) (x, y)
How can we utilize these structural properties in online learning?
* u(x′, h, y′)− u(x′, h, y) ≥ u(x, h, y′)− u(x, h, y) if x′ ≥ x, y′ ≥ y
![Page 18: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/18.jpg)
18
For each channel state h, we approximate the post-decision
state-value function such that (threshold).
Piece-wise linear approximation
• How to compactly represent post-decision state-value function and
monotonic policy?
x1 x2 x4x3
1δ
2δ
1δ
3δ
2δ
1, ,min max in i n
δ δ=
≤⋯
U(x, h)π(x, h)
x1 x2 x4x3AδAdaptive approximation operator ( )
![Page 19: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/19.jpg)
19
Online learning with adaptive
approximation
δ
Ut(x, ht−1)
1
δε
α=−
Vt(x, ht) = maxy∈Y
{u(x, ht, y) + αUt−1(x− y, ht)}
Ut(x, ht−1) = A{(1− βt)Ut−1(x, ht−1) + βtVt(x, ht)}
Foresighted decision
Online update
π(x, ht)
Theorem: Online learning with adaptive approximation
converges to an ε-optimal solution, where
Variant: Update and every T time slotsU(x, h) π(x, h)
Ut−1(x, h)
![Page 20: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/20.jpg)
20
Performance of learning with
approximation
#channel state=8Rayleigh fading channel
α = 0.95Average channel gain h2
σ2= 0.14
![Page 21: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/21.jpg)
21
Comparison with stability-constrained
optimization
• Stability-constrained optimization [Neely, 2006]
– Minimize the trade-off between Lyapunov drift and energy consumption
– Do not consider the effect of the utility function on post-decision state value
function
– Do not consider the time-correlation of the channel states
– Only ensure queue stability, but result in poor delay performance
Lyapunov drift
minλρ(ht, yt) + (xt − yt)2 − x2t
Post-decision state value functonUtility function
maxyt∈Y
−(xt − yt + λρ(ht, yt)) + xt − yt − (xt − yt)2 + x2t
u(xt, ht, yt) U(xt − yt, ht)
![Page 22: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/22.jpg)
22
Comparison to stability-constrained
optimization
Stability constrained optimization
• Minimize Lyapunov drift ≠Minimize delay
Our proposed solution
• Minimize queue size = Minimize delay
Channel: Markov chainChannel: long-term correlation(generated MA model)
![Page 23: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/23.jpg)
23
Comparison to Q-learning
• Markovian Rayleigh fading channel
• Q-learning: update the state-value function one state at each time slot (learn over
50000 time slots)
• Online learning with adaptive approximation: T=10, learn over 5000 time slots
![Page 24: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/24.jpg)
24
Roadmap
• Separation principle 1 (improving learning efficiency)
– Post-decision state-based formulation
– Structure-aware online learning with adaptive approximation
• Separation principle 2 (Separating the foresighted decision for
heterogeneous media data transmission)
– Context-based state
– Priority-based scheduling
• Separation principle 3 (decomposing multi-user coupling )
– Multi-user Markov decision process formulation
– Post-decision state value function decomposition
![Page 25: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/25.jpg)
25
Heterogeneous media data
Data Unit (DU)
Media data representation:
• Each DU has the following attributes:
– Arrival time: time at which the DU is ready for processing:
– Delay deadline:
– Size : in packets
– Distortion impact: per packet
– Interdependency between DUs: expressed by Directed Acyclic Graph (DAG)
ti
di
qiLi
![Page 26: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/26.jpg)
26
Context
• Fixed GOP (i.e. group of DUs) structure
• Context ( ) at each time slot t
– Include the DUs whose deadlines are within a time window W
• Context transition is deterministic
DU 1
DU 2
DU 3
DU 4
DU 5
DU 1
DU 2
DU 3…………
t
DU 1
DU 2
DU 3
DU 2
DU 3
DU 4
DU 5
DU 4
DU 5
DU 4
DU 5
DU 1
e.g. W = 3
1 2 3 4
c1 c2 c3 c4
ct
d1 d2, d3 d4, d5
![Page 27: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/27.jpg)
27
Foresighted optimization
• Multi-DU Foresighted decision
– Which DU should be transmitted first?
– How much data should be transmitted for each DU?
DU 2
DU 3
DU 4
DU 5
ct
State: (ct,xt, ht)
Current utility Post-decision state-value function
maxyit,i∈ct
{∑
i∈ct
ui(xit, h
it, y
it) + αU(ct,xt − yt, ht)
}
DU 4
DU 5
ct+1
xt = (x2t , x3t , x
4t , x
5t )
![Page 28: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/28.jpg)
28
Priority-based scheduling
• Prioritization
– Based on distortion impacts, delay deadlines and dependencies
DU 2
DU 3
DU 4
DU 5
ct
DU 2
DU 3
DU 4
DU 5
Priority graph
![Page 29: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/29.jpg)
29
Separate foresighted decision across DUs
• Priority-based scheduling
– If there is only one DU with the highest priority, transmit the data in this DU
by solving the foresighted optimization;
– If there are multiple DUs that have same priorities, solve the foresighted
optimization for each DU, transmit the data from the DU with highest long-
term utility.
DU 2
DU 3
DU 4
DU 5
V it = maxyit∈Y(ht)
{ui(xit, ht,
∑
j⊳i
yj∗t , yit) + αUi(ct, x
it − y
it, ht)}
Single-DU foresighted decision:
Multi-DU foresighted decision →→→→ Multiple single-DU foresighted decision
One dimensional concave function
given and .ct ht
It can be updated using the proposed
online learning.
: DU j has higher priority than DU i.j ⊳ i
![Page 30: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/30.jpg)
30
Simulation results for single-user
transmission
Foreman Coastguard
Channel: Rayleigh fading, modeled as 8-state Markov chain
![Page 31: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/31.jpg)
31
Roadmap
• Separation principle 1 (improving learning efficiency)
– Post-decision state-based formulation
– Structure-aware online learning with adaptive approximation
• Separation principle 2 (Separating the foresighted decision for
heterogeneous media data transmission)
– Context-based state
– Priority-based scheduling
• Separation principle 3 (decomposing multi-user coupling )
– Multi-user Markov decision process formulation
– Post-decision state value function decomposition
![Page 32: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/32.jpg)
32
Delay-sensitive multi-access
communications
Transmitter
Transmitter h1t
x1t
xMt
hMt
y1t
yMt
a1t
aMt
…
s.t. [y1t , · · · , yMt ] ∈ Π(ht),∀t ≥ 0
Resource constraint (e.g. transmission
time constraint in TDMA)
maxyt,∀t
E
∞∑
t=0
αtM∑
i=1
ui(xit, h
it, y
it)
![Page 33: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/33.jpg)
33
Foresighted optimization formulation
• Formulate as Multi-user MDP (MUMDP) and perform foresighted decision
tCurrent allocation Future allocation
User i
i−User
Resource
allocation
Resource
allocation
Resource
allocation
,i it tx h
,i it tx h− −
Network coordinator
Time slot
1 1,i it tx h− −+ +
1 1,i it tx h+ +
Resource requirement information Resource allocation
V (xt,ht) = maxyt∈Π(ht)
{M∑
i=1
ui(xit, h
it, y
it) + αU(xt − yt,ht)}
Depends on all users state
Our goal: decouple the post-decision state value function across users
![Page 34: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/34.jpg)
34
Decomposition of post-decision state-
value function
• Relax the resource constraints (e.g. TDMA-like access)
• Introduce scalar resource price , and compute post-decision state-value
function individually based on single-user MDP.
• Upper bound
Current allocation Future allocation
User i
i−User
Resource
allocation
,i it tx h
,i it tx h− −
1 1,i it tx h+ +
1 1,i it tx h+ +
Resource price λ
Resource price λ
Network coordinator
Access time
λUλi (x
it, h
it)
⊂⊂⊂⊂
Resource
allocation
Resource
allocation
1 1,i it tx h− −+ +
Uλi (xi, hi) = max
0≤yi≤xi{ui(x
i, hi, yi)− λyi/R(hi) + αUλi (xi − yi, hi)}
∑M
i=1yik
R(hik)≤ 1,∀k = t+ 1, · · ·
∑∞k=t+1 α
k∑M
i=1yik
R(hik)≤ 1
1−α
![Page 35: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/35.jpg)
35
Resource allocation
• Post-decision state value function decomposition
• Resource allocation
– Gradient-based allocation
• Lower bound
t
Current allocation Future allocation
User i
i−User
Resource
allocation
,i it tx h 1 1,i i
t tx h+ +
,i it tx h− −
1 1,i it tx h+ +
Resource price λ
Resource price λ
Network coordinator
maxyt∈Π(ht)
M∑
i=1
{ui(xit, h
it, y
it) + αUλi (x
it − y
it, h
it)}
Gradient information
U(xt, ht) ≈∑M
i=1 Uλi (x
it, h
it)
![Page 36: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/36.jpg)
36
• Subgradient method to update resource price
λk+1 = [λk + βk(
M∑
i=1
Zi −1
1− α)]+
Resource price update
The resource price is updated by
subgradient
where is the expected consumed resource by
user i and is individually computed by user i.Zi
![Page 37: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/37.jpg)
37
Relationship of different solutions
![Page 38: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/38.jpg)
38
Simulation results for multi-user
transmission
1. Each user uses multiple queues to represent video data;
2. Markov chain model for Rayleigh fading channel
3. TDMA-type channel access
Users experienced with average
channel conditions of 28dB
Foreman
Coastguard
Mobile
Upper bound
Lower bound
![Page 39: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/39.jpg)
39
Other applications developed in our lab
• Cross-layer optimization via layer separation [Fu 2009, Zhang 2010]
– Each layer performs dynamic optimization individually
– Message exchange across layers
• Media-TCP [Shiang 2010]
– Context-based congestion control
• Dynamic voltage scaling for video decoding [Mastronarde 2009]
– Post-decision state-based formulation
– Context-based scheduling
• Wireless video network with cooperation [Mastronarde 2010]
– Structure-aware online learning
![Page 40: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/40.jpg)
40
Summary: separation principle 1
• Foresighted optimization framework
• Separation principle 1
– Post-decision state-based foresighted optimization formulation: separation between foresighted decision and dynamics
– Structure-aware online learning
• Low complexity, fast convergence and achieving ε-optimal solutions
Current time slot Next time slot
u(s, y) V (f(s, y, w))Ewmaxy{ }+
Current time slot Next time slot
maxyu(s, y) + U(g(s, y)) U(s) = EwV (g
′(s, w))Post-decision state
s = g(s, y)
![Page 41: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/41.jpg)
41
Summary: separation principle 2
• Foresighted optimization framework
• Separation principle 2
– Context-based state to capture heterogeneity in data units at each
time slot
– Priority graph-based scheduling: separation across data units
context
DU 1
DU 2
DU 3
DU 4
DU 5
maxyu(s,y) + U(g(s,y))
DU: data unit
![Page 42: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/42.jpg)
42
Summary: separation principle 3
• Foresighted optimization framework
• Separation principle 3
– Decomposition of post-decision state value function: separation
across users
si si
User i
i−User
Resource
constraint
Resource
constraint
Resource
constraint
s−i s−iCurrent state Post-decision state
maxy∈Π
M∑
i=1
ui(si, yi) + U(g(s,y))
![Page 43: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/43.jpg)
43
Future research
• Extend the unified framework to
– Multi-hop delay-sensitive data transmission
– Non-collaborative multi-user data transmission
– Energy-efficient parallel data processing in media systems
![Page 44: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/44.jpg)
44
Related Journal Publications[Fu10a] Fangwen Fu, Mihaela van der Schaar, “Structural solutions for cross-layer optimization of wireless
multimedia transmission,” In submission.
[Fu10b] Fangwen Fu, Mihaela van der Schaar, “Structure-aware stochastic control for transmission scheduling”in submission.
[Fu10c] Fangwen Fu, Mihaela van der Schaar, “A Systematic Framework for Dynamically Optimizing Multi-User
Video Transmission,” IEEE J. Sel. Areas Commun., vol. 28, no. 3, pp. 308-320, Apr. 2010.
[Fu10d] Fangwen Fu, Mihaela van der Schaar, “Decomposition Principles and Online Learning in Cross-Layer Optimization for Delay-Sensitive Applications”, IEEE Trans. Signal Process., vol 58, no. 3, pp. 1401-1415, Feb. 2010.
[Fu09a] Mihaela van der Schaar and Fangwen Fu, "Spectrum Access Games and Strategic Learning in Cognitive Radio Networks for Delay-Critical Applications," Proc. of IEEE, Special issue on Cognitive Radio, vol. 97, no. 4, pp. 720-740, Apr. 2009.
[Fu09b] Yu Zhang, Fangwen Fu, Mihaela van der Schaar, “On-line Learning and Optimization for Wireless Video Transmission,” IEEE Transactions on Signal Processing, accepted, 2009.
[Fu09c] Fangwen Fu, Mihaela van der Schaar, "A New Systematic Framework for Autonomous Cross-Layer Optimization," IEEE Trans. Veh. Tech., vol. 58, no. 4, pp. 1887-1903, May, 2009.
[Fu09d] Fangwen Fu, Mihaela van der Schaar, "Learning to Compete for Resources in Wireless Stochastic Games," IEEE Trans. Veh. Tech., vol. 58, no. 4, pp. 1904-1919, May 2009.
![Page 45: A Unified Framework for Delay-Sensitive Communications€¦ · 2 Motivation Sensor networks Wireless video phone • Delay sensitive multimedia applications are booming over a variety](https://reader036.vdocument.in/reader036/viewer/2022071015/5fce25d8b3b59157fe1f00bb/html5/thumbnails/45.jpg)
45
Acknowledgements
• PhD committee: Professor Mihaela van der Schaar, Lixia Zhang, Jason
Speyer, Lieven Vandenberghe, and Gregory J. Pottie
• Labmates: Brian Foo, Hyunggon Park, Nick Mastronarde, Brian Foo, Xiaolin
Tong, Yi Su, Yu Zhang, Shaolei Ren, Jaeok Park, Khoa Tran Phan, Zhichu Lin,
and Yuanzhang Xiao
• Intern mentors: Dr. Deepak Turaga, Dr. Olivier Verscheure, and Dr. Ulas
Kozat
• Collaborators: Dr. Tudor Stoenescu, Dr. Ulrich Berthold, Dr. Ahmad Fattahi
• Family: my wife, parents, sister and brother