motivationdata compressionchannel codinganomalous dispersion finite blocklength...
TRANSCRIPT
![Page 1: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/1.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Finite blocklength IT: tutorial overview
Plan:
1. Motivation
2. Data compression (lossless)
3. Channel coding
3.1 Overview on the example of BSC3.2 Converse bounds3.3 Achievability bounds3.4 Channel dispersion3.5 Discussion: ARQ, real codes
4. Anomalous dispersion
4.1 Channels with feedback4.2 Quasi-static channels4.3 Energy-per-bit without CSI at receiver
![Page 2: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/2.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Voice communication
Tolerable delay: 50-200 ms−−−−−−−−−−−−−−−−−−−−−→
I Packetization: 20-30 ms=⇒ payload size ∼ 1500 bit (64 kbps, PCM)
I Skype: 1300 bit (avg.)GSM: 260 bit (uncoded), 456 bit (coded).
![Page 3: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/3.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Internet traffic: packet sizes
0 2000 4000 6000 8000 10000 120000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Packet size, bit
Fra
ctio
n
~10 bit4
500−1000 bit(mostly outgoing)
![Page 4: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/4.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Intra-PC communication
![Page 5: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/5.jpg)
Motivation Data compression Channel coding Anomalous dispersion
how many bits do we need to compress a
140-charactertwitter message?
![Page 6: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/6.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Finite sample size problems lack theoretical
simplicity but not practical applications.
T. Cover and M. Hellman
![Page 7: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/7.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Shannon, 1953
![Page 8: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/8.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Case study: 1000-bit BSC
I Consider channel BSC (n = 1000, δ = 0.11)
I How many data bits can we transmit with (block) Pe ≤ 10−3?
I Attempt 1: Repetition
k = 47 bits via [21,1,21]-code
I Attempt 2: Reed-Muller
k = 112 bits via [64,7,32]-code
I Shannon’s prediction: C = 0.5 bit so
k ≈ 500 bit
I Finite blocklength IT:
414 ≤ k ≤ 416
![Page 9: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/9.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Case study: 1000-bit BSC
I Consider channel BSC (n = 1000, δ = 0.11)
I How many data bits can we transmit with (block) Pe ≤ 10−3?
I Attempt 1: Repetition
k = 47 bits via [21,1,21]-code
I Attempt 2: Reed-Muller
k = 112 bits via [64,7,32]-code
I Shannon’s prediction: C = 0.5 bit so
k ≈ 500 bit
I Finite blocklength IT:
414 ≤ k ≤ 416
![Page 10: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/10.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Lossless data compression
![Page 11: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/11.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Compression: Setup
I Given discrete r.v. X ∼ PX , find lossless f : X → {0, 1}∗ s.t.
`(f (X ))→ min
Here: `(binary string)4= length
I Rename values of X : PX (1) ≥ PX (2) ≥ · · ·I The best compressor (for E , median, etc)
f ∗(1) = ∅f ∗(2) = 0
f ∗(3) = 1
f ∗(4) = 01
· · ·
I Analysis of E [`(f ∗(X ))]: cumbersome!
![Page 12: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/12.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Compression: Results
I Bounds on L∗ = `(f ∗(X ))I Distribution:
P[
log1
PX (X )≤ k
]≤ P[L∗ ≤ k] ≤ P
[log
1
PX (X )≤ k + τ
]+2−τ
I Expected length: [Alon-Orlitsky]
H(X )− log2(e(H(X ) + 1)) ≤ E [L∗] ≤ H(X )
I WLLN: When X = (A1, . . . ,An) – iid (or ergodic!) vector
1
nL∗ ≈ H(A)
I CLT: When X = (A1, . . . ,An) – iid (or weakly dependent!)vector
1
nL∗ ≈ H(A) +
√V (A)
nZ , Z ∼ N (0, 1)
where V (A) = Var[logPX (X )] – varentropy.
![Page 13: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/13.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Behavior of R∗(n, ε) – the ε-quantile of compression length24
0 500 1000 1500 20000.50
0.51
0.52
0.53
0.54
0.55
0.56
rate (bits)
n
normal approximation
Fig. 2: The optimum rate R⇤(n, 0.1), the Gaussian approximation R⇤(n, 0.1) in (141), and the upper bound Ru(n, 0.1),for a Bernoulli-0.11 source and blocklengths 200 n 2000.
Figure 3 focuses the comparison between R⇤(n, 0.1) and R⇤(n, 0.1) on the short blocklengthrange up to 200 not shown in Figure 2. For n > 60, the discrepancy between the two neverexceeds 4%.
The remainder of the section is devoted to justifying the use of (141) as an accurateapproximation to R⇤(n, ✏). To that end, in Theorems 17 and 16 we establish the boundsgiven in (137). Their derivation requires that we overcome two technical hurdles:
1) The distribution function of the optimal encoding length is not the same as the distri-bution of 1
n
Pni=1 ıX(Xi);
2) The distribution of 1n
Pni=1 ıX(Xi) is only approximately Gaussian.
To cope with the second hurdle we will appeal to the classical Berry-Esseen bound [19],[26]:
Theorem 15: Let {Zi} be independent and identically distributed random variables withzero mean and unit variance, and let Z be standard normal. Then, for all n � 1 and any a:
�����P"
1pn
nX
i=1
Zi a
#� P
⇥Z a
⇤�����
E[|Z1 � E[Z1]|3]2p
n. (142)
Invoking the Berry-Esseen bound, Strassen [32] claimed the following approximation for
biased coin flips: entropy = 0.5
![Page 14: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/14.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Takeaway: `(f ∗(X )) ≈ log 1PX (X )
Channel coding: (next two hours)I Information density i(X ;Y ) takes the role of log 1
PX (X )I Optimal transmission rate
R∗(n, ε) ≈ ε-quantile of1
ni(X n;Y n)
I iid/ergodic asymptotics:
R∗(n, ε) → C = maxX
I (X ;Y )
![Page 15: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/15.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Takeaway: `(f ∗(X )) ≈ log 1PX (X )
Channel coding: (next two hours)I Information density i(X ;Y ) takes the role of log 1
PX (X )I Optimal transmission rate
R∗(n, ε) ≈ ε-quantile of1
ni(X n;Y n)
I iid/ergodic asymptotics:
R∗(n, ε) → C = maxX
I (X ;Y )
![Page 16: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/16.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: intro
![Page 17: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/17.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Abstract communication problem
Noisy channel
Goal: Decrease corruption of data caused by noise
![Page 18: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/18.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: principles
Noisy channel
Data bits Redundancy
Goal: Decrease corruption of data caused by noise
Solution: Code to diminish probability of error Pe .
Key metrics: Rate and Pe
![Page 19: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/19.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: principles
Possible
Impossible
Data bits Redundancy
Pe Reliability−Rate tradeoff
Rate
Noisy channel
![Page 20: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/20.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: principles
Data bits Redundancy
Pe Reliability−Rate tradeoff
Rate
Noisy channelDecreasing Pe further:
1. More redundancyBad: loses rate
2. Increase blocklength!
n = 10
![Page 21: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/21.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: principles
Data bits Redundancy
Pe Reliability−Rate tradeoff
Rate
Noisy channelDecreasing Pe further:
1. More redundancyBad: loses rate
2. Increase blocklength!
n = 100
![Page 22: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/22.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: principles
Noisy channel
Data bits Redundancy
Pe Reliability−Rate tradeoff
Rate
Decreasing Pe further:
1. More redundancyBad: loses rate
2. Increase blocklength!
n = 1000
![Page 23: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/23.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: principles
Noisy channel
Data bits Redundancy
Pe Reliability−Rate tradeoff
Rate
Decreasing Pe further:
1. More redundancyBad: loses rate
2. Increase blocklength!
n = 106
![Page 24: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/24.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: Shannon capacity
Noisy channel
Data bits Redundancy
Pe Reliability−Rate tradeoff
C
Channel capacity
Rate
Shannon: Fix R < CPe ↘ 0 as n→∞
![Page 25: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/25.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: Shannon capacity
Noisy channel
Data bits Redundancy
Pe Reliability−Rate tradeoff
C Rate
Channel capacity
Shannon: Fix R < CPe ↘ 0 as n→∞
Question:For what n will Pe < 10−3?
![Page 26: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/26.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: Gaussian approximation
Noisy channel
Data bits Redundancy
Pe Reliability−Rate tradeoff
C Rate
Channel capacity
Channel dispersion
Shannon: Fix R < CPe ↘ 0 as n→∞
Question:For what n will Pe < 10−3?
Answer:
n & const · VC 2
![Page 27: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/27.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: Gaussian approximation
Noisy channel
Data bits Redundancy
Pe Reliability−Rate tradeoff
C Rate
Channel capacity
Channel dispersion
Shannon: Fix R < CPe ↘ 0 as n→∞
Question:For what n will Pe < 10−3?
Answer:
n & const · VC 2
![Page 28: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/28.jpg)
Motivation Data compression Channel coding Anomalous dispersion
How to describe evolution of the boundary?
Pe Reliability−Rate tradeoff
C Rate
Classical results:
I Vertical asymptotics: fixed rate, reliability functionElias, Dobrushin, Fano, Shannon-Gallager-Berlekamp
I Horizontal asymptotics: fixed ε, strong converse,√n terms
Wolfowitz, Weiss, Dobrushin, Strassen, Kemperman
![Page 29: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/29.jpg)
Motivation Data compression Channel coding Anomalous dispersion
How to describe evolution of the boundary?
Pe Reliability−Rate tradeoff
C Rate
XXI century:
I Tight non-asymptotic bounds
I Remarkable precision of normal approximation
I Extended results on horizontal asymptoticsAWGN, O(log n), cost constraints, feedback, etc.
![Page 30: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/30.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Finite blocklength fundamental limit
Pe Reliability−Rate tradeoff
C Rate
Definition
R∗(n, ε) = max
{1
nlogM : ∃(n,M, ε)-code
}
(max. achievable rate for blocklength n and prob. of error ε)
Note: Exact value unknown (search is doubly exponential in n).
![Page 31: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/31.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Minimal delay and error-exponents
Fix R < C . What is the smallest blocklength n∗ neededto achieve
R∗(n, ε) ≥ R ?
Classical answer: Approximation via reliability function[Shannon-Gallager-Berlekamp’67]:
n∗ ≈ 1
E (R)log
1
ε
E.g., take BSC (0.11) and R = 0.9C , prob. of error ε = 10−3:
n∗ ≈ 5000 (channel uses)
Difficulty: How to verify accuracy of this estimate?
![Page 32: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/32.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Minimal delay and error-exponents
Fix R < C . What is the smallest blocklength n∗ neededto achieve
R∗(n, ε) ≥ R ?
Classical answer: Approximation via reliability function[Shannon-Gallager-Berlekamp’67]:
n∗ ≈ 1
E (R)log
1
ε
E.g., take BSC (0.11) and R = 0.9C , prob. of error ε = 10−3:
n∗ ≈ 5000 (channel uses)
Difficulty: How to verify accuracy of this estimate?
![Page 33: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/33.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Bounds
I Bounds are implicit in Shannon’s theorem
limn→∞
R∗(n, ε) = C ⇐⇒{
R∗(n, ε) ≤ C + o(1) ,R∗(n, ε) ≥ C + o(1) .
(Feinstein’54, Shannon’57, Wolfowitz’57, Fano)
I Reliability function: even better bounds(Elias’55, Shannon’59, Gallager’65, SGB’67)
I Problems: derived for asymptotics (need “de-asymptotization”)unexpected sensitivity:
ε ≤ e−nEr (R) [Gallager’65]
ε ≤ e−nEr (R−o(1))+O(log n) [Csiszar-Korner’81]
![Page 34: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/34.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Bounds
I Bounds are implicit in Shannon’s theorem
limn→∞
R∗(n, ε) = C ⇐⇒{
R∗(n, ε) ≤ C + o(1) ,R∗(n, ε) ≥ C + o(1) .
(Feinstein’54, Shannon’57, Wolfowitz’57, Fano)
I Reliability function: even better bounds(Elias’55, Shannon’59, Gallager’65, SGB’67)
I Problems: derived for asymptotics (need “de-asymptotization”)unexpected sensitivity:
ε ≤ e−nEr (R) [Gallager’65]
ε ≤ e−nEr (R−o(1))+O(log n) [Csiszar-Korner’81]
For BSC(n = 103, 0.11): o(1) ≈ 0.1, eO(log n) ≈ 1024 (!)
![Page 35: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/35.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Bounds
I Bounds are implicit in Shannon’s theorem
limn→∞
R∗(n, ε) = C ⇐⇒{
R∗(n, ε) ≤ C + o(1) ,R∗(n, ε) ≥ C + o(1) .
(Feinstein’54, Shannon’57, Wolfowitz’57, Fano)
I Reliability function: even better bounds(Elias’55, Shannon’59, Gallager’65, SGB’67)
I Problems: derived for asymptotics (need “de-asymptotization”)unexpected sensitivity:
Strassen’62: Take n > 19600ε16 . . . (!)
I Solution: Derive bounds from scratch.
![Page 36: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/36.jpg)
Motivation Data compression Channel coding Anomalous dispersion
New achievability bound
0 s -ZZZZZ~
1− δ s01 s -�
����>
1− δs1δ
BSC
Theorem (RCU)
For a BSC (δ) there exists a code with rate R, blocklength n and
ε ≤n∑
t=0
(n
t
)δt(1− δ)n−t min
{1,
t∑
k=0
(n
k
)2−n+nR
}.
![Page 37: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/37.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Proof of RCU bound for the BSC
Hamming Space Fn2
I Input space: A = {0, 1}n
I Let c1, . . . , cM ∼ Bern( 12 )n
(random codebook)
I Transmit c1
I Noise displaces c1→Y
Y = c1 + Z , Z ∼ Bern(δ)n
I Decoder: find closest codeword to Y
![Page 38: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/38.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Proof of RCU bound for the BSC
Hamming Space Fn2
c1
c2
cM
c5
c3
c4
I Input space: A = {0, 1}n
I Let c1, . . . , cM ∼ Bern( 12 )n
(random codebook)
I Transmit c1
I Noise displaces c1→Y
Y = c1 + Z , Z ∼ Bern(δ)n
I Decoder: find closest codeword to Y
![Page 39: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/39.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Proof of RCU bound for the BSC
Hamming Space Fn2
c1
c2
cM
c5
c3
c4
I Input space: A = {0, 1}n
I Let c1, . . . , cM ∼ Bern( 12 )n
(random codebook)
I Transmit c1
I Noise displaces c1→Y
Y = c1 + Z , Z ∼ Bern(δ)n
I Decoder: find closest codeword to Y
![Page 40: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/40.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Proof of RCU bound for the BSC
Hamming Space Fn2
c1
c2
cM
c5
Y
c3
c4
I Input space: A = {0, 1}n
I Let c1, . . . , cM ∼ Bern( 12 )n
(random codebook)
I Transmit c1
I Noise displaces c1→Y
Y = c1 + Z , Z ∼ Bern(δ)n
I Decoder: find closest codeword to Y
![Page 41: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/41.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Proof of RCU bound for the BSC
Hamming Space Fn2
c1
c2
cM
c5
Y
c3
c4
I Input space: A = {0, 1}n
I Let c1, . . . , cM ∼ Bern( 12 )n
(random codebook)
I Transmit c1
I Noise displaces c1→Y
Y = c1 + Z , Z ∼ Bern(δ)n
I Decoder: find closest codeword to Y
![Page 42: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/42.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Proof of RCU bound for the BSC
Hamming Space Fn2
c1
c2
cM
c5
Y
c3
c4Success!
I Input space: A = {0, 1}n
I Let c1, . . . , cM ∼ Bern( 12 )n
(random codebook)
I Transmit c1
I Noise displaces c1→Y
Y = c1 + Z , Z ∼ Bern(δ)n
I Decoder: find closest codeword to Y
![Page 43: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/43.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Proof of RCU bound for the BSC
Hamming Space Fn2
c1
c2
cM
c5
c3
c4
I Input space: A = {0, 1}n
I Let c1, . . . , cM ∼ Bern( 12 )n
(random codebook)
I Transmit c1
I Noise displaces c1→Y
Y = c1 + Z , Z ∼ Bern(δ)n
I Decoder: find closest codeword to YI Probability of error analysis:
P[error|Y ,wt(Z ) = t] = P[∃j > 1 : cj ∈ Ball(Y , t)]
≤∑M
j=2P[cj ∈ Ball(Y , t)]
≤ 2nR∑t
k=0
(n
k
)2−n
![Page 44: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/44.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Proof of RCU bound for the BSC
Hamming Space Fn2
c1
c2
cM
c5
Y
c3
c4
I Input space: A = {0, 1}n
I Let c1, . . . , cM ∼ Bern( 12 )n
(random codebook)
I Transmit c1
I Noise displaces c1→Y
Y = c1 + Z , Z ∼ Bern(δ)n
I Decoder: find closest codeword to YI Probability of error analysis:
P[error|Y ,wt(Z ) = t] = P[∃j > 1 : cj ∈ Ball(Y , t)]
≤∑M
j=2P[cj ∈ Ball(Y , t)]
≤ 2nR∑t
k=0
(n
k
)2−n
![Page 45: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/45.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Proof of RCU bound for the BSC
Hamming Space Fn2
c1
c2
cM
c5
Y
c3
c4
I Input space: A = {0, 1}n
I Let c1, . . . , cM ∼ Bern( 12 )n
(random codebook)
I Transmit c1
I Noise displaces c1→Y
Y = c1 + Z , Z ∼ Bern(δ)n
I Decoder: find closest codeword to Y
I Probability of error analysis:
P[error|Y ,wt(Z ) = t] = P[∃j > 1 : cj ∈ Ball(Y , t)]
≤∑M
j=2P[cj ∈ Ball(Y , t)]
≤ 2nR∑t
k=0
(n
k
)2−n
![Page 46: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/46.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Proof of RCU bound for the BSC
Hamming Space Fn2
c1
c2
cM
c5
Y
c3
c4
Error!I Input space: A = {0, 1}n
I Let c1, . . . , cM ∼ Bern( 12 )n
(random codebook)
I Transmit c1
I Noise displaces c1→Y
Y = c1 + Z , Z ∼ Bern(δ)n
I Decoder: find closest codeword to Y
I Probability of error analysis:
P[error|Y ,wt(Z ) = t] = P[∃j > 1 : cj ∈ Ball(Y , t)]
≤∑M
j=2P[cj ∈ Ball(Y , t)]
≤ 2nR∑t
k=0
(n
k
)2−n
![Page 47: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/47.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Proof of RCU bound for the BSC
Hamming Space Fn2
c1
c2
cM
c5
Y
c3
c4
Error!I Input space: A = {0, 1}n
I Let c1, . . . , cM ∼ Bern( 12 )n
(random codebook)
I Transmit c1
I Noise displaces c1→Y
Y = c1 + Z , Z ∼ Bern(δ)n
I Decoder: find closest codeword to YI Probability of error analysis:
P[error|Y ,wt(Z ) = t] = P[∃j > 1 : cj ∈ Ball(Y , t)]
≤∑M
j=2P[cj ∈ Ball(Y , t)]
≤ 2nR∑t
k=0
(n
k
)2−n
![Page 48: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/48.jpg)
Motivation Data compression Channel coding Anomalous dispersion
. . . cont’d . . .
I Conditional probability of error:
P[error|Y ,wt(Z ) = t] ≤t∑
k=0
(n
k
)2−n+nR
I Key observation: For large noise t RHS is > 1. Tighten:
P[error|Y ,wt(Z ) = t] ≤ min
{1,
t∑
k=0
(n
k
)2−n+nR
}(∗)
I Average wt(Z ) ∼ Binomial(n, δ) =⇒ Q.E.D.
Note: Step (∗) tightens Gallager’s ρ-trick:
P
⋃
j
Aj
≤
∑
j
P[Aj ]
ρ
![Page 49: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/49.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Sphere-packing converse (BSC variation)
Theorem (Elias’55)
For any (n,M, ε) code over the BSC (δ):
ε ≥ f
(2n
M
),
where f (·) is a piecewise-linear decreasing convex function:
f
t∑
j=0
(n
j
) =
n∑
j=t+1
(n
j
)δj(1− δ)n−j t = 0, . . . , n
Note: Convexity of f follows from general properties of βα (below)
![Page 50: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/50.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Sphere-packing converse (BSC variation)
Hamming Space Fn2
D1D2
DM
Dj
Proof:I Denote decoding regions Dj :∐
Dj = {0, 1}n
I Probability of error is:
ε = 1M
∑j P[cj + Z 6∈ Dj ]
≥ 1M
∑j P[Z 6∈ Bj ]
I Bj = ball centered at 0 s.t. Vol(Bj) = Vol(Dj)
I Simple calculation:
P[Z 6∈ Bj ] = f (Vol(Bj))
I f – convex, apply Jensen:
ε ≥ f
1
M
M∑
j=1
Vol(Dj)
= f
(2n
M
)
![Page 51: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/51.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Sphere-packing converse (BSC variation)
Hamming Space Fn2
D1D2
DM
Dj
Bj
Proof:I Denote decoding regions Dj :∐
Dj = {0, 1}n
I Probability of error is:
ε = 1M
∑j P[cj + Z 6∈ Dj ]
≥ 1M
∑j P[Z 6∈ Bj ]
I Bj = ball centered at 0 s.t. Vol(Bj) = Vol(Dj)
I Simple calculation:
P[Z 6∈ Bj ] = f (Vol(Bj))
I f – convex, apply Jensen:
ε ≥ f
1
M
M∑
j=1
Vol(Dj)
= f
(2n
M
)
![Page 52: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/52.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Sphere-packing converse (BSC variation)
Hamming Space Fn2
D1D2
DM
Dj
Bj
Proof:I Denote decoding regions Dj :∐
Dj = {0, 1}n
I Probability of error is:
ε = 1M
∑j P[cj + Z 6∈ Dj ]
≥ 1M
∑j P[Z 6∈ Bj ]
I Bj = ball centered at 0 s.t. Vol(Bj) = Vol(Dj)
I Simple calculation:
P[Z 6∈ Bj ] = f (Vol(Bj))
I f – convex, apply Jensen:
ε ≥ f
1
M
M∑
j=1
Vol(Dj)
= f
(2n
M
)
![Page 53: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/53.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Bounds: example BSC (0.11), ε = 10−3
0 500 1000 1500 20000
0.1
0.2
0.3
0.4
0.5
Blocklength, n
Rat
e, b
it/ch
.use
Feinstein−Shannon
Gallager
RCU
Converse
Capacity
![Page 54: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/54.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Bounds: example BSC (0.11), ε = 10−3
0 500 1000 1500 20000
0.1
0.2
0.3
0.4
0.5
Blocklength, n
Rat
e, b
it/ch
.use
RCU
Converse
Capacity
Delay required to achieve 90% of C :
2985 ≤ n∗ ≤ 3106
Error-exponent: n∗ ≈ 5000
![Page 55: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/55.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Bounds: example BSC (0.11), ε = 10−3
0 500 1000 1500 20000
0.1
0.2
0.3
0.4
0.5
Blocklength, n
Rat
e, b
it/ch
.use
RCU
Converse
Capacity
Delay required to achieve 90% of C :
2985 ≤ n∗ ≤ 3106
Error-exponent: n∗ ≈ 5000
![Page 56: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/56.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Normal approximation
Theorem
For the BSC (δ) and 0 < ε < 1,
R∗(n, ε) = C −√
V
nQ−1(ε) +
1
2
log n
n+ O
(1
n
)
where
C (δ) = log 2 + δ log δ + (1− δ) log(1− δ)
V = δ(1− δ) log2 1− δδ
Q(x) =
∫ ∞
x
1√2π
e−y2
2 dy
Proof: Bounds + Stirling’s formulaNote: Now we see the explicit dependence on ε!
![Page 57: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/57.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Normal approximation: BSC (0.11); ε = 10−3
0 500 1000 1500 20000
0.1
0.2
0.3
0.4
0.5
Blocklength, n
Rat
e, b
it/ch
.use
Capacity
Non−asymptotic bounds
Normal approximationC −
√Vn Q−1(ε) + 1
2log nn
To achieve 90% of C :
n∗ ≈ 3150
![Page 58: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/58.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Normal approximation: BSC (0.11); ε = 10−3
0 500 1000 1500 20000
0.1
0.2
0.3
0.4
0.5
Blocklength, n
Rat
e, b
it/ch
.use
Capacity
Non−asymptotic bounds
Normal approximationC −
√Vn Q−1(ε) + 1
2log nn
To achieve 90% of C :
n∗ ≈ 3150
![Page 59: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/59.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Dispersion and minimal required delay
Delay needed to achieve R = ηC :
n∗ &(Q−1(ε)
1− η
)2
· VC 2
Note: VC2 is “coding horizon”.
Behavior of VC2 (BSC)
0 0.1 0.2 0.3 0.4 0.5
100
102
104
106
δ
−−−−−−−−−−−−−−−−−−−→Less noise More noise
![Page 60: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/60.jpg)
Motivation Data compression Channel coding Anomalous dispersion
BSC summary
Delay required to achieve 90 % of capacity:
I Error-exponents:n∗ ≈ 5000
I True value:2985 ≤ n∗ ≤ 3106
I Channel dispersion:n∗ ≈ 3150
![Page 61: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/61.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: converse bounds
![Page 62: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/62.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Notation
I Take a random transformation APY |X−→ B
(think A = An, B = Bn, PY |X = PY n|X n)
I Input distribution PX induces PY = PY |X ◦ PX
PXY = PXPY |XI Fix code:
Wencoder−→ X → Y
decoder−→ W
W ∼ Unif [M] and M = # of codewordsInput distribution PX associated to a code:
PX [·] 4= # of codewords ∈ (·)M
.
I Goal: Upper bounds on logM in terms of ε4= P[error ]
As by-product: R∗(n, ε) . C −√
Vn Q−1(ε)
![Page 63: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/63.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Fano’s inequality
Theorem (Fano)
For any code
encoder decoder
W WYX
PY |X
with W ∼ Unif{1, . . . ,M}:
logM ≤ supPXI (X ;Y ) + h(ε)
1− ε , ε = P[W 6= W ]
Implies weak converse:
R∗(n, ε) ≤ C
1− ε + o(1) .
Proof: ε-small =⇒ H(W |W )-small =⇒ I (X ;Y ) ≈ H(W )= logM
![Page 64: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/64.jpg)
Motivation Data compression Channel coding Anomalous dispersion
A (very long) proof of Fano via channel substitution
Consider two distributions on (W ,X ,Y , W ):
P : PWXYW = PW × PX |W × PY |X ×PW |YDAG: W → X → Y → W
Q : QWXYW = PW × PX |W × QY ×PW |YDAG: W → X Y → W
Under Q the channel is useless:
Q[W = W ] =M∑
m=1
PW (m)QW (m) =1
M
M∑
m=1
QW (m) =1
M
Next step: data-processing for relative entropy D(·||·)
![Page 65: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/65.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Data-processing for D(·||·)
TransformationPB|A
PA
Input distribution Output distribution
QA
PB
QB
(Random)
D(PA‖QA) ≥ D(PB‖QB)
Apply to transform: (W ,X ,Y , W ) 7→ 1{W 6= W }:
D(PWXYW ‖QWXYW ) ≥ d(P[W = W ] ‖Q[W = W ] )
= d(1− ε|| 1M )
where d(x ||y) = x log xy + (1− x) log 1−x
1−y .
![Page 66: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/66.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Data-processing for D(·||·)
TransformationPB|A
PA
Input distribution Output distribution
QA
PB
QB
(Random)
D(PA‖QA) ≥ D(PB‖QB)
Apply to transform: (W ,X ,Y , W ) 7→ 1{W 6= W }:
D(PWXYW ‖QWXYW ) ≥ d(P[W = W ] ‖Q[W = W ] )
= d(1− ε|| 1M )
where d(x ||y) = x log xy + (1− x) log 1−x
1−y .
![Page 67: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/67.jpg)
Motivation Data compression Channel coding Anomalous dispersion
A proof of Fano via channel substitution
So far:D(PWXYW ‖QWXYW ) ≥ d(1− ε|| 1
M )
Lower-bound RHS:
d(1− ε‖ 1M ) ≥ (1− ε) logM − h(ε)
Analyze LHS:
D(PWXYW ‖QWXYW ) = D(PXY ‖QXY )
= D(PXPY |X‖PXQY )
= D(PY |X‖QY |PX )
(Recall: D(PY |X‖QY |PX ) = E x∼PX[D(PY |X=x‖QY )])
![Page 68: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/68.jpg)
Motivation Data compression Channel coding Anomalous dispersion
A proof of Fano via channel substitution: last step
Putting it all together:
(1− ε) logM ≤ D(PY |X ||QY |PX ) + h(ε) ∀QY ∀code
Two methods:
1. Compute supPXinfQY
and recall
infQY
D(PY |X‖QY |PX ) = I (X ;Y )
2. Take QY = P∗Y = the caod (capacity achieving output dist.)and recall
D(PY |X‖P∗Y |PX ) ≤ supPX
I (X ;Y ) ∀PX
Conclude:(1− ε) logM ≤ sup
PX
I (X ;Y ) + h(ε)
Important: Second method is particularly useful for FBL!
![Page 69: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/69.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Tightening: from D(·||·) to βα(·, ·)Question: How about replacing D(·||·) with other divergences?
Answer:
D(·||·) relative entropy(KL divergence) weak converse
Dλ(·||·) Renyi divergence strong converse
βα(·, ·) Neyman-PearsonROC curve
FBL bounds
Next: What is βα?
![Page 70: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/70.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Neyman-Pearson’s βα
Definition
For every pair of measures P,Q
βα(P,Q)4= inf
E :P[E ]≥αQ[E ] .
iterate overall “sets” E
plot pairs (P[E ],Q[E ])=⇒
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
β=
Q[E
]
α = P [E]
βα(P,Q)
Important: Like relative entropy βα satisfies data-processing.
![Page 71: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/71.jpg)
Motivation Data compression Channel coding Anomalous dispersion
βα = binary hypothesis testing
Tester?
PY
QY
Y1, Y2 , . . .
Two types of errors:
P[Tester says “QY ”] ≤ ε
Q[Tester says “PY ”] → min
Hence: Solve binary HT ⇐⇒ compute βα(PnY ,Q
nY )
Stein’s Lemma: For many i.i.d. observations
log β1−ε(PnY ,Q
nY ) = −nD(PY ||QY ) + o(n) .
But in fact log βα(PnY ,Q
nY ) can also be computed exactly!
![Page 72: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/72.jpg)
Motivation Data compression Channel coding Anomalous dispersion
How to compute βα?
Theorem (Neyman-Pearson)
βα is given parametrically by −∞ ≤ γ ≤ +∞:
P[
logP(X )
Q(X )≥ γ
]= α
Q[
logP(X )
Q(X )≥ γ
]= βα(P,Q)
For product measures log Pn(X )Qn(X ) = sum of i.i.d. =⇒ from CLT:
log βα(Pn,Qn) = −nD(P||Q) +√nV (P||Q)Q−1(α) + o(
√n) ,
where
V (P||Q) = VarP
[log
P(X )
Q(X )
]
![Page 73: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/73.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Back to proving converse
Recall two measures:
P : PWXYW = PW × PX |W × PY |X ×PW |YDAG: W → X → Y → W
P[W = W ] = 1−ε
Q : QWXYW = PW × PX |W × QY ×PW |YDAG: W → X Y → W
Q[W = W ] = 1M
Then by definition of βα:
β1−ε(PWXYW ,QWXYW ) ≤ 1
M
But logPWXYWQWXYW
= logPXPY |XPXQY
=⇒
logM ≤ − log β1−ε(PXPY |X ,PXQY ) ∀QY ∀code
![Page 74: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/74.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Meta-converse: minimax version
Theorem
Every (M, ε)-code for channel PY |X satisfies
logM ≤ − log
{infPX
supQY
β1−ε(PXPY |X ,PXQY )
}.
![Page 75: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/75.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Meta-converse: minimax version
Theorem
Every (M, ε)-code for channel PY |X satisfies
logM ≤ − log
{infPX
supQY
β1−ε(PXPY |X ,PXQY )
}.
I Finding good QY for every PX is not needed:
infPX
supQY
β1−ε(PXPY |X ,PXQY ) = supQY
infPX
β1−ε(PXPY |X ,PXQY ) (∗)
I Saddle-point property of βα is similar to D(·||·):
infPX
supQY
D(PXPY |X‖PXQY ) = supQY
infPX
D(PXPY |X‖PXQY ) = C
![Page 76: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/76.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Meta-converse: minimax version
Theorem
Every (M, ε)-code for channel PY |X satisfies
logM ≤ − log
{infPX
supQY
β1−ε(PXPY |X ,PXQY )
}.
Bound is tight in several senses:
I There exist non-signalling assisted (NSA) codes attaining theupper-bound. [Matthews, T-IT’2012]
I For any (M, ε)-code with ML decoder [Vazquez-Vilar et al’13]
logM = − log
{supQY
β1−ε(PXPY |X ,PXQY )
}
I ISIT’2016: almost tight for regular codes [Barman-Fawzi]
![Page 77: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/77.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Meta-converse: minimax version
Theorem
Every (M, ε)-code for channel PY |X satisfies
logM ≤ − log
{infPX
supQY
β1−ε(PXPY |X ,PXQY )
}.
In practice: evaluate with a luckily guessed (suboptimal) QY .How to guess good QY ?
I Try caod P∗YI Analyze channel symmetries
I Use geometric intuition. −→good QY ≈ “center” of PY |X
I Exercise: Redo BSC.
Space of distributions on Y
PY |X=a1
PY |X=aj
QY
PY |X=a2
![Page 78: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/78.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Example: Converse for AWGN
The AWGN Channel
Z∼ N (0, σ2)↓
X −→ ⊕ −→ Y
Codewords X n ∈ Rn satisfy power-constraint:
n∑
j=1
|Xj |2 ≤ nP
Goal: Upper-bound # of codewords decodable with Pe ≤ ε.
![Page 79: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/79.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Example: Converse for AWGN
I Given {c1, . . . , cM} ∈ Rn with P[W 6= W ] ≤ ε on AWGN(1).I Yaglom-map trick: replacing n→ n + 1 equalize powers:
‖cj‖2 = nP ∀j ∈ {1, . . . ,M}
I Take QY n = N (0, 1 + P)n (the caod!)I Optimal test “PX nY n vs. PX nQY n” (Neyman-Pearson):
logPY n|X n
QY n= nC +
log e
2·(‖Y n‖2
1 + P− ‖Y n − X n‖2
)
where C = 12 log(1 + P).
I Under P: Y n = X n +N (0, In)By CLT distribution of LLR (even |X n = xn !)
≈ nC +√nVZ , Z ∼ N (0, 1)
Simple algebra: V = log2 e2
(1− 1
(1+P)2
)
![Page 80: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/80.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Example: Converse for AWGN
I Given {c1, . . . , cM} ∈ Rn with P[W 6= W ] ≤ ε on AWGN(1).I Yaglom-map trick: replacing n→ n + 1 equalize powers:
‖cj‖2 = nP ∀j ∈ {1, . . . ,M}
I Take QY n = N (0, 1 + P)n (the caod!)I Optimal test “PX nY n vs. PX nQY n” (Neyman-Pearson):
logPY n|X n
QY n= nC +
log e
2·(‖Y n‖2
1 + P− ‖Y n − X n‖2
)
where C = 12 log(1 + P).
I Under P: Y n = X n +N (0, In)By CLT distribution of LLR (even |X n = xn !)
≈ nC +√nVZ , Z ∼ N (0, 1)
Simple algebra: V = log2 e2
(1− 1
(1+P)2
)
![Page 81: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/81.jpg)
Motivation Data compression Channel coding Anomalous dispersion
. . . cont’d . . .I Under P: distribution of LLR (CLT approx.)
≈ nC +√nVZ , Z ∼ N (0, 1)
I Take γ = nC −√nVQ−1(ε) =⇒
P[
logdPdQ≥ γ
]≈ 1− ε .
I Under Q: standard change-of-measure shows
Q[
logdPdQ≥ γ
]≈ exp{−γ} .
I By Neyman-Pearson
log β1−ε(PY n|X n=c ,QY n) ≈ −nC +√nVQ−1(ε)
I Punchline: ∀(n,M, ε)-code
logM . nC −√nVQ−1(ε)
N.B.! RHS can be exactly expressed via non-central χ2 dist.. . . and computed in MATLAB (w/o any CLT approx).
![Page 82: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/82.jpg)
Motivation Data compression Channel coding Anomalous dispersion
. . . cont’d . . .I Under P: distribution of LLR (CLT approx.)
≈ nC +√nVZ , Z ∼ N (0, 1)
I Take γ = nC −√nVQ−1(ε) =⇒
P[
logdPdQ≥ γ
]≈ 1− ε .
I Under Q: standard change-of-measure shows
Q[
logdPdQ≥ γ
]≈ exp{−γ} .
I By Neyman-Pearson
log β1−ε(PY n|X n=c ,QY n) ≈ −nC +√nVQ−1(ε)
I Punchline: ∀(n,M, ε)-code
logM . nC −√nVQ−1(ε)
N.B.! RHS can be exactly expressed via non-central χ2 dist.. . . and computed in MATLAB (w/o any CLT approx).
![Page 83: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/83.jpg)
Motivation Data compression Channel coding Anomalous dispersion
AWGN: Converse from βα(P ,Q) with QY = N (0, 1)n
0 200 400 600 800 1000 1200 1400 1600 1800 20000
0.1
0.2
0.3
0.4
0.5
Blocklength, n
Rat
e R
, bi
ts/c
h.us
e
Capacity
Converse
Best achievability
Normal approximation
Channel parameters:
SNR = 0 dB , ε = 10−3
![Page 84: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/84.jpg)
Motivation Data compression Channel coding Anomalous dispersion
From one QY to many
Symmetric channel
“distance” = −βα(·, ·)inputs equidistant to P ∗
Y
caod P ∗Y
a1
aj a2
Space of distributions on Y
pointsm
ch. inputs(aj , bj , cj)
Symmetric channel: choice of QY is clear
![Page 85: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/85.jpg)
Motivation Data compression Channel coding Anomalous dispersion
From one QY to many
c1
cj c2
a1
aj a2
Space of distributions on Y
b1
bj b2
pointsm
ch. inputs(aj , bj , cj)
General channels: Inputs cluster (by composition, power-allocation, . . .)(Clusters ⇐⇒ orbits of channel symmetry gp.)
![Page 86: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/86.jpg)
Motivation Data compression Channel coding Anomalous dispersion
From one QY to many
c1
cj c2
a1
aj a2
caod P ∗Y
Space of distributions on Y
b1
bj b2
pointsm
ch. inputs(aj , bj , cj)
General channels: Caod is no longer equidistant to all inputs(read: analysis horrible!)
![Page 87: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/87.jpg)
Motivation Data compression Channel coding Anomalous dispersion
From one QY to many
c1
cj c2
a1
aj a2
caod P ∗Y
Natural choicesfor QY
Space of distributions on Y
b1
bj b2
pointsm
ch. inputs(aj , bj , cj)
Solution: Take QY different for each cluster!I.e. think of QY |X
![Page 88: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/88.jpg)
Motivation Data compression Channel coding Anomalous dispersion
General meta-converse principle
Steps:
I Select auxiliary channel QY |X (art)E.g.: QY |X=x = center of a cluster of x
I Prove converse bound for channel QY |XE.g.: Q[W = W ] . # of clusters
M
I Find βα(P,Q), i.e. compare:
P : PWXYW = PW × PX |W × PY |X × PW |Y
vs.
Q : PWXYW = PW × PX |W × QY |X × PW |Y
I Amplify converse for QY |X to a converse for PY |X :
β1−Pe(PY |X ) ≤ 1− Pe(QY |X ) ∀code
![Page 89: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/89.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Meta-converse theorem: point-to-point channels
Theorem
For any code ε4= P[error] and ε′
4= Q[error] satisfy
β1−ε(PXPY |X ,PXQY |X ) ≤ 1− ε′
Advanced examples of QY |X :
I General DMC: QY |X=x = PY |X ◦ Px
Why? To reduce DMC to symmetric DMC
I Parallel AWGN: QY |X=x = f (power-allocation)Why? Since water-filling is not FBL-optimal
I Feedback: Q[Y ∈ ·|W = w ] = P[Y ∈ ·|W 6= w ]Why? To get bounds in terms of Burnashev’s C1
I Block fading: QY n|X n=xn = f (‖xn‖4)
Why? To curb “flat” codewords with ‖x‖4
‖x‖2< 3
![Page 90: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/90.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Meta-converse generalizes many classical methods
Theorem
For any code ε4= P[error] and ε′
4= Q[error] satisfy
β1−ε(PXPY |X ,PXQY |X ) ≤ 1− ε′
Corollaries:
I Fano’s inequalityI Wolfowitz strong converseI Shannon-Gallager-Berlekamp’s sphere-packing
+ improvements: [Valembois-Fossorier’04], [Wiechman-Sason’08]I Haroutounian’s sphere-packingI list-decoding conversesI Berlekamp’s low-rate converseI Verdu-Han and Poor-Verdu information spectrum conversesI Arimoto’s converse (+ extension to feedback)
![Page 91: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/91.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Meta-converse generalizes many classical methods
Theorem
For any code ε4= P[error] and ε′
4= Q[error] satisfy
β1−ε(PXPY |X ,PXQY |X ) ≤ 1− ε′
Corollaries:I Fano’s inequalityI Wolfowitz strong converseI Shannon-Gallager-Berlekamp’s sphere-packing
+ improvements: [Valembois-Fossorier’04], [Wiechman-Sason’08]I Haroutounian’s sphere-packing
I list-decoding converses E.g.: Q[W ∈ {list}] = |{list}|M
I Berlekamp’s low-rate converseI Verdu-Han and Poor-Verdu information spectrum conversesI Arimoto’s converse (+ extension to feedback)
![Page 92: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/92.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Meta-converse in networks
W1
W2
W1
W2
YXPY |X
A
A1
B
B1
C
C1
{error} = {W1 6= W1} ∪ {W2 6= W2}
![Page 93: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/93.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Meta-converse in networks
W1
W2
W1
W2
YXPY |X
QY |X
A
A1
B
B1
C
C1
{error} = {W1 6= W1} ∪ {W2 6= W2}I Probability of error depends on channel:
P[error] = ε ,
Q[error] = ε′ .
I Same idea: use code as a suboptimal binary HT: PY |X vs. QY |XI . . . and compare to the best possible test:
D(PXY ‖QXY ) ≥ d(1− ε‖1− ε′)β1−ε(PXPY |X ,PXQY |X ) ≤ 1− ε′
![Page 94: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/94.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Example: MAC (weak-converse)
P :
W1
W2
W1
W2
Y
X1
X2
Q :
W1
W2
W1
W2
Y
X1
X2
P[W1,2 = W1,2] = 1− ε Q[W1,2 = W1,2] = 1M1
. . . apply data processing of D(·||·) . . .⇓
d(1− ε‖ 1M1
) ≤ D(PY |X1X2‖QY |X1
|PX1PX2)
Optimizing QY |X1:
logM1 ≤I (X1;Y |X2) + h(ε)
1− εAlso with X1 ↔ X2 =⇒ weak converse (usual pentagon)
![Page 95: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/95.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Example: MAC (FBL?)
P :
W1
W2
W1
W2
Y
X1
X2
Q :
W1
W2
W1
W2
Y
X1
X2
P[W1,2 = W1,2] = 1− ε Q[W1,2 = W1,2] = 1M1
. . . use βα(·, ·) . . .⇓
β1−ε(PX1X2Y ,PX1QX2Y ) ≤ 1M1
On-going work: This βα is highly non-trivial to compute.[Huang-Moulin, MolavianJazi-Laneman, Yagi-Oohama]
![Page 96: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/96.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: achievability bounds
![Page 97: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/97.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Notation
I A random transformation APY |X−→ B
I (M, ε) codes:
W → A→ B→ W W ∼ Unif {1, . . . ,M}P[W 6= W ] ≤ ε
I For every PXY = PXPY |X define information density:
ıX ;Y (x ; y)4= log
dPY |X=x
dPY(y)
I E [ıX ;Y (X ;Y )] = I (X ;Y )I Var[ıX ;Y (X ;Y )|X ] = VI Memoryless channels: ıAn;Bn(An;Bn) = sum of iid.
ıAn;Bn(An;Bn)d≈ nI (A;B) +
√nVZ , Z ∼ N (0, 1)
I Goal: Prove FBL bounds.
As by-product: R∗(n, ε) & C −√
Vn Q−1(ε)
![Page 98: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/98.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Achievability bounds: classical ideas
Goal: select codewords C1, . . . ,CM in the input space A.
Two principal approaches:
I Random coding: generate C1, . . . ,CM – iid with PX andcompute average probability of error [Shannon’48, Erdos’47].
I Maximal coding: choose Cj one by one until the output spaceis exhausted [Gilbert’52, Feinstein’54, Varshamov’57].
Complication: Many inequivalent ways to apply these ideas!Which ones are the best for FBL?
![Page 99: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/99.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Classical bounds
I Feinstein’55 bound: ∃(M, ε)-code:
M ≥ supγ≥0
{γ(ε− P[ıX ;Y (X ;Y ) ≤ log γ])
}
I Shannon’57 bound: ∃(M, ε)-code:
ε ≤ infγ≥0
{P[ıX ;Y (X ;Y ) ≤ log γ] +
M − 1
γ
}.
I Gallager’65 bound: ∃(n,M, ε)-code over memoryless channel:
ε ≤ exp
{−nEr
(logM
n
)}.
I Up to M ↔ (M − 1) Feinstein and Shannon are equivalent.
![Page 100: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/100.jpg)
Motivation Data compression Channel coding Anomalous dispersion
New bounds: RCU
Theorem (Random Coding Union Bound)
For any PX there exists a code with M codewords and
ε ≤ E [min {1, (M − 1)π(X ,Y )}]π(a, b) = P[ıX ;Y (X ;Y ) ≥ ıX ;Y (X ;Y ) |X = a,Y = b]
where PXY X (a, b, c) = PX (a)PY |X (b|a)PX (c)
Proof:
I Reason as in RCU for BSC with dHam(·, ·)↔ −ıX ;Y (·, ·)I For example ML decoder: W = argmaxj ıX ;Y (Cj ;Y )
I Conditional prob. of error:
P[error |X ,Y ] ≤ (M − 1)P[ıX ;Y (X ;Y ) ≥ ıX ;Y (X ;Y ) |X ,Y ]
I Same idea: take min{·, 1} before averaging over (X ,Y ).
![Page 101: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/101.jpg)
Motivation Data compression Channel coding Anomalous dispersion
New bounds: RCU
Theorem (Random Coding Union Bound)
For any PX there exists a code with M codewords and
ε ≤ E [min {1, (M − 1)π(X ,Y )}]π(a, b) = P[ıX ;Y (X ;Y ) ≥ ıX ;Y (X ;Y ) |X = a,Y = b]
where PXY X (a, b, c) = PX (a)PY |X (b|a)PX (c)
Highlights:
I Strictly stronger than Feinstein-Shannon and Gallager
I Not easy to analyze asymptotics
I Computational complexity O(n2(|X |−1)|Y |)
![Page 102: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/102.jpg)
Motivation Data compression Channel coding Anomalous dispersion
New bounds: DT
Theorem (Dependence Testing Bound)
For any PX there exists a code with M codewords and
ε ≤ E[exp
{−∣∣∣ıX ;Y (X ;Y )− log
M−1
2
∣∣∣+}]
.
Highlights:
I Strictly stronger than Feinstein-Shannon
I . . . and no optimization over γ!
I Easier to compute than RCU
I Easier asymptotics: ε ≤ E[e−n|
1nı(X n;Y n)−R|+
]
≈ Q(√
nV {I (X ;Y )− R}
)
I Has a form of f -divergence: 1− ε ≥ Df (PXY ‖PXPY )
![Page 103: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/103.jpg)
Motivation Data compression Channel coding Anomalous dispersion
DT bound: Proof
I Codebook: random C1, . . .CM ∼ PX iid
I Feinstein decoder:
W = smallest j s.t. ıX ;Y (Cj ;Y ) > γ
I j-th codeword’s probability of error:
P[error |W = j ] ≤ P[ıX ;Y (X ;Y ) ≤ γ]︸ ︷︷ ︸©a
+(j − 1)P[ıX ;Y (X ;Y ) > γ]︸ ︷︷ ︸©b
In ©a : Cj too far from YIn ©b : Ck with k < j is too close to Y
I Average over W :
P[error] ≤ P [ıX ;Y (X ;Y ) ≤ γ] +M−1
2P[ıX ;Y (X ;Y ) > γ
]
![Page 104: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/104.jpg)
Motivation Data compression Channel coding Anomalous dispersion
DT bound: Proof
I Recap: for every γ there exists a code with
ε ≤ P [ıX ;Y (X ;Y ) ≤ γ] +M−1
2P[ıX ;Y (X ;Y ) > γ
].
I Key step: closed-form optimization of γ.Note: ıX ;Y = log dPXY
dPXY
M+1
2
(2
M+1PXY
[dPXY
dPXY
≤ eγ]
+M−1
M+1PXY
[dPXY
dPXY
> eγ])
Bayesian dependence testing!
Optimum threshold: Ratio of priors =⇒ γ∗ = log M−1
2
I Change of measure argument:
P
[dP
dQ≤ τ
]+τQ
[dP
dQ> τ
]= EP
[exp
{−∣∣∣∣log
dP
dQ− log τ
∣∣∣∣+}]
.
![Page 105: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/105.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Example: Binary Erasure Channel BEC (0.5), ε = 10−3
0 500 1000 1500 20000.3
0.35
0.4
0.45
0.5
Blocklength, n
Rat
e, b
it/ch
.use
Converse
Achievability
Normal approximation
Capacity
At n = 1000 best k → n code:
(DT) 450 ≤ k ≤ 452 (meta-c.)
Converse: QY n = truncated caod
![Page 106: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/106.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Example: Binary Erasure Channel BEC (0.5), ε = 10−3
0 500 1000 1500 20000.3
0.35
0.4
0.45
0.5
Blocklength, n
Rat
e, b
it/ch
.use
Converse
Achievability
Normal approximation
Capacity
At n = 1000 best k → n code:
(DT) 450 ≤ k ≤ 452 (meta-c.)
Converse: QY n = truncated caod
![Page 107: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/107.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Input constraints: κβ bound
Theorem
For all QY and τ there exists an (M, ε)-code inside F ⊂ A
M ≥ κτsupx β1−ε+τ (PY |X=x ,QY )
where
κτ = inf{E : PY |X [E |x]≥τ ∀x∈F}
QY [E ]
Highlights:
I Key for channels with cost constraints (e.g. AWGN).
I Bound parameterized by the output distribution.
I Reduces coding to binary HT.
![Page 108: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/108.jpg)
Motivation Data compression Channel coding Anomalous dispersion
κβ bound: idea
Decoder:
• Take received y.
• Test y against each codeword ci , i = 1, . . .M:
Run optimal binary HT for :
H0 : PY |X=ci
H1 : QY
P[detectH0] = 1− ε+ τQ[detectH0] = β1−ε+τ (PY |X=x ,QY )
• First test that returns H0 becomes the decoded codeword.
• If all H1 – declare error.
![Page 109: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/109.jpg)
Motivation Data compression Channel coding Anomalous dispersion
κβ bound: idea
Decoder:
• Take received y.
• Test y against each codeword ci , i = 1, . . .M:
Run optimal binary HT for :
H0 : PY |X=ci
H1 : QY
P[detectH0] = 1− ε+ τQ[detectH0] = β1−ε+τ (PY |X=x ,QY )
• First test that returns H0 becomes the decoded codeword.
• If all H1 – declare error.
![Page 110: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/110.jpg)
Motivation Data compression Channel coding Anomalous dispersion
κβ bound: idea
Codebook:
• Pick codewords s.t. “balls” are τ -disjoint: P[Y ∈ Bx∩others|x ] ≤ τ• Key step: Cannot pick more codewords =⇒
M⋃j=1{j-th decoding region} is a composite HT:
H0 : PY |X=x x ∈ FH1 : QY
• Performance of the best test:
κτ = inf{E : PY |X [E |x]≥τ ∀x∈F}
QY [E ] .
• Thus:
κτ ≤ Q[all M “balls”]
≤ M supxβ1−ε+τ (PY |X=x ,QY )
![Page 111: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/111.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Hierarchy of achievability bounds (no cost constr.)
Gallager
Feinstein
DT
RCU
I Arrows show logical implication
I Performance ↔ computation rule of thumb.I RCU* in [Haim-Kochman-Erez, ISIT’13]
![Page 112: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/112.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Hierarchy of achievability bounds (no cost constr.)
Tighter
Looser
Computation & Analysis
EA
SY
HA
RD
Gallager
Feinstein
DT
RCU
Typic
al perf
orm
ance
I Arrows show logical implicationI Performance ↔ computation rule of thumb.
I RCU* in [Haim-Kochman-Erez, ISIT’13]
![Page 113: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/113.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Hierarchy of achievability bounds (no cost constr.)
Tighter
Looser
Computation & Analysis
EA
SY
HA
RD
Gallager
Feinstein
DT
RCU RCU*
Typic
al perf
orm
ance
I Arrows show logical implicationI Performance ↔ computation rule of thumb.I RCU* in [Haim-Kochman-Erez, ISIT’13]
![Page 114: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/114.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel coding: dispersion
![Page 115: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/115.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Connection to CLT
Recap:I Let PY n|X n = Pn
Y |X be memoryless. FBL fundamental limit:
R∗(n, ε) = max
{1
nlogM : ∃(n,M, ε)-code
}
I Converse bounds (roughly):
R∗(n, ε) . ε-th quantile of1
nlog
dPY n|X n
dQY n
I Achievability bounds (roughly):
R∗(n, ε) & ε-th quantile of1
nıX n;Y n(X n;Y n)
I Both random variables have form: 1n · (sum of iid) =⇒ by CLT
R∗(n, ε) = C + θ
(1√n
)
This section: Study√n-term.
![Page 116: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/116.jpg)
Motivation Data compression Channel coding Anomalous dispersion
General definition of channel dispersion
Definition
For any channel we define channel dispersion as
V = limε→0
lim supn→∞
n (C − R∗(n, ε))2
2 ln 1ε
Rationale is the expansion (see below)
R∗(n, ε) = C −√
V
nQ−1(ε) + o
(1√n
)(∗)
and the fact Q−1(ε) ∼ 2 ln 1ε for ε→ 0
Recall: Approximation via (∗) is remarkably tight
![Page 117: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/117.jpg)
Motivation Data compression Channel coding Anomalous dispersion
General definition of channel dispersion
Definition
For any channel we define channel dispersion as
V = limε→0
lim supn→∞
n (C − R∗(n, ε))2
2 ln 1ε
Heuristic connection to error exponents E (R):
E (R) =(R − C )2
2· ∂
2E (R)
∂R2+ o((R − C )2)
and thus
V =
(∂2E (R)
∂R2
)−1
![Page 118: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/118.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Dispersion of memoryless channels
I DMC [Dobrushin’61,Strassen’62]:
V = Var[ıX ;Y (X ;Y )] X ∼ capacity-achieving
I AWGN channel [PPV’08]:
V = log2 e2
[1−
(1
1+SNR
)2]
I Parallel AWGN [PPV’09]:
V =L∑
j=1VAWGN
(Wj
σ2j
){Wj}–waterfilling powers
I DMC with input constraints [Hayashi’09,P’10]:
V = Var[ıX ;Y (X ;Y )|X ] X ∼ capacity-achieving
![Page 119: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/119.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Dispersion of channels with memory
I Non-white Gaussian noise with PSD N(f ):
V =log2 e
2
1/2∫
−1/2
[1− |N(f )|4
P2ξ2
]+
df ,
1/2∫
−1/2
[ξ − |N(f )|2
P
]+
df = 1
I AWGN subject to stationary fading process Hi (CSI at receiver):
V = LVar
{1
2log(1 + PH2
i )
}+
log2 e
2
(1− E 2
[1
1 + PH20
])
where LVar{Xi}4= lim 1
n Var[∑n
i=1 Xi ].I State-dependent discrete additive noise (CSI at receiver):
V = LVar {C (Si )}+E [V (S)]
I Quasi-static fading channels [Yang-Durisi-Koch-P.’14]:
V = 0 (!)
![Page 120: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/120.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Dispersion: product vs generic channels
I Relation to alphabet size:
V ≤ 2 log2 min(|A|, |B|)− C 2 .
I Dispersion is additive:
{A1 → DMC1 → B1
A2 → DMC2 → B2
}= A1×A2 → DMC → B1×B2
C = C1 + C2 , Vε = V1,ε + V2,ε
I =⇒ product DMCs have atypically low dispersion.
![Page 121: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/121.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Dispersion and normal approximation
Let PY |X be DMC and
Vε4=
{maxPX
Var[i(X ,Y )|X ] , ε < 1/2 ,
minPXVar[i(X ,Y )|X ] , ε > 1/2
where optimization is over all PX s.t. I (X ;Y ) = C .
Theorem (Strassen’62)
R∗(n, ε) = C −√
VεnQ−1(ε) + O
(log n
n
)
But [PPV’10]: a counter-example with
R∗(n, ε) = C + Θ(n−
23
)
![Page 122: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/122.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Dispersion and normal approximation
Let PY |X be DMC and
Vε4=
{maxPX
Var[i(X ,Y )|X ] , ε < 1/2 ,
minPXVar[i(X ,Y )|X ] , ε > 1/2
where optimization is over all PX s.t. I (X ;Y ) = C .
Theorem (Strassen’62, PPV’10)
R∗(n, ε) = C −√
VεnQ−1(ε) + O
(log n
n
)
unless DMC is exotic in which case O(
log nn
)becomes O(n−
23 ).
![Page 123: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/123.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Further results on O(
log nn
)
I For BEC we have:
R∗(n, ε) = C −√
V
nQ−1(ε) + 0 · log n
n+ O
(1
n
)
I For most other symmetric channels (incl. BSC and AWGN∗):
R∗(n, ε) = C −√
V
nQ−1(ε) +
1
2
log n
n+ O
(1
n
)
I For most DMC (under mild conditions):
R∗(n, ε) ≥ C −√
VεnQ−1(ε) +
1
2
log n
n+ O
(1
n
)
I For all DMC [Moulin’12, Tomamichel-Tan’13]
R∗(n, ε) ≤ C −√
VεnQ−1(ε) +
1
2
log n
n+ O
(1
n
)
![Page 124: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/124.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Applications
![Page 125: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/125.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Evaluating performance of real-world codes
I Comparing codes: usual method – waterfall plots Pe vs. SNR
I Problem: Not fair for different rates.
=⇒ define rate-invariant metric:
Definition (Normalized rate)
Given rate R code find SNR at which Pe = ε.
Rnorm =R
R∗(n, ε,SNR)
I Agreement: ε = 10−3 or 10−4
I Take R∗(n, ε,SNR) ≈ C −√
Vn Q−1(ε)
I A family of channels needed (e.g. AWGN or BSC)
![Page 126: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/126.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Evaluating performance of real-world codes
I Comparing codes: usual method – waterfall plots Pe vs. SNR
I Problem: Not fair for different rates.
=⇒ define rate-invariant metric:
Definition (Normalized rate)
Given rate R code find SNR at which Pe = ε.
Rnorm =R
R∗(n, ε,SNR)
I Agreement: ε = 10−3 or 10−4
I Take R∗(n, ε,SNR) ≈ C −√
Vn Q−1(ε)
I A family of channels needed (e.g. AWGN or BSC)
![Page 127: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/127.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Codes vs. fundamental limits (from 1970’s to 2012)
102
103
104
105
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1Normalized rates of code families over BIAWGN, Pe=0.0001
Blocklength, n
Nor
mal
ized
rat
e
Turbo R=1/3Turbo R=1/6Turbo R=1/4VoyagerGalileo HGATurbo R=1/2Cassini/PathfinderGalileo LGAHermitian curve [64,32] (SDD)BCH (Koetter−Vardy)Polar+CRC R=1/2 (List dec.)ME LDPC R=1/2 (BP)
![Page 128: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/128.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Codes vs. fundamental limits (from 1970’s to 2016)
102
103
104
105
0.6
0.65
0.7
0.75
0.8
0.85
0.9
0.95
1Normalized rates of code families over BIAWGN, Pe=0.0001
Blocklength, n
Nor
mal
ized
rat
e
Turbo R=1/3Turbo R=1/6Turbo R=1/4VoyagerGalileo HGATurbo R=1/2Cassini/PathfinderGalileo LGABCH (Koetter−Vardy)Polar+CRC R=1/2 (L=32)Polar+CRC R=1/2 (L=256)Huawei NB−LDPCHuawei hybrid−CyclicME LDPC R=1/2 (BP)
![Page 129: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/129.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Performance of short algebraic codes (BSC, ε = 10−3)
101
102
103
0.75
0.8
0.85
0.9
0.95
1
1.05
Blocklength, n
Nor
mal
ized
rat
e
HammingGolayBCHExtended BCHReed−Muller: 1 ord.Reed−Muller: 2 ord.Reed−Muller: 3 ord.Reed−Muller: 4+ ord.
![Page 130: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/130.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Optimizing ARQ systems
I End-user wants Pe = 0I Usual method: automatic repeat request (ARQ)
average throughput = Rate× (1− P[error])
I Question: Given k bits what rate (equiv. ε) maximizesthroughput?
I Assume (C ,V ) is known. Then approximately
T ∗(k) ≈ maxR
R·(
1− Q
(√kR
V
{C
R− 1
}))
0 0.5 1 1.50
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
I Solution: ε∗(k) ∼ 1√kt log kt
, t = CV
I Punchline: For k ∼ 1000 bit and reasonable channels
ε ≈ 10−3..10−2
(almost) universal adaptation rule!
![Page 131: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/131.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Optimizing ARQ systems
I End-user wants Pe = 0I Usual method: automatic repeat request (ARQ)
average throughput = Rate× (1− P[error])
I Question: Given k bits what rate (equiv. ε) maximizesthroughput?
I Assume (C ,V ) is known. Then approximately
T ∗(k) ≈ maxR
R·(
1− Q
(√kR
V
{C
R− 1
}))
0 0.5 1 1.50
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
I Solution: ε∗(k) ∼ 1√kt log kt
, t = CV
I Punchline: For k ∼ 1000 bit and reasonable channels
ε ≈ 10−3..10−2
(almost) universal adaptation rule!
![Page 132: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/132.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Anomalous dispersion
![Page 133: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/133.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Recap: From I (X ;Y ) to βα
I Asymptotics (Shannon):
limn→∞
R∗(n, ε) = C4= max
PX
I (X ;Y ) .
I Non-asymptotic analog of C [PPV’10]:
R∗(n, ε) = maxPX
− 1
nlog β1−ε(PXPY |X ,PXQY )+O
(log n
n
)
I By Neyman-Pearson:
R∗(n, ε) ≈ ε-th quantile of1
nlog
dPY n|X n
dQY n
I What is PX n , QY n?A: saddle-point of βα (or approx.)
![Page 134: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/134.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Recap: From I (X ;Y ) to βα
I Asymptotics (Shannon):
limn→∞
R∗(n, ε) = C4= max
PX
I (X ;Y ) .
I Non-asymptotic analog of C [PPV’10]:
R∗(n, ε) = maxPX
minQY
−1
nlog β1−ε(PXPY |X ,PXQY )+O
(log n
n
)
I By Neyman-Pearson:
R∗(n, ε) ≈ ε-th quantile of1
nlog
dPY n|X n
dQY n
I What is PX n , QY n?A: saddle-point of βα (or approx.)
![Page 135: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/135.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Recap: From I (X ;Y ) to βα
I Asymptotics (Shannon):
limn→∞
R∗(n, ε) = C4= max
PX
I (X ;Y ) .
I Non-asymptotic analog of C [PPV’10]:
R∗(n, ε) = maxPX
minQY
−1
nlog β1−ε(PXPY |X ,PXQY )+O
(log n
n
)
I By Neyman-Pearson:
R∗(n, ε) ≈ ε-th quantile of1
nlog
dPY n|X n
dQY n
I What is PX n , QY n?A: saddle-point of βα (or approx.)
![Page 136: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/136.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channel dispersion
I Memoryless channels: saddle-point PX ,QY are bothmemoryless (approx.)
I =⇒ from CLT:
R∗(n, ε) ≈ ε-th quantile of1
nlog
dPY n|X n
dQY n
≈ C −√
V
nQ−1(ε) ,
where
C = E[
logPY |X (Y |X )
QY (Y )
]V = Var
[log
PY |X (Y |X )
QY (Y )
]
I Punchline:Channel dispersion = Variance of info density
![Page 137: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/137.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Anomalous dispersion terms
I Feedback coding + variable-length [PPV’11]:
R∗(n, ε) =1
1− εC + 0 ·√
1
n+ O
(log n
n
)
I Var-length data-compression [Kostina-P.-Verdu’14]:
R∗(n, ε) = (1− ε)H −√
V
2πne−
(Q−1(ε))2
2 + O
(log n
n
)
I Long-term (“average-over-the-codebook”) power-constraint[Yang-Caire-Durisi-P.’15]:
R∗(n, ε) = C
(P
1− ε
)−√
V
(P
1− ε
)loge n
n+ O
(1√n
)
![Page 138: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/138.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Anomalous dispersion terms (cont’d)
I Quasi-static fading channels [Yang-Durisi-Koch-P.’14]:
R∗(n, ε) = Cε + 0 ·√
1
n+ O
(log n
n
)
I Energy-per-bit under multiplicative fading [Yang-Durisi-P.’15]:
Eb(k , ε) ≈ (−1.59 dB) + cε3
√log k
k
Contrast with CLT-like AWGN: Eb(k, ε) ≈ (−1.59 dB) +√
VkQ−1(ε)
![Page 139: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/139.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Channels with feedback
![Page 140: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/140.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Benefits of feedback: From ARQ to Hybrid ARQ
Encoder Channel DecoderW
z−1Y t−1
Xt Yt W
I Memoryless channels: feedback does not improve C[Shannon’56]
I Question: What about higher order terms?
Theorem
For any DMC with capacity C and 0 < ε < 1 we have for codeswith feedback and variable length:
R∗f (n, ε) =C
1− ε + O
(log n
n
).
Note: dispersion is zero!
![Page 141: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/141.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Benefits of feedback: From ARQ to Hybrid ARQ
Encoder Channel DecoderW
z−1Y t−1
Xt Yt W
I Memoryless channels: feedback does not improve C[Shannon’56]
I Question: What about higher order terms?
Theorem
For any DMC with capacity C and 0 < ε < 1 we have for codeswith feedback and variable length:
R∗f (n, ε) =C
1− ε + O
(log n
n
).
Note: dispersion is zero!
![Page 142: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/142.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Stop feedback bound (BSC version)
Theorem
For any γ > 0 there exists a stop feedback code of rate R, averagelength ` = E [τ ] and probability of error over BSC (δ)
ε ≤ E [f (τ)] ,
where
f (n)4= E
[1{τ ≤ n}2`R−Sτ
]
τ4= inf{n ≥ 0 : i(X n;Y n) ≥ γ}
i(X n;Y n)4= sum of iid Bernoulli(δ) .
![Page 143: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/143.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Feedback codes for BSC(0.11), ε = 10−3
0 100 200 300 400 500 600 700 800 900 10000
0.1
0.2
0.3
0.4
0.5
Avg. blocklength
Rat
e, b
its/c
h.us
e
ConverseAchievabilityNo feedback
No feedback
Stop feedback
![Page 144: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/144.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Why dispersion is zero?
I Channel coding goal: have LLR ≥ logM w.h.p.
I LLR = Sn = sum-of-iid.
I Fixed blocklength: set n so large that
Pe ≈ P [Sn < logM] .
Thus get√n fluctuations.
I Expected-length:
Sτ = logM ± const
Thus P[Sτ < logM] ≈ 0
I Key: ` = E [τ ] averages-out normal fluctuations!
![Page 145: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/145.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Effects of flow control
Encoder Channel Decoder
STOP
W
z−1Y t−1
Xt Yt W
I Modeling of packet termination
I Often: reliability of start/end � reliability of payload
Theorem
If reliable termination is available, then there exist codes withvariable length and feedback achieving
R∗t (n, 0) ≥ C + O
(1
n
).
![Page 146: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/146.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Effects of flow control
Encoder Channel Decoder
STOP
W
z−1Y t−1
Xt Yt W
I Modeling of packet termination
I Often: reliability of start/end � reliability of payload
Theorem
If reliable termination is available, then there exist codes withvariable length and feedback achieving
R∗t (n, 0) ≥ C + O
(1
n
).
![Page 147: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/147.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Codes employing feedback + termination (BSC version)
Theorem
Consider a BSC (δ) with feedback and reliable termination. Thereexists a code sending k bits with zero error and average length
` ≤∞∑
n=0
n∑
t=0
(n
t
)δt(1− δ)n−t min
{1,
t∑
k=0
(n
k
)2k−n
}.
![Page 148: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/148.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Feedback + termination for the BSC(0.11)
0 100 200 300 400 500 600 700 800 900 10000
0.1
0.2
0.3
0.4
0.5
0.6
Avg. blocklength
Rat
e, b
its/c
h.us
e
VLFTVLFNo feedback
No feedback
Feedback + Termination
Stop feedback
![Page 149: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/149.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Benefit of feedback
Delay to achieve 90% of the capacity of the BSC(0.11):
I No feedback:n ≈ 3100
I Stop feedback + variable-length:
n . 200
I Feedback + variable-length + termination:
n . 20
![Page 150: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/150.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Benefit of feedback
Delay to achieve 90% of the capacity of the BSC(0.11):
I No feedback:n ≈ 3100 penalty term: O
(1√n
)
I Stop feedback + variable-length:
n . 200 penalty term: O(
log nn
)
I Feedback + variable-length + termination:
n . 20 penalty term: O(
1n
)
![Page 151: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/151.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Quasi-static wireless channels
![Page 152: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/152.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Wireless channels: fading and AWGN
Encoder Decoder
Fading Process
CSIT CSIR
Hi
Xi Yi
Zi ∼ CN (0, 1)
I AWGN channel with fluctuating SNR:
Yi = HiXi + Zi .
I Channel gain: known at Rx (CSIR) and/or Tx (CSIT)
I Channel dynamics: quasi-static H1 = H2 = · · ·
![Page 153: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/153.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Quasi-static fading channel: SISO
I/O relation
+ = 𝑛
𝒚 ℎ 𝒙 𝒘
Notes:I H–random but fixed for all channel uses (nonergodic)
I Power-constraint: ‖x‖2 ≤ nP
I Outage capacity:
Cε4= lim
n→∞R∗(n, ε) = sup{R : Pout(R) ≤ ε}
Pout(R)4= P
[log(1 + P|H|2) ≤ R
]
I Cε is same in all 4 cases: noCSI, CSIR, CSIT, CSIRTReason: Can easily learn the channel.
![Page 154: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/154.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Main result: Dispersion is zero (SISO)
Theorem
If pdf of H is smooth and P ′out(Cε) > 0, then
R∗(n, ε) = Cε +����Z
ZZZ
0 · 1√n
+ O
(log n
n
)
In all 4 cases: noCSI, CSIR, CSIT, CSIRT
![Page 155: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/155.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Why zero dispersion?
I AWGN channel
R∗(n, ε) ≈ C −√
V
nQ−1(ε)
m
ε ≈ P
[C +
√V
nZ ≤ R∗(n, ε)
], Z ∼ N (0, 1)
I Quasi-static case
ε ≈ P
[C(H) +
√V (H)
nZ ≤ R∗(n, ε)
]
= P
[Z√n≤ R∗(n, ε)− C(H)√
V (H)
]
→ P [C(H) ≤ R∗(n, ε)] , n→∞
![Page 156: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/156.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Why zero dispersion?
I AWGN channel
R∗(n, ε) ≈ C −√
V
nQ−1(ε)
m
ε ≈ P
[C +
√V
nZ ≤ R∗(n, ε)
], Z ∼ N (0, 1)
I Quasi-static case
ε ≈ P
[C(H) +
√V (H)
nZ ≤ R∗(n, ε)
]
= P
[Z√n≤ R∗(n, ε)− C(H)√
V (H)
]
→ P [C(H) ≤ R∗(n, ε)] , n→∞
![Page 157: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/157.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Why zero dispersion?
I AWGN channel
R∗(n, ε) ≈ C −√
V
nQ−1(ε)
m
ε ≈ P
[C +
√V
nZ ≤ R∗(n, ε)
], Z ∼ N (0, 1)
I Quasi-static case
ε ≈ P
[C(H) +
√V (H)
nZ ≤ R∗(n, ε)
]
= P
[Z√n≤ R∗(n, ε)− C(H)√
V (H)
]
→ P [C(H) ≤ R∗(n, ε)] , n→∞
![Page 158: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/158.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Why zero dispersion?
I AWGN channel
R∗(n, ε) ≈ C −√
V
nQ−1(ε)
m
ε ≈ P
[C +
√V
nZ ≤ R∗(n, ε)
], Z ∼ N (0, 1)
I Quasi-static case
ε ≈ P
[C(H) +
√V (H)
nZ ≤ R∗(n, ε)
]
= P
[Z√n≤ R∗(n, ε)− C(H)√
V (H)
]
→ P [C(H) ≤ R∗(n, ε)] , n→∞
![Page 159: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/159.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Why zero dispersion?
Lemma
Let A ⊥⊥ B s.t. B has smooth pdf, E [A] = 0,E [A2] = 1:
P[B ≥ A√
n
]= P[B ≥ 0] + O
(1
n
).
Pout(Cε) =
ε ≈ P
[Z√n≤ R∗(n, ε)− C(H)√
V (H)
]
= P[C(H) ≤ R∗(n, ε)
]+ O(1/n)
= Pout
(R∗(n, ε)
)+ O(1/n)
R∗(n, ε) = Cε + O
(log n
n
)
![Page 160: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/160.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Why zero dispersion?
Lemma
Let A ⊥⊥ B s.t. B has smooth pdf, E [A] = 0,E [A2] = 1:
P[B ≥ A√
n
]= P[B ≥ 0] + O
(1
n
).
Pout(Cε) =
ε ≈ P
[Z√n≤ R∗(n, ε)− C(H)√
V (H)
]
= P[C(H) ≤ R∗(n, ε)
]+ O(1/n)
= Pout
(R∗(n, ε)
)+ O(1/n)
R∗(n, ε) = Cε + O
(log n
n
)
![Page 161: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/161.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Why zero dispersion?
Lemma
Let A ⊥⊥ B s.t. B has smooth pdf, E [A] = 0,E [A2] = 1:
P[B ≥ A√
n
]= P[B ≥ 0] + O
(1
n
).
Pout(Cε) =
ε ≈ P
[Z√n≤ R∗(n, ε)− C(H)√
V (H)
]
= P[C(H) ≤ R∗(n, ε)
]+ O(1/n)
= Pout
(R∗(n, ε)
)+ O(1/n)
R∗(n, ε) = Cε + O
(log n
n
)
![Page 162: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/162.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Why zero dispersion?
Lemma
Let A ⊥⊥ B s.t. B has smooth pdf, E [A] = 0,E [A2] = 1:
P[B ≥ A√
n
]= P[B ≥ 0] + O
(1
n
).
Pout(Cε) = ε ≈ P
[Z√n≤ R∗(n, ε)− C(H)√
V (H)
]
= P[C(H) ≤ R∗(n, ε)
]+ O(1/n)
= Pout
(R∗(n, ε)
)+ O(1/n)
R∗(n, ε) = Cε + O
(log n
n
)
![Page 163: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/163.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Why zero dispersion?
Lemma
Let A ⊥⊥ B s.t. B has smooth pdf, E [A] = 0,E [A2] = 1:
P[B ≥ A√
n
]= P[B ≥ 0] + O
(1
n
).
Pout(Cε) = ε ≈ P
[Z√n≤ R∗(n, ε)− C(H)√
V (H)
]= P
[C(H) ≤ R∗(n, ε)
]+ O(1/n)
= Pout
(R∗(n, ε)
)+ O(1/n)
R∗(n, ε) ≈ Cε + O
(1
n
)
R∗(n, ε) = Cε + O
(log n
n
)
![Page 164: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/164.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Why zero dispersion?
Lemma
Let A ⊥⊥ B s.t. B has smooth pdf, E [A] = 0,E [A2] = 1:
P[B ≥ A√
n
]= P[B ≥ 0] + O
(1
n
).
Pout(Cε) = ε ≈ P
[Z√n≤ R∗(n, ε)− C(H)√
V (H)
]
= P[C(H) ≤ R∗(n, ε)
]+ O(1/n)
= Pout
(R∗(n, ε)
)+ O(1/n)
R∗(n, ε) = Cε + O
(log n
n
)
![Page 165: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/165.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Normal Approximation Rapprox(n, ε)
1
Outage capacity (Cε)
Normal Approximation
Converse (CSIRT)
Achievability (CSIR)
Blocklength, n
WGN
Rat
e,bi
ts/c
h.us
e
1000100 200 300 400 500 600 700 800 900
0.2
0.4
0.6
0.8
00
1
July 5, 2013 DRAFT
![Page 166: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/166.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Upper Bound on R∗(n, ε) (CSIRT)
I From meta-converse:
R∗(n, ε) ≤ supPX
1
nlog
1
β1−ε(PXYH ,PX × QYH)
I SIMO: choose QYH to be caod
I MIMO: choose QYH |X weakly dependent on X
![Page 167: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/167.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Upper Bound on R∗(n, ε) (CSIRT)
I From meta-converse:
R∗(n, ε) ≤ supPX
1
nlog
1
β1−ε(PXYH ,PX × QYH)
I SIMO: choose QYH to be caod
I MIMO: choose QYH |X weakly dependent on X
![Page 168: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/168.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Lower bound on R∗(n, ε) (no CSI)
I Use κβ bound [PPV ‘10]
R∗(n, ε) ≥ 1
nlog
κτ (F ,QY )
supx∈F β1−ε+τ (PY |X=x ,QY )
I A stochastically degraded channel
X → Y → span(Y )
I β1−ε+τ : angle thresholding
I κτ : computed in closed form
![Page 169: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/169.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Lower bound on R∗(n, ε) (no CSI)
I Angle thresholding
= 𝒚 ℎ 𝒙
𝒙
𝒚
𝜃𝜖 𝒙𝑖
𝒚
𝟎
𝒙1
𝒙𝑘
𝒙𝑗
I Declare xi iff θ(xi ,Y ) ≤ θε
I Declare error otherwise
I Knowledge of h or statistics of h notrequired
![Page 170: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/170.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Lower bound on R∗(n, ε) (no CSI)
I Angle thresholding
= 𝒚 ℎ 𝒙
𝒙
𝒚
𝜃𝜖 𝒙𝑖
𝒚
𝟎
𝒙1
𝒙𝑘
𝒙𝑗
I Declare xi iff θ(xi ,Y ) ≤ θε
I Declare error otherwise
I Knowledge of h or statistics of h notrequired
![Page 171: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/171.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Bounds on R∗(n, ε)
1
Outage capacity (Cε)
Achievability (no CSI)
Converse (CSIRT)
Achievability (CSIR)
Blocklength, n
WGN
Rat
e,bi
ts/c
h.us
e
ε = 10−3
SNR = −1.55 dB
#RX antennas = 2
K = 20 dB
1000100 200 300 400 500 600 700 800 900
0.2
0.4
0.6
0.8
00
1
July 5, 2013 DRAFT
![Page 172: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/172.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Bounds on R∗(n, ε)
1
Outage capacity (Cε)
Blocklength, n
WGN
Rat
e,bi
ts/c
h.us
e
1000100 200 300 400 500 600 700 800 900
0.2
0.4
0.6
0.8
00
1
July 5, 2013 DRAFT
![Page 173: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/173.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Does zero dispersion imply fast convergence to Cε?
R∗(n, ε) = Cε + O
(log n
n
)
O
(log n
n
)→√
VAWGN
nQ−1(ε) + O
(log n
n
), MR →∞
1
MR = 32
MR = 16
MR = 8
MR = 4MR = 2
‖h‖2/MR
43.53
2.5
2.5
2
2
1.5
1.5
1
1
0.5
0.500
July 5, 2013 DRAFT
![Page 174: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/174.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Does zero dispersion imply fast convergence to Cε?
R∗(n, ε) = Cε + O
(log n
n
)
I n∗: blocklength required to achieve 90% of Cε
I n∗AWGN ≈ 1400
O
(log n
n
)→√
VAWGN
nQ−1(ε) + O
(log n
n
), MR →∞
1
MR = 32
MR = 16
MR = 8
MR = 4MR = 2
‖h‖2/MR
43.53
2.5
2.5
2
2
1.5
1.5
1
1
0.5
0.500
July 5, 2013 DRAFT
![Page 175: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/175.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Does zero dispersion imply fast convergence to Cε?
R∗(n, ε) = Cε + O
(log n
n
)
1
100
101
102
103
Number of RX antennas
n∗
ε = 10−3
SNR = 0 dB
250
350
200
300
400
500
600
700
450
550
650
July 4, 2013 DRAFT
O
(log n
n
)→√
VAWGN
nQ−1(ε) + O
(log n
n
), MR →∞
1
MR = 32
MR = 16
MR = 8
MR = 4MR = 2
‖h‖2/MR
43.53
2.5
2.5
2
2
1.5
1.5
1
1
0.5
0.500
July 5, 2013 DRAFT
![Page 176: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/176.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Does zero dispersion imply fast convergence to Cε?
O
(log n
n
)→√
VAWGN
nQ−1(ε) + O
(log n
n
), MR →∞
1
MR = 32
MR = 16
MR = 8
MR = 4MR = 2
‖h‖2/MR
43.53
2.5
2.5
2
2
1.5
1.5
1
1
0.5
0.500
July 5, 2013 DRAFT
![Page 177: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/177.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Performance of LTE-Advanced coding schemes
1
Outage capacity (Cε)
LTE-Advanced codes
Converse
Achievability
Normal Approximation
Blocklength, n
Rat
e,bi
ts/c
h.us
e
1000100 200 300 400 500 600 700 800 900
0.2
0.4
0.6
0.8
00
1
July 1, 2013 DRAFT
![Page 178: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/178.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Quasi-static fading channels: MIMO
Yi = HXi+Zi , H ∈ Cmr×mtTx Rx
Hkmt mr
Theorem
If H has smooth density with polynomial tails:
R∗(n, ε) = Cε +
����@@@@
0 · 1√n
+ O
(log n
n
)
Under all 4 cases: noCSI, CSIT, CSIR, CSIRT.
I Technical heavy-lifting
I Input achieving Cε unknown (Telatar conjecture)
![Page 179: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/179.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Energy-efficiency in wireless channels
![Page 180: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/180.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Energy per bit: Gaussian channel
Zi∼ N(
0, N02
)
↓Xi −→
⊕ −→ Yi
I Problem: Encode k bits into finite-energy waveforms:
E
[ ∞∑
i=1
|Xi |2]≤ kEb .
I No rate-constraint (aka wideband limit)
I Asymptotically: [Shannon’49]
min
(Eb
N0
)→ loge 2 = −1.59 dB , k →∞ .
I . . . and also must have R → 0.
![Page 181: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/181.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Energy per bit vs. # of information bits (ε = 10−3)
100
101
102
103
104
105
106
−2
−1
0
1
2
3
4
5
6
7
8
Information bits, k
Eb/
No,
dB
Achievability (non−feedback)Converse (non−feedback)Full feedback (optimal)
![Page 182: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/182.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Energy per bit vs. # of information bits (ε = 10−3)
100
101
102
103
104
105
106
−2
−1
0
1
2
3
4
5
6
7
8
Information bits, k
Eb/
No,
dB
Achievability (non−feedback)Converse (non−feedback)Stop feedbackFull feedback (optimal)
![Page 183: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/183.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Energy per bit: Enter fading
Encoder Decoder
Fading Process
Hi
Xi Yi
Zi ∼ CN (0, N0)
I AWGN channel with fluctuating SNR:
Yi = HiXi + Zi .
I Channel gain: known (CSIR) or unknown (noCSI)
![Page 184: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/184.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Energy per bit: Enter fading
Encoder Decoder
Fading Process
Hi
Xi Yi
Zi ∼ CN (0, N0)
I [Jacobs’63], [Kennedy’69]: For both CSIR and noCSI (!)
min
(Eb
N0
)→ loge 2 = −1.59 dB , k →∞ .
I Achieved by very wideband (flash) signals
![Page 185: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/185.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Energy per bit: Enter fading
Encoder Decoder
Fading Process
Hi
Xi Yi
Zi ∼ CN (0, N0)
I [Verdu’02]:(Eb
N0
)= (−1.59 dB) + ε =⇒ R ≈ e−
1ε (noCSI )
=⇒ R ≈ ε (CSIR)
I Note: R = kn – spectral efficiency
![Page 186: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/186.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Energy per bit: Enter fading
Encoder Decoder
Fading Process
Hi
Xi Yi
Zi ∼ CN (0, N0)
I All of the above: k →∞.I Our work: For finite k
noCSI : Eb(k , ε) ≈ (−1.59 dB) + cε3
√log k
k
w/ CSIR : Eb(k , ε) ≈ (−1.59 dB) + cε
√V
k
![Page 187: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/187.jpg)
Motivation Data compression Channel coding Anomalous dispersion
noCSI fading: equivalent channel
Yi = HiXi + Zi , Y ,H,X ,Z ∈ C
I Fading: assume Hiiid∼ CN (0, 1) (Rayleigh (!) fast-fading)
I Input: Only depends on |Xi |2I Output: Suff. statistic is |Yi |2I Equivalent channel:
|Yi |2 = (1 + |Xi |2)Si , Siiid∼ Exp(1) .
I Info density:
i(x ; y)4= log
dPY∞|X∞=x
dPY∞|X∞=0(y)
I . . . and its distribution given X = x :
i(x ;Y )d=∑
i
|xi |2Si log e − log(1 + |xi |2)
![Page 188: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/188.jpg)
Motivation Data compression Channel coding Anomalous dispersion
noCSI fading: information density
I Info density:
i(x ;Y )d=∑
i
|xi |2Si log e − log(1 + |xi |2)
I Bit-per-energy: Maximize (1− ε)-quantile of i(x ;Y )E s.t.
∑
i
|xi |2 ≤ E
I Classical asymptotics: Take E [i(x ;Y )] and E →∞
max
∑i |xi |2 log e − log(1 + |xi |2)∑
i |xi |2= log e
(bit
energy/N0
)
Optimizer: xi 6= 0 =⇒ |xi | � 1 (flash!)
![Page 189: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/189.jpg)
Motivation Data compression Channel coding Anomalous dispersion
noCSI fading: information density
I Goal: Maximize (1− ε)-quantile of i(x ;Y )E .
i(x ;Y )d=∑
i
|xi |2Si log e − log(1 + |xi |2)
I Fundamental tension:1. To have LLN need lots of small atoms.2. To have
∑i log(1+|xi |2)∑
i |xi |2→ 0 need large atoms.
I Add extra constraint |xi |2 ≤ A: [Sethuraman-Hajek’05]
max
∑i |xi |2 log e − log(1 + |xi |2)∑
i |xi |2=
{log e − log(1 + A)
A
}
I By CLT:
1
Ei(x ;Y ) ∼
{log e − log(1 + A)
A
}+
√A log2 e
EZ
I Optimize: A∗ ∼ (Q−1(ε))−23E
13 log
23 E
![Page 190: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/190.jpg)
Motivation Data compression Channel coding Anomalous dispersion
noCSI fading: information density
I Goal: Maximize (1− ε)-quantile of i(x ;Y )E .
i(x ;Y )d=∑
i
|xi |2Si log e − log(1 + |xi |2)
I Fundamental tension:1. To have LLN need lots of small atoms.2. To have
∑i log(1+|xi |2)∑
i |xi |2→ 0 need large atoms.
I Add extra constraint |xi |2 ≤ A: [Sethuraman-Hajek’05]
max
∑i |xi |2 log e − log(1 + |xi |2)∑
i |xi |2=
{log e − log(1 + A)
A
}
I By CLT:
1
Ei(x ;Y ) ∼
{log e − log(1 + A)
A
}+
√A log2 e
EZ
I Optimize: A∗ ∼ (Q−1(ε))−23E
13 log
23 E
![Page 191: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/191.jpg)
Motivation Data compression Channel coding Anomalous dispersion
noCSI fading: information density
I Goal: Maximize (1− ε)-quantile of i(x ;Y )E .
i(x ;Y )d=∑
i
|xi |2Si log e − log(1 + |xi |2)
I Fundamental tension:1. To have LLN need lots of small atoms.2. To have
∑i log(1+|xi |2)∑
i |xi |2→ 0 need large atoms.
I Add extra constraint |xi |2 ≤ A: [Sethuraman-Hajek’05]
max
∑i |xi |2 log e − log(1 + |xi |2)∑
i |xi |2=
{log e − log(1 + A)
A
}
I By CLT:
1
Ei(x ;Y ) ∼
{log e − log(1 + A)
A
}+
√A log2 e
EZ
I Optimize: A∗ ∼ (Q−1(ε))−23E
13 log
23 E
![Page 192: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/192.jpg)
Motivation Data compression Channel coding Anomalous dispersion
noCSI fading: information density
I Goal: Maximize (1− ε)-quantile of i(x ;Y )E .
i(x ;Y )d=∑
i
|xi |2Si log e − log(1 + |xi |2)
I Fundamental tension:1. To have LLN need lots of small atoms.2. To have
∑i log(1+|xi |2)∑
i |xi |2→ 0 need large atoms.
I Add extra constraint |xi |2 ≤ A: [Sethuraman-Hajek’05]
max
∑i |xi |2 log e − log(1 + |xi |2)∑
i |xi |2=
{log e − log(1 + A)
A
}
I By CLT:
1
Ei(x ;Y ) ∼
{log e − log(1 + A)
A
}+
√A log2 e
EZ
I Optimize: A∗ ∼ (Q−1(ε))−23E
13 log
23 E
This is only a heuristic!
![Page 193: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/193.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Achievability: non-asymptotic c.p.u.c. bound
Theorem (Verdu’90)
For a memoryless channel PY |X the capacity-per-unit-cost is
Cp.u.c. = supx
D(PY |X=x‖PY |X=0)
cost(x)
![Page 194: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/194.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Achievability: non-asymptotic c.p.u.c. bound
Theorem
There exists a k-bit code with P[error] = ε and
2k ≥ sup0<τ<ε
τ
β1−ε+τ (P⊗NY |X=x ,P⊗NY |X=0)
cost
bit=
N · cost(x0)
k
where N ∈ Z+ and x0 ∈ X .
Proof:I As in [Verdu’90] use codewords:
cj = [0, . . . , 0︸ ︷︷ ︸(j−1)N
, x0, . . . , x0︸ ︷︷ ︸N
, 0, . . .] j ∈ [2k ]
I And run 2k binary tests “X = cj vs X = 0”I Can be seen as a version of κβ-bound from [PPV’10]
![Page 195: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/195.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Asymptotics: noCSI fading channel
Theorem (Yang-Durisi-P.’15)
Max # of bits transmittable with energy budget E
k∗(E , ε) = E log2 e − const ·(Q−1(ε)E
) 23 log
132 E + · · ·
Proof:I For any codeword x ∈ C∞:
i(x ;Y )d=∞∑
n=1
|xi |2Si − log(1 + |xi |2), Siiid∼ Exp(1)
I Analysis (tedious!) shows: ε-quantile is maximized by
x∗ = [√A,√A, . . . ,
√A,︸ ︷︷ ︸
N-times
0, . . .]
N = c(Q−1(ε)E log−1 E
) 23 , N · A = E
![Page 196: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/196.jpg)
Motivation Data compression Channel coding Anomalous dispersion
What about perfect CSI at Receiver?
Yi = HiXi + Zi , Y ,H,X ,Z ∈ C
Theorem
Fading with CSIR = non-fading AWGN (for energy-per-bit).
Small print: Achievability and asymptotics coincide.
I Simulate AWGN channel
Tx: X1 = · · · = Xn =x√n
Rx: Y =1√n
n∑
i=1
HiYi
I As n→∞ we have by LLN and CLT: (E |H|2 = 1)
Y = x · 1
n
n∑
i=1
|Hi |2 +1√n
n∑
i=1
HiZi → x + Z .
![Page 197: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/197.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Energy-per-bit: noCSI vs CSIR
100 101 102 103 104 105 106 107 108−2
0
2
4
6
8
10
12
14
Achievability
Converse
−1.59 dB
Rayleigh fading, noCSI
fading+CSIR, non-fading AWGN
Information bits, k
Eb
N0,dB
To achieve −1.5 dB :
k ≈ 6 · 104 (CSIR)
k ≈ 7 · 107 (noCSI)
![Page 198: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/198.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Energy-per-bit: noCSI vs CSIR
100 101 102 103 104 105 106 107 108−2
0
2
4
6
8
10
12
14
Achievability
Converse
−1.59 dB
Rayleigh fading, noCSI
fading+CSIR, non-fading AWGN
Information bits, k
Eb
N0,dB
To achieve −1.5 dB :
k ≈ 6 · 104 (CSIR)
k ≈ 7 · 107 (noCSI)
![Page 199: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/199.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Extensions: non-iid fading
Non-iid fading process:
I Block-fading: H1 = · · · = Hnc .
I Achievability: Simulate iid H (use one symbol per block)
I Converse: info density
i(x ;Y )d=∞∑
i=1
‖xi‖2Si − log(1 + ‖xi‖2)
where xi ∈ Cnc is input to i-th fading block.Proof: Apply unitary mtx to each xi .
I =⇒ Achievability/Converse/Asymptotics same as for iid H
I =⇒ Learning the channel is futile (!)
![Page 200: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/200.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Extensions: Multiple antennas
Multiple antennas:I mt Tx antennas: Apply U(mt)× U(nc) symmetries to reduce
Xi ∼
xi,1 0 0 0
. . .
0 0 xi,m 0
=⇒ equivalent to iid SIMO fading.
I mr Rx antennas: just reduce noise from N0 to N0mr
I =⇒ Achievability/Converse/Asymptotics same as for SISO iid H
I =⇒ STBC, V-BLAST etc – useless (!)
![Page 201: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/201.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Energy-per-bit: conclusion
Punchline:
I AWGN and fading channels with CSIR:
Eb(k , ε) ≈ (−1.59 dB) +
√V
kQ−1(ε)
I Rayleigh fading noCSI:
Eb(k , ε) ≈ (−1.59 dB) +3
√log k
k(Q−1(ε))2
I Unchanged under MIMO and/or block-fading( Optimal design: 1 Tx antenna and no pilots. )
![Page 202: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/202.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Finite blocklength IT codebase:
http://GitHub.com/yp-mit/spectre
(MATLAB, many contributors)
Ask questions (now or later)!
Yury Polyanskiy <[email protected]>
Thank you!
![Page 203: MotivationData compressionChannel codingAnomalous dispersion Finite blocklength …people.lids.mit.edu/yp/homepage/data/ESIT-Tutorial-YP.pdf · 2016-04-12 · MotivationData compressionChannel](https://reader035.vdocument.in/reader035/viewer/2022081606/5e5cc1e4f49b883c4e5e0317/html5/thumbnails/203.jpg)
Motivation Data compression Channel coding Anomalous dispersion
Finite blocklength IT codebase:
http://GitHub.com/yp-mit/spectre
(MATLAB, many contributors)
Ask questions (now or later)!
Yury Polyanskiy <[email protected]>
Thank you!