algebraic structures for distributed source codingdineshk/thesis/ita_09_thesis_overview.pdf ·...

Algebraic Structures for Distributed Source Coding

Dinesh Krithivasan and S. Sandeep Pradhan

University of Michigan

ITA 2009

D. Krithivasan & S.S. Pradhan (U of M) Algebraic Structures for DSC ITA 2009 1 / 21

Introduction

Random coding - common in information theory


Introduction


Structured codes - explored mainly for complexity reasons


Introduction



Structured codes have performance unattainable by random codes


Introduction




Can achieve expurgated bound on the BSC


Introduction





Some works on first order performance gains offered by linear codes


Introduction






This thesis: Unified framework for using structured codes


Introduction







Attains/Exceeds known performance limits for many problems


Introduction







Attains/Exceeds known performance limits for many problems

Suggests practical code constructions


Thesis Overview

Very general framework - includes many famous problems


Thesis Overview


Describe avenues for rate gains


Thesis Overview



Role of structured codes in obtaining these rate gains


Thesis Overview




Gaussian sources - Lattice codes


Thesis Overview





Discrete sources - Group codes


Thesis Overview





Discrete sources - Group codes

Applications to sensor networks, video coding etc.


A Distributed Source Coding Problem

Xn

Y n

f1(·)

f2(·)

Z

Encoder 1

Encoder 2

Decoder

R1

R2

Ed(X, Y, Z) ≤ D

Set of encoders observe different components of a vector source

Central decoder receives quantized observations from the encoders

Decoder interested in minimizing a joint distortion criterion


Joint Distortion Criterion

Distortion criterion depends on all the sources and decoder

reconstructions




reconstructions

Special cases of this framework

Slepian-Wolf problem

Wyner-Ziv problem

Ahlswede-Korner-Wyner problem

Berger-Yeung problem

Berger-Tung problem

Korner-Marton binary XOR problem




reconstructions

Special cases of this framework

Slepian-Wolf problem

Wyner-Ziv problem

Ahlswede-Korner-Wyner problem

Berger-Yeung problem

Berger-Tung problem

Korner-Marton binary XOR problem

Best known rate region - Berger-Tung based


Motivation for our coding scheme

Suppose decoder receives auxiliary random variables U ,V




Encoders don’t communicate with each other: V −Y −X −U





Decoder reconstructs G(U ,V )

G(U ,V ) , arg minG :Z=G(U ,V )d (X ,Y , Z )







Decoder not interested in (U ,V ). Only in G(U ,V )







Decoder not interested in (U ,V ). Only in G(U ,V )

Encode such that decoder can only reconstruct what it needs


An Illustrative Example

X ,Y - 3 bit correlated binary sources, dH (X ,Y )≤ 1




Lossless reconstruction of Z = X ⊕2 Y ∈ {000,001,010,100}





Berger-Tung based coding scheme:

Reconstruct sources X ,Y . Compute Z = X ⊕2 Y







Sum rate: H(X ,Y ) = 5 bits







Sum rate: H(X ,Y ) = 5 bits

Can we do better?


An Illustrative Example contd.

A linear coding scheme:

Z1 ⊕ Z2

Z1 ⊕ Z3

X1X2X3

1 1 0

1 0 1

1 1 0

1 0 1

Y1Y2Y3

X1 ⊕ X2

X1 ⊕ X3

Y1 ⊕ Y2

Y1 ⊕ Y3




Z1 ⊕ Z2

Z1 ⊕ Z3

X1X2X3

1 1 0

1 0 1

1 1 0

1 0 1

Y1Y2Y3

X1 ⊕ X2

X1 ⊕ X3

Y1 ⊕ Y2

Y1 ⊕ Y3

Sum rate: 2+2 = 4 bits = 2H (Z )




Z1 ⊕ Z2

Z1 ⊕ Z3

X1X2X3

1 1 0

1 0 1

1 1 0

1 0 1

Y1Y2Y3

X1 ⊕ X2

X1 ⊕ X3

Y1 ⊕ Y2

Y1 ⊕ Y3


Significant features:

Identical binning at both encoders.




Z1 ⊕ Z2

Z1 ⊕ Z3

X1X2X3

1 1 0

1 0 1

1 1 0

1 0 1

Y1Y2Y3

X1 ⊕ X2

X1 ⊕ X3

Y1 ⊕ Y2

Y1 ⊕ Y3



Identical binning at both encoders. Binning performed by Linear codes




Z1 ⊕ Z2

Z1 ⊕ Z3

X1X2X3

1 1 0

1 0 1

1 1 0

1 0 1

Y1Y2Y3

X1 ⊕ X2

X1 ⊕ X3

Y1 ⊕ Y2

Y1 ⊕ Y3



Identical binning at both encoders. Binning performed by Linear codes

Lossless reconstruction. Entire space is binned


Jointly Gaussian sources and linear function reconstruction

Source (X1, X2) is bivariate Gaussian

X1, X2 ∼N (0,1), E(X1 X2)= ρ > 0




X1, X2 ∼N (0,1), E(X1 X2)= ρ > 0

Decoder interested in Z = X1 −c X2 with mean square distortion, c > 0




X1, X2 ∼N (0,1), E(X1 X2)= ρ > 0

Decoder interested in Z = X1 −c X2 with mean square distortion, c > 0

Berger-Tung based coding scheme

Encoders: Quantize X1 to W1, X2 to W2. Transmit W1,W2

Decoder: Reconstruct Z = E(Z |W1,W2)

Optimal for c < 0 with Gaussian test channels


Nested Lattice codes

Lattice codes: Equivalent to linear codes for continuous sources

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5




Lattice code: points in Rn closed under addition

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5





Need to do quantization first - nested lattice codes

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5






Quantizer - nearest neighbour

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5







Binning - coset leader of the

quantized fine lattice point in

the coarse lattice

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5









the coarse lattice

Fine code - Quantizes the source

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5









the coarse lattice

Fine code - Quantizes the source

Coarse code - bins fine code−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

2.5


Rate region using nested lattice codes

Nested lattice codes Λ2 ⊂Λ11,Λ2 ⊂Λ12




Λ11,Λ12 - good lattice source codes




Λ11,Λ12 - good lattice source codes

Λ2 - good lattice channel code



Achievable tuples satisfy 2−2R1 +2−2R2 ≤

(

σ2z

D

)−1




(

σ2z

D

)−1

For certain sources and distortions: better than Berger-Tung based

scheme




(

σ2z

D

)−1

For certain sources and distortions: better than Berger-Tung based

scheme

Gains based on alignment of [1, −c] with eigenvectors of source

covariance matrixD. Krithivasan & S.S. Pradhan (U of M) Algebraic Structures for DSC ITA 2009 11 / 21

Lattice coding vs Berger-Tung based coding

0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10

2

4

6

8

10

12

14

16

18

20

Distortion D

Sum

rat

e R

1 + R

2

Comparison between the two coding schemes

Berger−Tung Sum RateLattice Sum Rate

ρ = 0.95c = 1


Discrete sources and arbitrary distortions

Arbitrary discrete sources and arbitrary distortions




Fix test channels: PX Y UV = PX Y PU |X PV |Y





Decoder interested only in G(U ,V )






Function G(U ,V ) equivalent to addition in some abelian group G







Abelian groups decomposable into primary cyclic groups Zpr








g ∈G ⇔ g = (g1, . . . , gk ), gi ∈Zpiei








g ∈G ⇔ g = (g1, . . . , gk ), gi ∈Zpiei

Suffices to build codes over Zpr








g ∈G ⇔ g = (g1, . . . , gk ), gi ∈Zpiei

Suffices to build codes over Zpr - nested group codes








g ∈G ⇔ g = (g1, . . . , gk ), gi ∈Zpiei

Suffices to build codes over Zpr - nested group codes

Extension to arbitrary groups through sequential coding


Nested Group codes

Group code over Znpr : C <Z

npr

C = ker(φ) for some homomorphism φ : Znpr →Z

kpr

(C1,C2) nested if C2 ⊂C1


Nested Group codes


npr


kpr


We need:

C1 <Znpr : “good” source code

C2 <Znpr : “good” channel code


Nested Group codes


npr


kpr


We need:


Can find un ∈C1 jointly typical with source xn



Nested Group codes


npr


kpr


We need:


Can find un ∈C1 jointly typical with source xn


Can distinguish between typical channel noise sequences


Group source codes: Example

Consider Z4 = {0,1,2,3}



Consider Z4 = {0,1,2,3}

One non-trivial subgroup 2Z4 = {0,2}



Consider Z4 = {0,1,2,3}


Good group source code for (X ,U ,PXU )



Consider Z4 = {0,1,2,3}



Assume U =Z4



Consider Z4 = {0,1,2,3}



Assume U =Z4

Exists for large n if 1n

log |C1| ≥ log 4−min{H (U |X ),2|H (U |X )−1|+}



Consider Z4 = {0,1,2,3}



Assume U =Z4



Optimal random code’s size: H (U )−H (U |X ) = I (X ;U )



Consider Z4 = {0,1,2,3}



Assume U =Z4



Optimal random code’s size: H (U )−H (U |X ) = I (X ;U )

Penalty for imposing group structure


Group Channel codes: Example

Consider Z4 = {0,1,2,3}




Consider Z4 = {0,1,2,3}


Good group channel code for (Z ,S ,PZ S )



Consider Z4 = {0,1,2,3}



Assume Z =Z4



Consider Z4 = {0,1,2,3}



Assume Z =Z4

Exists for large n if1n log |C2| ≤ log 4−max{H (Z |S),2(H (Z |S)−H ([Z ]1|S))}



Consider Z4 = {0,1,2,3}



Assume Z =Z4


Optimal random code’s size: log 4−H (Z |S)



Consider Z4 = {0,1,2,3}



Assume Z =Z4


Optimal random code’s size: log 4−H (Z |S)

Penalty for presence of non-trivial subgroups


Coding Strategy

Nested group codes C2 <C11,C12

yn

C11

C12

vn

un

xn

H2un

H2vn

H2zn

1n log |C11| ≥ log 4−

min{H (U |X ),2|H (U |X )−1|+}

1n

log |C12| ≥ log 4−

min{H (V |Y ),2|H (V |Y )−1|+}

1n log |C2| ≤ log 4−

max{H (Z ),2(H (Z )−H ([Z ]1))}


Achievable Rates

The set of tuples (R1,R2,D) that satisfy

R1 ≥ max{H (Z ),2(H (Z )−H ([Z ]1))}−min{H (U |X ),2|H (U |X )−1|+}

R2 ≥ max{H (Z ),2(H (Z )−H ([Z ]1))}−min{H (V |Y ),2|H (V |Y )−1|+}

D ≥ Ed (X ,Y ,G(U ,V ))

are achievable.


Achievable Rates

The set of tuples (R1,R2,D) that satisfy

R1 ≥ max{H (Z ),2(H (Z )−H ([Z ]1))}−min{H (U |X ),2|H (U |X )−1|+}

R2 ≥ max{H (Z ),2(H (Z )−H ([Z ]1))}−min{H (V |Y ),2|H (V |Y )−1|+}

D ≥ Ed (X ,Y ,G(U ,V ))

are achievable.

More general rate region possible


Group codes vs Berger-Tung based coding

0.02 0.04 0.06 0.08 0.1 0.12 0.14

1.2

1.4

1.6

1.8

Distortion D

Comparison of the two lower convex envelopes

Sum

Rat

e R

1 + R

2 Berger−Tung based coding schemeGroup code based coding scheme

Reconstruction of XOR with Hamming distortion


Group codes vs Berger-Tung based coding

0.02 0.04 0.06 0.08 0.1 0.12 0.14

1.2

1.4

1.6

1.8

Distortion D

Comparison of the two lower convex envelopes

Sum

Rat

e R

1 + R

2 Berger−Tung based coding schemeGroup code based coding scheme

Reconstruction of XOR with Hamming distortion

Implies Berger-Tung bound not tight for more than 2 users


Other results

Existence of “good” nested lattices - arbitrary finite levels of nesting


Other results


Lossless and lossy compression using abelian group codes - achievable

rates


Other results



rates

Nested linear codes achieve Shannon rate-distortion bound for

arbitrary discrete sources and additive distortions


Other results



rates



Nested linear codes recover known rate regions of


Other results



rates




Berger-Tung problem


Other results



rates




Berger-Tung problem

Wyner-Ziv problem, Wyner-Ahlswede-Korner problem


Other results



rates




Berger-Tung problem


Yeung-Berger problem


Other results



rates




Berger-Tung problem


Yeung-Berger problem

Slepian-Wolf problem, Korner-Marton problem


Future research directions

Extension of lattice coding schemes to arbitrary continuous sources




Construction of practical group codes using LDPC/LDGM codes with

iterative decoding





iterative decoding

Use of structured codes in multi-terminal channel coding problems





iterative decoding


Connections between information theory (typicality) and abstract

algebra (subgroups)





iterative decoding


Connections between information theory (typicality) and abstract

algebra (subgroups)

Codes over non-abelian groups


algebraic structures for distributed source codingdineshk/thesis/ita_09_thesis_overview.pdf ·...

Documents