introduction to coding theory - tu berlin · introduction to coding theory 16. outline 1 code...

Introduction to Coding Theory

Sven PuchingerInstitute of Communications Engineering, Ulm University

COSIP Winter Retreat, 7.12.2016

Introduction to Coding Theory 1

Coding Theory

code node

Encoder Channel Decoder

Coding Theory

code node

Coding Theory

code node

Coding Theory

Code C = Set of codewords dH = Hamming Metric

C = {code, node, core}

dH(code,node) = 1, dH(code, core) = 1, dH(node, core) = 2

C = {simplifications, overgeneralized}

dH(simplifications, overgeneralized) = 13

Encoding (Rate R = kn )

Information Codeword

Coding Theory

Decoding

simplifications

overgeneralized

ovmrleniralxzeddH = 10

dH = 4

Decoding

simplifications

overgeneralized

ovmrleniralxzed

dH = 10

dH = 4

Decoding

simplifications

overgeneralized

ovmrleniralxzeddH = 10

dH = 4

Coding Theory

Claude E. Shannon,A Mathematical Theory ofCommunication, 1948

If R < C, Perr → 0 (n→∞)

Richard Hamming,Error Detection and ErrorCorrection Codes, 1950

First Practical Codes

Coding Theory

Claude E. Shannon,A Mathematical Theory ofCommunication, 1948

If R < C, Perr → 0 (n→∞)

Richard Hamming,Error Detection and ErrorCorrection Codes, 1950

First Practical Codes

Jim Massey’s History of Channel Coding

Experts’ opinions

In the 50s and 60s

Coding is dead! All interesting problems are already solved.

In the 70s

Coding is dead as a doornail, except on the deep-space channel.

In the 80s

Coding is quite dead, except on wideband channels such as thedeep-space channel and narrowband channels such as the telephonechannel.

In the 90s

Coding is truly dead, except on single sender channels.

Applications

CD/DVD/Blu-Ray Mobile Comm. Compressed Sensing

DE78 0192 2932 3627 6328 10

Biology Distributed Storage

Network Coding Cryptography

Alice Bob

S ·G ·P

m m · S ·G ·P+ e

r′ = c+ e+ e′

Helper

Data Ge-

neration

r = c+ e eHelper

Storage

Repro-

duction

r = c+ eHash Key

Applications

Experts’ opinion

Coding is surely dead, except on the deep-space channel, on narrowbandone-sender channels such as the telephone channel, on many senderchannels, storage, networks, ...

Outline

1 Code Constructions

2 Channels & Metrics

3 Decoding

4 Connection to Compressed Sensing

Outline

3 Decoding

K Field (usually finite, e.g. K = F2)

C ⊂ Kn

Goal: Codewords c1, c2 ∈ C “far apart” w.r.t. a metric

d : Kn ×Kn → R≥0

Example: Hamming Metric

dH(x,y) = wtH(x− y), wtH(x) = |supp(x)|

Linear Block Codes

C(n, k, d) ⊆ Kn

Length n

K-subspace of dimension k

Minimum distance d

d = minc1,c2∈Cc1 6=c2

d(c1, c2) = minc∈C\{0}

Singleton Bound d ≤ n− k + 1

Generator & Parity Check Matrix

Generator Matrix = A Basis of C

c1...ck

⇒ C =

i cG :

i∈ Kk

}Dual Code C⊥ =

{c⊥ : 〈c⊥, c〉 = 0 ∀c ∈ C

}(orth. complement)

Parity Check Matrix = A Basis of C⊥

c⊥1...

c⊥n−k

n− k

⇒ C ={c ∈ Kn : HcT = 0

c1...ck

⇒ C =

i cG :

i∈ Kk

Dual Code C⊥ ={c⊥ : 〈c⊥, c〉 = 0 ∀c ∈ C

}(orth. complement)

c⊥1...

c⊥n−k

n− k

⇒ C ={c ∈ Kn : HcT = 0

c1...ck

⇒ C =

i cG :

i∈ Kk

}Dual Code C⊥ =

{c⊥ : 〈c⊥, c〉 = 0 ∀c ∈ C

}(orth. complement)

c⊥1...

c⊥n−k

n− k

⇒ C ={c ∈ Kn : HcT = 0

c1...ck

⇒ C =

i cG :

i∈ Kk

}Dual Code C⊥ =

{c⊥ : 〈c⊥, c〉 = 0 ∀c ∈ C

}(orth. complement)

c⊥1...

c⊥n−k

n− k

⇒ C ={c ∈ Kn : HcT = 0

c1...ck

⇒ C =

i cG :

i∈ Kk

}Dual Code C⊥ =

{c⊥ : 〈c⊥, c〉 = 0 ∀c ∈ C

}(orth. complement)

c⊥1...

c⊥n−k

n− k

⇒ C ={c ∈ Kn : HcT = 0

}Introduction to Coding Theory 13

Example 1: Repetition Code

K = F2

C(n, 1, n) = {(00 . . . 0), (11 . . . 1)}

Generator Matrix

G =(1 1 . . . 1

)∈ F1×n

Parity Check Matrix

. . ....

∈ Fn−1×n2

Example 2: Reed–Solomon Codes

α1, . . . , αn ∈ K distinct

C(n, k) ={(f(α1), . . . , f(αn)

): f(x) ∈ K[x], deg f(x) < k

Theorem

d = n− k + 1

Proof: Singleton Bound: d ≤ n− k + 1.c 6= 0⇒ f(x) 6= 0 and deg f(x) ≤ k − 1⇒ At most k − 1 many αi give f(αi) = 0⇒ At least n− (k − 1) many αi give f(αi) 6= 0⇒ wtH(c) ≥ n− k + 1

C(n, k) ={(f(α1), . . . , f(αn)

): f(x) ∈ K[x], deg f(x) < k

}Theorem

d = n− k + 1

C(n, k) ={(f(α1), . . . , f(αn)

): f(x) ∈ K[x], deg f(x) < k

}Theorem

d = n− k + 1

Code Classes (Selection)

ConvolutionalCodes

BlockCodes

Reed–Solomon(+) Decoding(–) n ≤ |K|

LDPC(+) Achieve C (n→∞)(–) Few analytical tools

Reed–Muller

CodeConcatenation

There is no (known) universally superior code!

ConvolutionalCodes

BlockCodes

Reed–Muller

CodeConcatenation

ConvolutionalCodes

BlockCodes

Reed–Muller

CodeConcatenation

ConvolutionalCodes

BlockCodes

Reed–Muller

CodeConcatenation

ConvolutionalCodes

BlockCodes

Reed–Muller

CodeConcatenation

ConvolutionalCodes

BlockCodes

Reed–Muller

CodeConcatenation

ConvolutionalCodes

BlockCodes

Reed–Muller

CodeConcatenation

Outline

3 Decoding

Channels

General

c P(r | c) r

Binary Symmetric Channel (0 < ε < 0.5)

1− ε

ei ∼ Ber(ε)

ri = ci + ei⊕⇐⇒

⇒ Hamming Metric

Channels

General

c P(r | c) r

1− ε

ei ∼ Ber(ε)

⇒ Hamming Metric

Channels

General

c P(r | c) r

1− ε

ei ∼ Ber(ε)

⇒ Hamming Metric

Channels

General

c P(r | c) r

1− ε

ei ∼ Ber(ε)

⇒ Hamming Metric

Channels

AWGN Channel

ei ∼ N (0, σ2)

ri = ci + ei⊕

⇒ Euclidean Metric

Rank Metric

dR : Km×n ×Km×n → N0

(X,Y ) 7→ rank(X − Y )

Corresponding Channels

Random Linear Network CodingMIMO Transmission Systems

Channels

AWGN Channel

ei ∼ N (0, σ2)

ri = ci + ei⊕

Rank Metric

(X,Y ) 7→ rank(X − Y )

Channels

AWGN Channel

ei ∼ N (0, σ2)

ri = ci + ei⊕

Rank Metric

(X,Y ) 7→ rank(X − Y )

Channels

AWGN Channel

ei ∼ N (0, σ2)

ri = ci + ei⊕

Rank Metric

(X,Y ) 7→ rank(X − Y )

Outline

3 Decoding

Decoding

c P(r | c) r

Maximum-Likelihood (ML) Decoding

c = argmaxc∈C

P(c | r) = argmaxc∈C

P(r | c)P(c)P(r)

If metric d fits to channel

c = argminc∈C

d(r, c)

In Practice

Convolutional Codes: ML-decodable (Viterbi Algorithm)

Block Codes: Often not ML-decodableE.g. C(400, 272, d) code over F2: |C| ≈ 1082

Decoding

c P(r | c) r

c = argmaxc∈C

P(r | c)P(c)P(r)

c = argminc∈C

d(r, c)

In Practice

Decoding

c P(r | c) r

c = argmaxc∈C

P(r | c)P(c)P(r)

c = argminc∈C

d(r, c)

In Practice

Bounded Minimum Distance Decoding

Example: Reed–Solomon Codes

r =(f(α1), . . . , f(αn)

Welch–Berlekamp

Find a non-zero Q(x, y) = Q0(x) + yQ1(x) ∈ K[x, y] s.t.

i) Q(αi, ri) = 0 ∀iii) degQ0(x) ≤ n− 1− d

iii) degQ1(x) ≤ n− k − d2

If dH(r, c) <d2 , then

Q(x, y) exists

f(x) = −Q0(x)Q1(x)

Complexity

Naive O(n3)

Practical O(n2)

Optimal O∼(n)

Beyond d2?

r =(f(α1), . . . , f(αn)

Welch–Berlekamp

Q(x, y) exists

f(x) = −Q0(x)Q1(x)

Complexity

Naive O(n3)

Practical O(n2)

Optimal O∼(n)

Beyond d2?

r =(f(α1), . . . , f(αn)

Welch–Berlekamp

Q(x, y) exists

f(x) = −Q0(x)Q1(x)

Complexity

Naive O(n3)

Practical O(n2)

Optimal O∼(n)

Beyond d2?

r =(f(α1), . . . , f(αn)

Welch–Berlekamp

Q(x, y) exists

f(x) = −Q0(x)Q1(x)

Complexity

Naive O(n3)

Practical O(n2)

Optimal O∼(n)

Beyond d2?

r =(f(α1), . . . , f(αn)

Welch–Berlekamp

Q(x, y) exists

f(x) = −Q0(x)Q1(x)

Complexity

Naive O(n3)

Practical O(n2)

Optimal O∼(n)

Beyond d2?

List Decoding

Outline

3 Decoding

Connection to Compressed Sensing

Compressed Sensing (m� n)

Given b ∈ Cm,A ∈ Cm×n, find sparsest x ∈ Cn s.t.

b = Ax

Decoding Problem (n− k � n)

Given s ∈ Kn−k, H ∈ Kn−k×n, find e ∈ Kn of minimal wtH(e) s.t.

s = He

Solution:

Find some solution r = c+ e of

s = He = H(c+ e) = Hr.

Decode r −→ obtain c, e with minimal dH(r, c) = wtH(e)

b = Ax

s = He

Solution:

s = He = H(c+ e) = Hr.

b = Ax

s = He

Solution:

s = He = H(c+ e) = Hr.

introduction to coding theory - tu berlin · introduction to coding theory 16. outline 1 code...

Documents