information engineering 2 coding theory nt 4
TRANSCRIPT
1
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 1FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Coding Theory
NT 4Prof. Dr.-Ing. Andreas Czylwik
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 2FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Organization
Lecture 2 hours per weekTransparencies will be available at:http://nts.uni-duisburg.deExercise 1 hour per week, Dipl.-Ing. Batu ChaliseWritten examination
New research areas in the Department of CommunicationSystems (Nachrichtentechnische Systeme)Subjects for master theses
2
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 3FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Textbooks
Textbooks for the lecture:M. Bossert: Channel coding for telecommunications, John WileyBlahut: Theory and practice of error control codes, Addison-WesleyJ. H. van Lint: Introduction to coding theoryB. Friedrichs: Kanalcodierung, Springer-VerlagH. Schneider-Obermann: Kanalcodierung, Vieweg-Verlag
Textbooks in the field of digital communications incl. coding theory:S. Benedetto, E. Biglieri, V. Castellani: Digital transmission theory, Prentice-HallJ.G. Proakis: Digital communications, McGraw-HillS. Haykin: Communication systems, John Wiley
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 4FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Contents
IntroductionInformation theoryChannel coding in digital communication systemsAlgebraic foundations for codingBlock codesConvolutional codesCoding techniquesOutlook
3
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 5FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Block diagram of a digital communication system
Information Engineering 2Introduction
Source Sourceencoder
Channelencoder Modulator
Transmissionchannel Noise
DemodulatorChanneldecoder
SourcedecoderSink
Digital source
Digital sinkDiscrete channel
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 6FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Introduction
Source codingCompression of the communication signal to a minimum numberof symbols without the loss of information (reduction ofredundancy)Further compression when the loss of information is tolerated (e.g.video or audio transmission)
Coding for encryption Security against unwanted listeningInformation recovery only with a secret key
4
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 7FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Introduction
Error control coding: defined addition of redundancy toimprove transmission quality
FEC (forward error correction)Simplified model:
Channel quality determines residual error rate after decodingData rate does not depend on channel quality
Datasource
Channelencoder Channel Channel
decoderDatasink
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 8FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Introduction
Channel coding for error detection (CRC – cyclic redundancycheck, used for automatic repeat request methods (ARQ))
Reverse channel necessaryAdaptive insertion of redundancy (addition redundancy only incase of errors)Residual error probability does not depend on channel qualityNet data rate (throughput) depends on channel quality
ARQcontrol
Channelencoder Channel Channel decoder
(error detection)ARQ
control
Error informationBackward channel
5
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 9FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Introduction
Application: secure data transmission via wave guides or radio channels (especially mobile radio channels), secure data storageFather of information theory − Claude E. Shannon:Using channel coding the error probability can be reducedto an arbitrarily small value, if the data rate is smaller than the channel capacity. (Shannon does not present aconstruction method.)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 10FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Introduction
Basic idea of channel coding:Insertion of redundancyGoal: error detection or error correctionBlock encoding process
Input vector: u = (u1, ... , uk)Output vector: x = (x1, ... , xn)Code rate:
nkR =C (1)
Length k Length n
Input vector u Output vector x
6
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 11FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Introduction
Code cube with n = 3, k = 3
Uncoded transmission: RC = 1Smallest distance between code words: dmin = 1Error detection or error correction not possible
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 12FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Introduction
Code cube with n = 3, k = 2
Coded transmission: RC = 2/3Smallest distance between code words: dmin = 2Detection of a single error is possible – error correction is not possible
7
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 13FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Introduction
Code cube with n = 3, k = 1
Coded transmission: RC = 1/3Smallest distance between code words: dmin = 3Detection of two errors and correction of a single error is possible
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 14FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Information theory: mathematical description ofinformation and of its transmissionCentral questions:
Quantitative calculation of the information content of messagesCalculation of the capacity of transmission channelsAnalysis and optimization of source and channel coding
8
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 15FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Messages from the point of view of information theory
irrelevant relevant
errorinformation
redundant non redundant
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 16FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Information content of a message: entropyQualitative ordering of messages: a message is more important if it is more difficult to predict it
Example:Tomorrow, the sun rises.Tomorrow, there will be bad weather.Tomorrow, there will be a heavy thunderstorm so that the electric power network will break down.
Messages from a digital source: sequence of symbols
Importance
Probability
9
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 17FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Number of binary decisions that is needed to select amessage (symbol) from a source: H0
Total number of symbols: N
H0 = ld(N) bit/symbol
with ld(x) = logarithm to base 2 (logarithmus dualis) Unit: bit = binary digitExample: English text as a source
26 ⋅ 2 characters, 14 special characters including „space“: ‘ ’ “ ” ( ) - . , ; : ! ?In total: 66 alphanumeric charactersH0 = ld(66) = ln(66) / ln(2) = 6.044 bit/character
Information Engineering 2Information theory
(2)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 18FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Example: one page of English text with 40 lines and 70 charactersper line
Number of different pages: N = 6640⋅70
Number of binary decisions to select a page: H0 = ld(6640⋅70) = 40 ⋅ 70 ⋅ ld (66) = 16.92 kbit/page
The number of binary decisions H0 does not take into account the probability of symbols!
10
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 19FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Source alphabet: X = {x1, ... , xN}Probabilities of the symbols: p(x1), ... , p(xN)Desired properties of the information content I = f(p):
I(xi) ≥ 0 for 0 ≤ p(xi) ≤ 1I(xi) → 0 for p(xi) → 1I(xi) > I(xj) for p(xi) < p(xj)Two subsequent statistically independent symbols xi and xj
with p(xi,xj) = p(xi) ⋅ p(xj) :I(xi,xj) = I(xi) + I(xj)
General solution: I(xi) = −k ⋅ logb(p(xi)) (3)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 20FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Definition of the information content:
Entropy H(X) = average information content of a source:
bit/symbol)(
1ld)(
=
ii xp
xI
bit/symbol)(
1ld)(
)()()()(
1
1
∑
∑
=
=
⋅=
⋅==
N
i ii
N
iiii
xpxp
xIxpxIXH
(4)
(5)
11
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 21FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Example: binary source with X = {x1, x2}Probabilities: p(x1) = p , p(x2) = 1 − pEntropy: H(X) = −p ld(p) − (1 − p) ld(1 − p)Shannon function: H(X) = S(p)
(6)
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
S(p) / bit
p
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 22FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
The entropy becomes maximum for equiprobable symbols: ⇒ H(X) ≤ H0 = ld N.
Proof:
Using the relation:
∑
∑∑
=
==
⋅⋅=
⋅−⋅=−
N
i ii
N
ii
N
i ii
Nxpxp
Nxpxp
xpNXH
1
11
)(1ld)(
ld)()(
1ld)(ld)(
1ln −≤ xx
(7)
(8) -1.0
0.0
1.0
1.0 2.0 x
ln(x)
x − 1
12
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 23FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
with
■
Redundancy of a source: RS = H0 − H(X)
0112ln
1)(12ln
1
1)(
12ln
1)(ld)(
1
1
=
−⋅=
−≤
−
⋅⋅≤−
∑
∑
=
=
NNxp
N
NxpxpNXH
N
ii
N
i ii
)1(2ln
1ld1ln −≤⇒−≤ xxxx
(9)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 24FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Transmission via a binary channel:design of binary coding schemes for a discrete sourceTasks of source coding:
Assignment of binary code words of length L(xi) to all symbols xi
Minimizing the average length of the code words
∑=
⋅==N
iiii xLxpxLL
1)()()(
L x1 1001x2 011
xN 010111L(xN)
....
....
(10)
13
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 25FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Examples for binary coding of symbols:ASCII-Code: fixed code word length L(xi) = 8 (block code)Morse code (alphabet with dots, dashes and pauses for the separation of code words): more frequently occuring characters are assigned to shorter code words
Prefix condition for codes with variable length: no codeword equals the prefix of any other longer codeword
Example for a code without prefix condition:x1 → 0x2 → 01x3 → 10x4 → 100
Unique decoding of a bit sequence is not possible!Possible decoding results for the sequence010010: x1x3x2x1, x2x1x1x3, x1x3x1x3, x2x1x2x1,
x1x4x3
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 26FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Example for a code with prefix condition:
The code is uniquely and instantaneously decodable!Decoding of the sequence 010010110111100: x1x2x1x2x3x4x2x1
Synchronization: begin of the sequence
x1 → 0x2 → 10x3 → 110x4 → 111
14
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 27FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Decoding with a binary tree:
x1 → 0x2 → 10x3 → 110x4 → 111
Level 1: L(x1) = 1
0 1
0 1
0 1Level 2: L(x2) = 2
Level 3: L(x3) = L(x4) = 3
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 28FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Kraft inequality: a binary code with the code word lengths L(x1), L(x2), ... , L(xN) fulfilling the prefix condition exists only, if:
Proof:Length of the tree structure = maximum code word lengthLmax = max(L(x1), L(x2), ... , L(xN))A code word in level L(xi) eliminates of all possible code words in level Lmax .Sum over all eliminated code words ≤ maximum number in the level Lmax
■
Information Engineering 2Information theory
121
)( ≤∑=
−N
i
xL i
)(max2 ixLL −
maxmax 221
)( LN
i
xLL i ≤∑=
−
(11)
15
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 29FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
The equality holds, if all ends of the code tree are used by code words. General relation:Example for the special case that all probabilities are given by powers of 2:
Assignment of code words corresponding to the relation:L(xi) = Ki
Special case:
Information Engineering 2Information theory
( ) iKixp 2
1)( =
)(XHL ≥
815)( == XHL
xi p(xi) code words x1 1/2 1 x2 1/4 00 x3 1/8 010 x4 1/16 0110 x5 1/16 0111
(12)
(13)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 30FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Shannons coding theorem: For any source a binary coding can be found with :
Proof:left-hand side: average length of code words ≥ average information content (entropy)right-hand side: selection of code words with
I(xi) ≤ L(xi) ≤ I(xi) + 1Multiplication with p(xi) and summing up over all i ⇒ (14)
1)()( +≤≤ XHLXH (14)
(15)
16
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 31FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Proof, that a code with the prefix condition (11) exists:left-hand side of (15):
Inserting in (11): ■
Shannon CodingCode word length corresponding (15): I(xi) ≤ L(xi) ≤ I(xi) + 1Accumulated probabilities:
Information Engineering 2Information theory
1)(211
)( ∑∑==
− =≤N
ii
N
i
xL xpi
)()(
1 2)()(ld)( ii
xLiixpi xpxLxI −≥⇒≤
=
∑−
==
1
1)(
i
jji xpP
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 32FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Code words are decimals of Pi in binary notation:
Example:0.90 = 1⋅2−1 + 1⋅2−2 + 1⋅2−3 + 0⋅2−4 + 0⋅2−5 + 1⋅2−6 + 1⋅2−7 + 0⋅2−8 +0⋅2−9 + 1⋅2−10 + ...
Information Engineering 2Information theory
i p(xi) I(xi) L(xi) Pi code 1 0.22 2.18 3 0.00 000 2 0.19 2.40 3 0.22 001 3 0.15 2.74 3 0.41 011 4 0.12 3.06 4 0.56 1000 5 0.08 3.64 4 0.68 1010 6 0.07 3.84 4 0.76 1100 7 0.07 3.84 4 0.83 1101 8 0.06 4.06 5 0.90 11100 9 0.04 4.64 5 0.96 11110
bit54.3=L
17
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 33FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Tree representation of a Shannon code:
Disadvantage: not all ends of the tree are used for code words
Information Engineering 2Information theory
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 34FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Lengths of code words can be reduced ⇒ the code is not optimumRedundancy of a code:Redundancy of a source:
Huffman CodingRecursive procedure Starting point: symbols with smallest probabilitiesSame code word lengths for the two symbols with smallest probabilitiesHuffman coding minimizes the average code word length
Information Engineering 2Information theory
)(C XHLR −=)(0Q XHHR −=
18
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 35FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Algorithm:
Step 1: Ordering of symbols corresponding to their probability
Step 2: Assignment of 0 and 1 to the two symbols with lowest probability
Step 3: Combination of the two symbols with lowest probability xN−1 and xN to a new symbol with probability p(xN−1 ) + p( xN)
Step 4: Repetition of steps 1 - 3, until only one symbol is left
Example
Information Engineering 2Information theory
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 36FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
x1 x2 x3 x4 x5 x6 x7 x8 x9 0.22 0.19 0.15 0.12 0.08 0.07 0.07 0.06 0.04
0 1
x1 x2 x3 x4 x8 x9 x5 x6 x7 0.22 0.19 0.15 0.12 0.10 0.08 0.07 0.07
0 1 0 1
x1 x2 x3 x6 x7 x4 x8 x9 x5 0.22 0.19 0.15 0.14 0.12 0.10 0.08
0 1 00 01 1
x1 x2 x8 x9 x5 x3 x6 x7 x4 0.22 0.19 0.18 0.15 0.14 0.12
00 01 1 00 01 1
19
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 37FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
x6 x7 x4 x1 x2 x8 x9 x5 x3 0.26 0.22 0.19 0.18 0.15
00 01 1 000 001 01 1
x8 x9 x5 x3 x6 x7 x4 x1 x2 0.33 0.26 0.22 0.19
000 001 01 1 00 01 1 0 1
x1 x2 x8 x9 x5 x3 x6 x7 x4 0.41 0.33 0.26
0 1 0000 0001 001 01 100 101 11
x8 x9 x5 x3 x6 x7 x4 x1 x2 0.59 0.41
00000 00001 0001 001 0100 0101 011 10 11
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 38FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Tree structure of the Huffman code example
Information Engineering 2Information theory
bit01.3=L
20
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 39FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Discrete source without memoryJoint entropy of sequences of symbols
Two independent symbols: p(xi,yk) = p(xi) ⋅ p(yk)
Information Engineering 2Information theory
[ ]
)()(),(
))((ld)()())((ld)()(
))((ld))((ld)()(
)),((ld),(),(
1´111
1 1
1 1
YHXHYXH
ypypxpxpxpyp
ypxpypxp
yxpyxpYXH
N
kkk
N
ii
N
iii
N
kk
N
i
N
kkiki
N
i
N
kkiki
+=
⋅⋅−⋅⋅−=
+⋅⋅−=
−=
∑∑∑∑
∑∑
∑∑
====
= =
= =
(17)
(16)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 40FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
H(X1,...,XM) ≤ ≤ H(X1,...,XM) + 1
M⋅H(X) ≤ ≤ M⋅H(X) + 1
H(X) ≤ ≤ H(X) + 1/M
M independent symbols from the same source:
H(X1, X2, ... , XM) = M ⋅ H(X)
More efficient coding by coding of symbol sequencesShannon coding theorem:
Disadvantage of coding of symbol sequences: rapidly increasing computational effort (exponential increase of number of combined symbols)
Information Engineering 2Information theory
),,( 1 MM XXL K
LM ⋅
L
(18)
(19)
21
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 41FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Example: Coding of symbol sequencesBinary source with X = {x1, x2}Probabilities: p(x1) = 0.2, p(x2) = 0.8Entropy: H(X) = 0.7219 bits/symbolCoding of single symbols:
Average code word length:
Information Engineering 2Information theory
symbol p(xi) code L(xi) p(xi)⋅L(xi) x1 0.2 0 1 0.2 x2 0.8 1 1 0.8
Σ = 1
bit/symbol 1=L
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 42FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Coding of pairs of symbols:
Average code word length:
Information Engineering 2Information theory
pair of symbols
p(xi) code L(xi) p(xi)⋅L(xi)
x1x1 0.04 101 3 0.12 x1x2 0.16 11 2 0.32 x2x1 0.16 100 3 0.48 x2x2 0.64 0 1 0.64
Σ = 1.56
bit/symbol 78.0=L
22
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 43FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Coding of triplets of symbols:
Average Code word length:
Information Engineering 2Information theory
triplet of symbols
p(xi) code L(xi) p(xi)⋅L(xi)
x1x1x1 0.008 11111 5 0.040 x1x1x2 0.032 11100 5 0.160 x1x2x1 0.032 11101 5 0.160 x1x2x2 0.128 100 3 0.384 x2x1x1 0.032 11110 5 0.160 x2x1x2 0.128 101 3 0.384 x2x2x1 0.128 110 3 0.384 x2x2x2 0.512 0 1 0.512
Σ = 2.184
bit/symbol 728.0=L
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 44FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Discrete source with memoryReal sources: correlation between individual symbolsTwo correlated symbols: p(xi,yk) = p(xi) ⋅ p(yk|xi) = p(yk) ⋅ p(xi|yk)
Information Engineering 2Information theory
[ ]
)|()(),(
))|((ld),())((ld)()|(
))|((ld))((ld)|()(
)),((ld),(),(
1 11 1
1 1
1 1
XYHXHYXH
xypyxpxpxpxyp
xypxpxypxp
yxpyxpYXH
N
i
N
kikki
N
i
N
kiiik
N
i
N
kikiiki
N
i
N
kkiki
+=
⋅−⋅⋅−=
+⋅⋅−=
−=
∑∑∑∑
∑∑
∑∑
= == =
= =
= =
(20)
23
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 45FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
H(Y | X) = conditional entropy
Entropy of a symbol ≥ conditional entropy
H(Y) ≥ H(Y | X )
Proof:
Information Engineering 2Information theory
(21)∑∑= =
⋅−=N
i
N
kikki xypyxpXYH
1 1))|((ld),()|(
(22)
∑∑
∑∑∑
= =
= ==
⋅=−
⋅−=⋅−=
N
i
N
k ik
kki
N
i
N
kkki
N
kkk
xypypyxpYHXYH
ypyxpypypYH
1 1
1 11
)|()(ld),()()|(
))((ld),())((ld)()(
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 46FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Using the inequality:
with p(xi,yk) = p(xi) ⋅ p(yk|xi)
■
Information Engineering 2Information theory
∑∑= =
−⋅⋅≤−
N
i
N
k ik
kki xyp
ypyxpYHXYH1 1
1)|(
)(2ln
1),()()|(
)1(2ln
1ld1ln −≤⇒−≤ xxxx
0),()()(2ln
1)()|(1 11 1
=
−≤− ∑∑∑∑
= == =
N
i
N
kki
N
i
N
kki yxpypxpYHXYH
= 1 = 1
24
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 47FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Entropy of a source with memory < entropy of a source withoutmemoryVery efficient source coding for sequences of symbols withmemory
General description of a discrete source with memory as a Markov source
Markov processes:Sequence of random variables z0, z1, z2, ... ,zn, (n = time axis)zi and zj are statistically independent:
Information Engineering 2Information theory
)(),...,,|( 021,...,,| 021 nznnnzzzz zfzzzzfnnnn
=−−−− (23)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 48FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
zi and zj are stat. dependent (Markov process of mth order):
Often: first order Markov process (m = 1):
zi are from a limited number of discrete values: zi ∈ {x1, ... , xN}
⇒ Markov chain Complete description of a Markov chain by transitionprobabilities:
Information Engineering 2Information theory
)|(),...,,|( 1|021,...,,| 1021 −−− −−−= nnzznnnzzzz zzfzzzzf
nnnnn
),...,|(),...,|(101 101 mnnn imninjniinjn xzxzxzpxzxzxzp
−−−======= −−−
(25)
(26)
),...,|(),...,,|( 1,...,|021,...,,| 1021 mnnnzzznnnzzzz zzzfzzzzfmnnnnnn −−−− −−−−
=
(24)
25
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 49FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Homogeneous Markov chain: transition probabilities do notdepend on time:
stationary Markov chain: steady-state does not depend on initialprobabilities
Homogeneous and stationary Markov chain of first order (m = 1):
Information Engineering 2Information theory
jjjnkn
ikjnkn
wxpxzpxzxzp ====== +∞→
+∞→
)()(lim)|(lim
ijinjn pxzxzp === − )|( 1
(28)
(29)
),...,|(),...,|(11 11 mm imkikjkimninjn xzxzxzpxzxzxzp ======= −−−−
(27)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 50FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Transition matrix:
Properties of the transition matrix:
Probability vector:
Calculation of w utilizing the steady state condition:
Information Engineering 2Information theory
11
=∑=
N
jijp
))()()(()( 2121 NN xpxpxpwww LL ==w
Pww =
(31)
(32)
(33)
=
NNNN
N
N
ppp
pppppp
L
LOLL
L
L
21
22221
11211
P (30)
26
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 51FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Stationary Markov source: first order Markov chainEntropy of a stationary Markov source = steady state entropy:
Information Engineering 2Information theory
∑ ∑
∑∑
∑
= =−−
= =−−
=−−
−−−∞→
∞
==⋅==⋅−=
==⋅==−=
=⋅===
==
N
i
N
jinjninjni
N
i
N
jinjninjn
N
iinniiinn
nnnnnn
xzxzpxzxzpw
xzxzpxzxzp
xzzHwxzzH
zzHzzzzHZH
1 111
1 111
111
1021
))|((ld)|(
))|((ld),(
)|()|(
)|(),,,|(lim)( K
(37)
(34)
(35)
(36)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 52FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
⇒ Entropy H∞(Z) = conditional entropy, since symbols fromthe past are already known
Coding of a Markov source:Taking into account the memorye. g. Huffman coding which takes into account theinstantaneous state of the source
Fundamental problem for variable length source codes: catastrophic error propagation
Information Engineering 2Information theory
)()()|()( 01 nnnn zHzHzzHZH ≤≤= −∞ (38)
27
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 53FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Example of a Markov source:States = transmitted symbolszn ∈ {x1, x2, x3}
Transition probabilities:
Information Engineering 2Information theory
( ) ( )
===== −
3.01.06.02.05.03.04.04.02.0
)|( 1 Pijinjn pxzxzp
zn zn−1
x1 x2 x3
x1 0.2 0.4 0.4 x2 0.3 0.5 0.2 x3 0.6 0.1 0.3
x1 x2
x3
0.20.3
0.4
0.2
0.1
0.4
0.5
0.3
0.6
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 54FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Calculation of steady-state probabilities with:
w1 = 0.2 w1 + 0.3 w2 + 0.6 w3
w2 = 0.4 w1 + 0.5 w2 + 0.1 w3
w3 = 0.4 w1 + 0.2 w2 + 0.3 w3
1 = w1 + w2 + w3
Linear dependence !Solution:
Information Engineering 2Information theory
1 and 1
== ∑=
N
iiwPww
3011.03441.03548.0 9328
39332
29333
1 ≈=≈=≈= www
28
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 55FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Calculation of the entropy with (35):
Inserting numbers:
Information Engineering 2Information theory
∑ ∑
∑
= =
=−∞
⋅⋅−=
=⋅=
N
i
N
jijiji
N
iinni
ppw
xzzHwZH
1 1
11
)(ld
)|()(
(39)
symbolbit / 2955.1ld3.0ld1.0ld6.0)|(
symbolbit / 4855.1ld2.0ld5.0ld3.0)|(
symbolbit / 5219.1ld4.0ld4.0ld2.0)|(
3.01
0.11
0.61
31
2.01
0.51
0.31
21
4.01
0.41
0.21
11
≅⋅+⋅+⋅==
≅⋅+⋅+⋅==
≅⋅+⋅+⋅==
−
−
−
xzzH
xzzH
xzzH
nn
nn
nn
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 56FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Entropy:
For comparison:
Information Engineering 2Information theory
symbol/bit441.1)|()|()|()( 313212111
≅=+=+== −−−∞ xzzHwxzzHwxzzHwZH nnnnnn
symbol/bit5814.11ld)(lim)(1
≅⋅== ∑=∞→
N
i iin
n wwzHZH
symbol/bit5850.13ld0 ≅=H
(40)
29
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 57FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
State-dependent Huffman coding for the example: different codings for each state zn−1
pij code words
average code word length:
Information Engineering 2Information theory
zn
zn−1
x1 x2 x3 ⟨L⟩|zn−1
x1 11 10 0 1.6 x2 10 0 11 1.5 x3 0 11 10 1.4
symbol/bit5054.1
)|()|(1 1
11
≅
==⋅⋅==== ∑∑= =
−−N
i
N
jinjnijiinjn xzxzLpwxzxzLL
(41)
zn zn−1
x1 x2 x3
x1 0.2 0.4 0.4 x2 0.3 0.5 0.2 x3 0.6 0.1 0.3
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 58FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Source coding without knowledge of statistical parametersRun-length coding
Replacing sequences of repeated symbols with the count and only one single symbolExample:
Source sequence:aaaabbbccccccccdddddeeeeeaaaaaaabddddd.....Encoded sequence:4a3b8c5d5e7a1b5d.....
Encoding with dictionariesIdea: repetitions within the data sequence are replaced by (shorter) references to the dictionary
Information Engineering 2Information theory
30
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 59FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Static dictionary: little compression for most data sourcespoor matching of the dictionary to particular data
Semi-adaptive dictionarydictionary tailored for the message to be encodeddictionary has to transmitted via the channeltwo passes over the data:
» to build up the dictionary» to encode the data
Adaptive dictionarysingle pass for building up the dictionary and encodingLempel Ziv algorithm: *.zip *.gzip files
Information Engineering 2Information theory
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 60FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Lempel Ziv algorithm (LZ77)
Search for the longest match between the symbols of the look-ahead buffer and a sequence within the search bufferFixed-length codewords contain:
position of match (counting from 0) length of matchnext symbol in the look-ahead buffer
Information Engineering 2Information theory
search buffer look-ahead buffer
..... a b c b a a b c b d f e e a a b a a c c d d d c d .....
sliding window
31
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 61FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Parameters:symbol alphabet: {x0,x1,x2, .... , xα−1}input sequence: S = {S1,S2,S3,S4, ....}length of look-ahead buffer: LS
window size: n
Codewords: Ci = {pi,li,Si}position of match: pi
length of match: li
next symbol: Si
length of codewords:same alphabet for data as well as codewords
Information Engineering 2Information theory
1)(log)(log SSC ++−= LLnL αα
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 62FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Example:symbol alphabet: {0,1,2}input sequence: S = {0010102102102120210212001120.....}length of look-ahead buffer: LS = 9window size: n = 18Step 1:
Step 2:
Information Engineering 2Information theory
1 0 2 10 0 0 0 0 0 0 0 0 0 2 1 0 2 . . . .1000
C1 = {22 02 1}
1 0 2 10 0 0 0 0 0 0 1 2 0 2 1 2 0 . . . .0100
C2 = {21 10 2}
32
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 63FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Step 3:
Step 4:
Number of encoded source symbols after 4 steps: 3 + 4 + 8 + 9 = 24Number of code word symbols after 4 steps: 4 × 5 = 20
Information Engineering 2Information theory
0 2 1 20 0 0 1 0 1 0 2 1 0 2 1 0 2 . . . .2010
C3 = {20 21 2}
2 1 2 01 0 2 1 0 2 1 2 0 0 1 1 2 0 . . . .1202
C4 = {02 22 0}
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 64FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Decoding:Step 1: C1 = {22 02 1}
Step 2: C2 = {21 10 2}
Step 3: C3 = {20 21 2}
Step 4: C4 = {02 22 0}
Information Engineering 2Information theory
0 0 0 0 0 0 0 0 1000
0 0 0 0 0 0 0 1 20100
0 2 1 20 0 0 1 0 1 0 2 12010
2 1 2 01 0 2 1 0 2 1 2 0 01202
33
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 65FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Transmission of information via a discrete memorylesschannel
Noiseless channel: information at the output = information at the input ⇒ transmitted information = entropy of the sourceNoisy channel: information at the output < information at the input ⇒ transmitted information < entropy of the sourceDefinition:average mutual information = actually transmitted information
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 66FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Discrete memoryless channel (DMC)
Input signal:
Output signal:
},...,,{ 21 XNxxxX ∈
},...,,{ 21 YNyyyY ∈
discretememoryless
channel
X Y
p11
p21
p12
p22
x1
y2
xNX
x2
y1
yNYpNXNY
34
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 67FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Transition matrix:
with
Input probabilities:
Output probabilities:
Relation between input and output probabilities:
( ) ( )
=====
YXXX
Y
Y
NNNN
N
N
ijij
ppp
pppppp
pxXyYp
L
LOLL
L
L
21
22221
11211
)|(P
11
=∑=
N
jijp
(42a)
(42b)
Ppp ⋅= XY
( ))(),...,(),( 21 XNX xpxpxp=p
( ))(),...,(),( 21 YNY ypypyp=p
(42c)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 68FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Chain of two channels:
Output probabilities:
Resulting transition matrix:
(45))()( 2121 PPpPPpp ⋅⋅=⋅⋅= XXZ
(46)
P1
X YP2
Z
21 PPP ⋅=
35
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 69FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Binary channel:
Probabilities at input and output:
Error probability of a binary channel:
212121
212121)()(
)|()()|()()error(pxppxp
xypxpxypxpp⋅+⋅=
⋅+⋅=
=2221
1211pppp
P
(47)
(48)
⋅=
22211211
2121 ))(),(())(),(( ppppxpxpypyp
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 70FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Binary symmetric channel (BSC)
Error probability:
errerr21
212121)]()([
)()()error(ppxpxp
pxppxpp=⋅+=
⋅+⋅=
−
−=
errerr
errerr1
1pp
ppP (49)
(50)
y2
y11−perr
perr
perr
x1
x21−perr
36
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 71FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Example for mutual information – qualitative consideration:Transmission of 1000 binary statistically independent andequiprobable symbols (p(0) = p(1) = 0.5)Binary symmetric channel with perr = 0.01Average number of correctly received symbols: 990But: T(X,Y) < 0.99 bit/symbolReason: exact position of errors is not known
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 72FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Definitions of entropy at a discrete memoryless channel:Input entropy = average information content of the inputsymbols:
Output entropy = average information content of the outputsymbols:
Joint entropy = average information content (uncertainty) of the whole transmission system:
)(1ld)()(
1 i
N
ii xp
xpXHX∑=
⋅=
)(1ld)()(
1 j
N
jj yp
ypYHY∑=
⋅=
∑ ∑= =
⋅=X YN
i
N
j jiji yxp
yxpYXH1 1 ),(
1ld),(),(
(51a)
(51b)
(51c)
37
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 73FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Conditional entropy H(Y|X) = average information content at the output for known input symbols = entropy of theirrelevance
Conditional entropy H(X|Y) = average information content at the input for known output symbols = entropy of theinformation that is lost in the channel = entropy of the equivocation
∑ ∑= =
⋅=X YN
i
N
j ijji xyp
yxpXYH1 1 )|(
1ld),()|(
∑ ∑= =
⋅=X YN
i
N
j jiji yxp
yxpYXH1 1 )|(
1ld),()|(
(51d)
(51e)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 74FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Relations between different types of entropy:
Average information flow (average mutual information):
)|()()|()(),(),( YXHYHXYHXHXYHYXH +=+==
)()|( XHYXH ≤
)()|( YHXYH ≤
),()()(
)|()(
)|()(),(
YXHYHXH
XYHYH
YXHXHYXT
−+=
−=
−=
(53)
(54)
(55)
(56a)
(56b)
(56c)
38
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 75FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Information flow
source sourceencoder
sourcedecoder sink
H(U)H(X)
H(Y)
H(X|Y)
H(Y|X)
equivocation
irrelevance
information flow
)ˆ(UH
transmission channel
T(X,Y)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 76FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Examples – information flow:Ideal noiseless channel:
Entropies: H(X|Y) = 0, H(Y|X) = 0, H(X,Y) = H(X) = H(Y)T(X,Y) = H(X) = H(Y)
Useless, very noisy channel: p(xi,yj) = p(xi) ⋅ p(yj) = p(yj|xi) ⋅ p(xi) ⇒ p(yj|xi) = p(yj) ⇒ pij = pkj
Entropies: H(X|Y) = H(X), H(Y|X) = H(Y), H(X,Y) = H(X) + H(Y)T(X,Y) = 0
≠=
=jiji
pij for 0for 1
(57)
(58)
(59)
(60)
39
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 77FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Example for mutual information – quantitative consideration:Transmission of 1000 binary statistically independent and equiprobable symbols (p(0) = p(1) = 0.5)Binary symmetric channel with perr = 0.01
bit/symbol9192.0
)(11ld1
1ld)1())1()0((1
)|(1ld)|()(
)(1ld)(
)|()(),(
errerr
errerr
err
1 11
≅
−=
+
−−+−=
⋅⋅−⋅=
−=
∑ ∑∑= ==
pSp
pp
ppp
xypxypxp
ypyp
XYHYHYXTX YY N
i
N
j ijiji
N
j jj
(61)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 78FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Channel capacityThe average mutual information (average information flow) depends on the probability of the source symbols.Definition of channel capacity:
Channel capacity = maximum average mutual information flow∆T = symbol periodUnit for channel capacity: bit/sC depends on properties of the channel – it does not depend on thesource!
),(max1
)(YXT
TC
ixp∆= (62)
40
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 79FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Definition of the average information flow ...Average information flow = entropy / time:H′(X) = H(X) / ∆T
Average mutual information flow = average mutual information /
time:T′(X,Y) = T(X,Y) / ∆T
Decisions for selecting symbols / time: H0′(X) = H0(X) /∆T
(63)
(64)
(65)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 80FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Example: binary symmetric channel – BSC:
−
−+−
+−−+−−+
−+−+=
−=
errerr
errerr
err1err1err1err1
err1err1err1err1
11ld)1(1ld
211ld)21(
21ld)2(
)|()(),(
pp
pp
pppppppp
pppppppp
XYHYHYXT
(66)
41
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 81FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Average mutual information
Maximum for p1 = p(x1) = 0.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 p(x1)
T(X,Y)bi
t/sym
bol
perr = 0
perr = 0.1
perr = 0.2
perr = 0.3
perr = 0.5
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 82FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Channel capacity
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 perr
C⋅∆T / bit
)(11
1ld)1(1ld1),(max errerr
errerr
err)(
pSp
pp
pYXTTCixp
−=
−
−+−==∆⋅(67)
42
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 83FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Example: binary erasure channel – BEC
−
−=
err
err
err
err1
00
1pp
pp
P err1 pTC −=∆⋅
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 perr
C⋅∆T / bit
(69)(68)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 84FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Theorem of channel capacity (Shannon 1948)For any ε > 0 and any information flow of a source R smaller thanchannel capacity C (R < C ), a binary block code of length n (nsufficiently large) can be found, so that the residual error probabilityafter decoding in the receiver is smaller than ε .Reverse statement: even with largest coding effort the residual errorprobability cannot decrease below a certain limit for R > C. Proof of the theorem uses random block codes (random coding argument):
Proof for an average over all codesAll known codes are bad codes
No rule for construction of codes
43
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 85FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Channel capacity: optimum for infinite code word length ⇒ infinite time delay, infinite complexityMore practical theorem related to the theorem of channel capacity by definition of Gallager‘s error exponent for a DMC with NX input symbols:
There exists always an (n,k) block code with RC = k / n ld NX < C ∆T, so that the word error probability is given by:
⋅−⋅−= ∑ ∑
=
+
=
+≤≤
Y X
i
N
j
sN
i
sijixps
xypxpRsRE1
1
1
11
C)(10
CG )|()(ldmaxmax)(
)(w CG2 REnP ⋅−<
(70)
(71)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 86FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Gallager‘s error exponentProperties:EG(RC) > 0 for RC < C·∆TEG(RC) = 0 for RC ≥ C·∆T
Definition of R0(computational cut-off rate) :
R0 = EG(RC = 0) 0.0
0.2
0.4
0.6
0.8
0.0 0.2 0.4 0.6 0.8 1.0
EG(RC)
RC
R0
R0 C ·∆T
(72)
44
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 87FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Computational cut-off rate R0:Maximum of EG(RC = 0) lies at s = 1
Comparison for s = 1:
⋅−=== ∑ ∑
= =
Y X
i
N
j
N
iiji
xpxypxpRER
1
2
1)(CG0 )|()(ldmax)0(
C01
2
1C
)(CG )|()(ldmax)( RRxypxpRRE
Y X
i
N
j
N
iiji
xp−=
⋅−−≥ ∑ ∑
= =
(73)
(74)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 88FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
R0-theoremThere exists always an (n,k) block code with RC = k / n ld NX < C ∆T, so that the word error probability (for maximum likelihood decoding)is given by:
No rules for construction of good codesRange of values for the code rate:
0 ≤ RC ≤ R0 PW is bounded by (75)R0 ≤ RC ≤ C ∆T an upper bound for PW is difficult to calculateRC > C ∆T PW cannot become arbitrarily small
)(w C02 RRnP −−< (75)
45
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 89FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Information theory
Comparison between channel capacity and computational cut-off rate R0 :
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.1 0.2 0.3 0.4 0.5 perr
R0
C ∆T
bit/s
ymbo
l
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 90FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Channel coding in digital communication systems
Goals: principles and examples for block codes Design of code words
Binary code words Redundant codesCode C = set of all code words
Code word c = (c0, c1, ... , cn-1) with c ∈ CEncoding is a memoryless assignment:
information word code word u = (u0, u1, ... , uk-1) → c = (c0, c1, ... , cn-1)
k information bits n code bits n ≥ k
46
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 91FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Channel coding in digital communication systems
Identical codes: codes with the same code words
Equivalent codes: Codes, which become identical after a permutationof bits
General notation: (n,k,dmin)q block code
q = number of symbols (size of alphabet)
Code rate:
Number of code words: N = qk = 2k
1C ≤=nkR (76)
(77)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 92FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Systematic codes:Code word c = (u, p)
m = n-kparity-check bits
Non-systematic codes: information and parity-check bits cannot be separatedLinear block codes can always be converted into equivalent systematic codes
u0 u1 u2 u3 ... ... uk-1
↓ ↓ ↓ ↓ ↓ ↓ ↓c0 c1 c2 c3 ... ... ck-1 ck ck+1 ... ... cn-1
(78)
Information Engineering 2Channel coding in digital communication systems
47
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 93FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Addition and multiplication in a binary number system (modulo 2):
Two binary vectors x and y with the same lengthHamming distance:
dH(x,y) = number of different bits for x and y
Example: dH( 0 1 1 1 0 1 0 1,1 0 1 0 0 1 0 1 ) = 3
⊕ 0 10 0 11 1 0
⊗ 0 10 0 01 0 1
Information Engineering 2Channel coding in digital communication systems
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 94FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
(Hamming) weight of a vector x:
Example: wH(0 1 1 1 0 1 0 1) = 5
Hamming distance: dH(x,y) = wH(x + y)
Example: x = ( 0 1 1 1 0 1 0 1 )y = ( 1 0 1 0 0 1 0 1 )
x + y = ( 1 1 0 1 0 0 0 0 )wH(x + y) = 3
Information Engineering 2Channel coding in digital communication systems
(80)
0 fromdifferent bits ofnumber )(1
0H == ∑
−
=
n
iixw x (79)
48
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 95FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Transmission via a binary channel:
y = x + e
e = error vector
Repetition codek = 1 information bit → n − 1 repetitions2k = 2 code words: c1 = (0 0 0 ... 0) and c2 = (1 1 1 ... 1) Repetition code is a systematic codeSimple decoding by majority vote (especially, if n is odd)(n − 1)/2 errors can be corrected, n − 1 errors can be detected
Information Engineering 2Channel coding in digital communication systems
(81)x y
e
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 96FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Channel coding in digital communication systems
Example for a repetition code:n = 5 ⇒ RC = 1/5 u1 = (0) → c1 = (0 0 0 0 0) and u2 = (1) → c2 = (1 1 1 1 1)
Transmission via a noisy channel:e1 = (0 1 0 0 1) and x1 = (1 1 1 1 1) y1 = x1 + e1 = (1 0 1 1 0) →
e2 = (1 1 0 1 0) and x2 = (0 0 0 0 0) y2 = x2 + e2 = (1 1 0 1 0) →
Two errors can be corrected, four detected
)1(ˆ1 =u
)1(ˆ 2 =u
49
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 97FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Channel coding in digital communication systems
Parity check code
k information bits, a single parity check bit (m = 1) → n = k + 1 2k code words
c = (u , p) with p = u0 + u1 + u2 + ... + uk−1
(number of ones in code words c is even)
The parity check code is a systematic code. No error can be corrected, an odd number of errors can be detected.Parity check:s0 = y0 + y1 + y2 + ... + yn−1 = 0 → no errors0 = y0 + y1 + y2 + ... + yn−1 = 1 → error
(82)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 98FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Channel coding in digital communication systems
Example: k = 3
y1 = ( 0 1 1 0) → no error y2 = ( 1 1 1 0) → error
code word information bits
Parity check bit
c0 000 0 c1 001 1 c2 010 1 c3 011 0 c4 100 1 c5 101 0 c6 110 0 c7 111 1
50
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 99FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Channel coding in digital communication systems
Hamming codeA single error in a codeword can be corrected.Example: (7,4) Hamming code
Parity check bits:c4 = c0 + c1 + c2
c5 = c0 + c1 + c3
c6 = c0 + c2 + c3
u0 u1 u2 u3
↓ ↓ ↓ ↓c0 c1 c2 c3 c4 c5 c6
(83a)(83b)(83c)
Prof. Dr.-Ing. Andreas Czylwik Information Engineering 2 SS 2004
p. 100FachgebietNachrichtentechnische Systeme
N T SUNIVERSITÄT
D U I S B U R GE S S E N
Information Engineering 2Channel coding in digital communication systems
Matrix representation of block codesx = u G
G = generator matrix (k × n matrix)
Systematic block codes:
G = [Ik P]
Ik = identity matrix k × k, P = parity bit matrix k × (n − k)
Example: (7,4) Hamming code
(84)
(85)
=
110|101|011|111|
1000010000100001
G (86)