Download - Transmitters Compression
-
8/8/2019 Transmitters Compression
1/18
Increasing Information per Bit Information in a source
Mathematical Models of Sources
Information Measures
Compressing information Huffman encoding
Optimal Compression for DMS?
Lempel-Ziv-Welch Algorithm For Stationary Sources?
Practical Compression
Quantization of analog data Scalar Quantization
Vector Quantization
Model Based Coding
Practical Quantization Q-law encoding Delta Modulation
Linear Predictor Coding (LPC)
-
8/8/2019 Transmitters Compression
2/18
Huffman encoding Variable length binary code for DMS
finite alphabet, fixed probabilities
Code satisfies thePrefix Condition
Codes are instantaneously and unambiguously
decodable as they arrive
e.g, 0,10,110,111 is OK0,01,011,111 is not OK 01 = 0,111, or 01
-
8/8/2019 Transmitters Compression
3/18
Huffman encoding
Use Probabilities to order coding priorities of letters Low probability get codes first (more bits) This smoothes out the information per bit
-
8/8/2019 Transmitters Compression
4/18
Huffman encoding Use a code tree to make the code
Combine the symbols with lowest probability to make a new block symbol Assign a 1 to one of the old symbols code word and 0 to the other symbol
Now reorder and combine the two lowest probability symbols of the new set
Each time the synthesized block symbol has lowest probability the code wordsget shorter
x1 00x2 01
x3 10
x4 110
x5 1110
x6 11110
x7 11111
D0 D1 D2 D3 D4
-
8/8/2019 Transmitters Compression
5/18
Huffman encoding
Result: Self Information or Entropy
H(X) = 2.11 (The best possible average number of bits)
Average number of bits per letter
21.2
)(7
1
_
!
! !k
kkxPnR
nk= number of bits per symbol
So the efficiency = %5.95)(
_!
R
XH
-
8/8/2019 Transmitters Compression
6/18
Huffman encoding
Lets compare to simple 3 bit code
x1 00 2 0.35 000 3 0.35
x2 01 2 0.3 001 3 0.3x3 10 2 0.2 010 3 0.2
x4 110 3 0.1 011 3 0.1
x5 1110 4 0.04 100 3 0.04
x6 11110 5 0.005 101 3 0.005x7 11111 5 0.005 110 3 0.005
R= 2.21 R= 3
Efficiency 95% 70%
-
8/8/2019 Transmitters Compression
7/18
Huffman encoding
Another example
x1 00x2 010
x3 011
x4 100
x5 101
x6 110
x7 1110
X8 1111
D0 D1 D2 D3
-
8/8/2019 Transmitters Compression
8/18
Huffman encoding
Multi-symbol Block codes Use symbols made of original symbols
)(lim
1)(
,
_
_
2
772111
XHJ
R
JXH
J
R
LJxxxxxx
J
J
j
!
!!
gp
- Symbols
Can show the new codes information per bit satisfies:)(_
JR
So large enough block code gets you as close to H(X) as you want
-
8/8/2019 Transmitters Compression
9/18
Huffman encoding
Lets consider
a J=2 block
code example
-
8/8/2019 Transmitters Compression
10/18
Encoding Stationary Sources
Now there are joint probabilities of blocks
of symbols that depend on previous bits
)()()()(
)|()|()|()()(
321
1121312121
k
kkk
xPxPxPxP
xxxPxxxPxxPxPxxxP
-
---
{
!
Unless DMS
Can show the joint entropy is:
!
e
!k
m
m
kkk
XH
XXXXHXXHXHXXXH
1
12112121
)(
)|()|()()( ...
Which means less bits than a symbol by symbol code can be used
-
8/8/2019 Transmitters Compression
11/18
Encoding Stationary Sources H(X| Y) is the conditional entropy
! !
!n
i
m
j
jijji yxyyxYXH1 1
2 )(log)()()(
Joint (total) probability ofxi
Information inxi givenyj
))()(
)((log)()()(
1 1
2 i
n
i
m
j j
ij
iij xPyP
xyPxPxyPYXH
! !
!
Can show:
-
8/8/2019 Transmitters Compression
12/18
Conditional Entropy Plotting this forn = m = 2 we see that when
Ydepends strongly on Xthen H(X|Y) is low
Conditional Entropy H(X|Y) vs P(X=0) for various P(Y|X)
0.00E+00
2.00E-01
4.00E-01
6.00E-01
8.00E-01
1.00E+00
1.20E+00
0.00E+00 2.00E-01 4.00E-01 6.00E-01 8.00E-01 1.00E+00 1.20E+00
P(X=0)
H(X|
Y)
0.05
0.15
0.35
0.5
0.75
0.9999
P(Y=0|X=0)
P(Y=1|X=1)
-
8/8/2019 Transmitters Compression
13/18
Conditional Entropy To see how P(X|Y) and P(Y|X) relate consider:
They are very simlar when P(X=0) ~ 0.5
0.00E+00
2.00E-01
4.00E-01
6.00E-01
8.00E-01
1.00E+00
1.20E+00
0.00E+00 2.00E-01 4.00E-01 6.00E-01 8.00E-01 1.00E+00 1.20E+00
P(X=0)
0.05
0.15
0.35
0.5
0.75
0.95
P(X=0| =0)
P(Y=0|X=0)
-
8/8/2019 Transmitters Compression
14/18
Optimal Codes for Stationary Sources
Can show that for large blocks of symbols
Huffman encoding is efficient
gp
p
e
gp
e
!
gg
J
XHRXH
J
JXHRXH
XXXHJXH
JJ
JJ
0
)()(
1)()(
)(
1
)(
_
_
21
I
I
.Define
Then Huffman code gives:
Now if
Get:
i.e., Huffman is optimal
-
8/8/2019 Transmitters Compression
15/18
Lempel-Ziv-Welch Code
Huffman encoding is efficient but need to
know joint probabilities of large blocks of
symbols
Finding joint probabilities is hard
LZW is independent of source statistics
Is a universal source code algorithm
Is not optimal
-
8/8/2019 Transmitters Compression
16/18
Lempel-Ziv-Welch
Build a table from strings not already in the table Output table location for strings in the table
Build the table again to decode
Input String = /WED/WE/WEE/WEB/WET
Characters Input Code Output New code value New String
/ / 256 /W
W W 257 WE
E E 258 ED
D D 259 D/
/W 256 260 /WE
E E 261 E/
/WE 260 262 /WEE
E/ 261 263 E/W
WE 257 264 WEB
B B 265 B/
/WE 260 266 /WET
T T
Source: http://dogma.net/markn/articles/lzw/lzw.htm
-
8/8/2019 Transmitters Compression
17/18
Lempel-Ziv-Welch
DecodeInput Codes: / W E D 256 E 260 261 257 B 260 T
Input/
NEW_CODEOLD_CODE
STRING/
OutputCHARACTER New table entry
/ / /
W / W W 256 = /W
E W E E 257 = WE
D E D D 258 = ED
256 D /W / 259 = D/
E 256 E E 260 = /WE
260 E /WE / 261 = E/
261 260 E/ E 262 = /WEE
257 261 WE W 263 = E/W
B 257 B B 264 = WEB
260 B /WE / 265 = B/
T 260 T T 266 = /WET
-
8/8/2019 Transmitters Compression
18/18
Lempel-Ziv-Welch Typically takes hundreds of entries to table
before compression occurs
Some nasty patents make licensing an issue