9/13/2005J. Liang: SFU ENSC 424 1

ENSC 424 - Multimedia Communications EngineeringHuffman Coding (1)

Jie LiangEngineering Science

Simon Fraser UniversityJieL@sfu.ca

9/13/2005J. Liang: SFU ENSC 424 2

Outline

�Entropy Coding�Prefix code�Kraft-McMillan inequality�Huffman Encoding�Minimum Variance Huffman Coding�Extended Huffman Coding

9/13/2005J. Liang: SFU ENSC 424 3

Entropy Coding

�Design the mapping from source symbols to codewords

�Lossless mapping�Goal: minimizing the average codeword length

�Approach the entropy of the source

9/13/2005J. Liang: SFU ENSC 424 4

Example: Morse Code� Represent English characters and numbers by

different combinations of dot and dash (codewords)� Examples:

E I A

T O S Z

�Letters have to be separated by space,Or paused when transmitting over radio.SOS:

pause

�Problem:Not uniquely decodable!

9/13/2005J. Liang: SFU ENSC 424 5

Entropy Coding: Prefix-free Code� No codeword is a prefix of another one.� Can be uniquely decoded.� Also called prefix code� Example: 0, 10, 110, 111� Binary Code Tree

0 1

0 1

0 1

010

110 111

Root node

leaf node

Internal node

� Prefix-free code contains leaves only.� How to express the requirement mathematically?

9/13/2005J. Liang: SFU ENSC 424 6

Kraft-McMillan Inequality� Let C be a code with N codewords with length l i,

i=1,…N. If C is uniquely decodable, then

121

≤∑=

−N

i

l i

� If a set of l i satisfies the inequality above, then there exists a prefix-free code with codeword lengths

l i, i=1,…N.

9/13/2005J. Liang: SFU ENSC 424 7

Kraft-McMillan Inequality

22 1211

LN

i

lLN

i

l ii ≤⇔≤ ∑∑=

−

=

−

� To see this, expand the binary code tree to depth L = max(li)

010

110 111

� Number of nodes in the last level:

� Each code has a sub-tree:� The number of offsprings in the last level:

� K-M inequality:

# of L-th level offsprings of all codes is less than 2^L.

ilL−2

L2

L = 3 010

110 111

11

01011110111

Leads to more than2^L offspring

9/13/2005J. Liang: SFU ENSC 424 8

Outline

�Entropy Coding�Prefix code�Kraft-McMillan inequality�Huffman Encoding�Minimum Variance Huffman Coding�Extended Huffman Coding

9/13/2005J. Liang: SFU ENSC 424 9

Huffman Coding

a

b a b

truncate

� A procedure to construct optimal prefix-free code � Result of David Huffman’s term paper in 1952 when

he was a PhD student at MIT� Shannon � Fano � Huffman (1925-1999)

� Observations:� Assign short codes to frequent symbols.� In an optimum prefix-free code, the two codewords that

occur least frequently will have the same length.

9/13/2005J. Liang: SFU ENSC 424 10

Huffman Code Design� Another property of Huffman coding:

� The codewords of the two lowest probability symbols differ only in the last bit.

� Requirement:� The source probability distribution

(Not available in most cases)� Procedure:

1. Sort the probability of all source symbols in a descending order.2. Merge the last two into a new symbol, add their probabilities.3. Repeat Step 1, 2 until only one symbol (the root) is left.4. Code assignment:

Traverse the tree from the root to each leaf node, assign 0 to the top branch and 1 to the bottom branch.

9/13/2005J. Liang: SFU ENSC 424 11

Example 3.2.1� Source alphabet A = {a1, a2, a3, a4, a5}� Probability distribution: {0.2, 0.4, 0.2, 0.1, 0.1}

a2 (0.4)

a1(0.2)

a3(0.2)

a4(0.1)

a5(0.1)

Sort

0.2

merge Sort

0.4

0.2

0.2

0.2

0.4

merge Sort

0.4

0.2

0.40.6

merge

0.6

0.4

Sort

1

merge

Assign code

0

1

1

00

01

1

000

001

01

1

000

01

0010

0011

1

000

01

0010

0011

9/13/2005J. Liang: SFU ENSC 424 12

Huffman code is prefix-free

01

000

0010 0011

1

000

01

0010

0011

1

�All codewords are leaf nodes� No code is a prefix of any other code.

(Prefix free)

9/13/2005J. Liang: SFU ENSC 424 13

Average Codeword Length vs Entropy

� Source alphabet A = {a, b, c, d, e}� Probability distribution: {0.2, 0.4, 0.2, 0.1, 0.1}� Code: {01, 1, 000, 0010, 0011}

� Entropy:H(S) = - (0.2*log2(0.2)*2 + 0.4*log2(0.4)+0.1*log2(0.1)*2)= 2.122 bits / symbol

� Average Huffman codeword length:L = 0.2*2+0.4*1+0.2*3+0.1*4+0.1*4 = 2.2 bits / symbol

� In general: H(S) ≤ L < H(S) + 1

9/13/2005J. Liang: SFU ENSC 424 14

Huffman Code is not unique

0.4

0.2

0.40.6

0.6

0.4

0

1

1

00

01

� Multiple ordering choices for tied probabilities

� Two choices for each split: 0, 1 or 1, 0

0.4

0.2

0.4 0.6

0.6

0.4

1

0

0

10

11

a

b

c

0.4

0.2

0.40.6

0.6

0.4

1

0

0

10

11

b

a

c

0.4

0.2

0.40.6

0.6

0.4

1

0

0

10

11

9/13/2005J. Liang: SFU ENSC 424 15

Minimum Variance Huffman Code�Put the combined symbol as high as possible

in the sorted list�Prevent unbalanced tree:

�Reduce memory requirement for decoding�(revisited later)

�Repeat previous example�Compute average codeword length

9/13/2005J. Liang: SFU ENSC 424 16

Extended Huffman Code

�Code multiple symbols jointly�Composite symbol: (X1, X2, …, Xk)

�Code symbols of different meanings jointly�JPEG: Run-level coding�H.264 CAVLC: context-adaptive variable length

coding� # of non-zero coefficients and # of trailing ones

�Revisited later

�Alphabet increased exponentioally: Nk

9/13/2005J. Liang: SFU ENSC 424 17

Example

� P(Xj = 0) = P(Xj = 1) = 1/2� Entropy H(Xj) = 1 bit / symbol

� Joint probability: P(X2i, X2i+1)� P(0, 0) = 3/8, P(0, 1) = 1/8� P(1, 0) = 1/8, P(1, 1) = 3/8

symbol / bits 0.9056or symbols, 2 / bits 1.8113),( 122 =+ii XXH

� Huffman code for Xj: 0, 1� Average code length 1 bit / symbol� Huffman code for (X2i, X2i+1):

00: 1, 11: 00, 01: 010, 10: 011� Average code length: 0.9375 bit /symbol

3/81/81

1/83/80

10X2i+1X2i

Joint Prob P(X2i, X2i+1)

� Second order entropy:

9/13/2005J. Liang: SFU ENSC 424 18

Summary� Goal of entropy coding:

� Reduce the average codeword length (the entropy is the lower bound)

� Prefix-free code: uniquely decodable code� Kraft-McMillan Inequality:

� Characteristic of prefix-free code

� Huffman Code:� Optimal prefix-free code� Minimum variance code

� Next: � Canonical Huffman� Encoding and decoding