coding techs

7/29/2019 Coding Techs

1/45

1

Chapter 5

VARIABLE-LENGTH CODING

----INFORMATION THEORY RESULTS (II)

5.1 Some Fundamental Results

5.1.1Coding an Information Source

Consider an information source, representedby a source alphabet S.

},,,{21 m

sssS L= , (5.1)

where is s are source symbols.

Terms source symbol and informationmessage.

Used interchangeably in the literature.


2/45

2 CHAPTER 5 VARIABLE-LENGTH CODING

We like to consider: an informationmessage can be a source symbol, or acombination of source symbols.

We denote code alphabetbyA and},,,{

21 raaaA L= ,

where ja s are code symbols.

A message code is a sequence of codesymbols that represents a given informationmessage.

In the simplest case, a message consists ofonly a source symbol.Encoding is then a procedure to assign a

codewordto the source symbol. Namely,

),,,(21 ikiiii

aaaAs L= ,

where the codeword iA is a string of k codesymbols assigned to the source symbol is .

The term message ensemble is defined asthe entire set of messages.


3/45

5.1 Coding An information Source 3

A code, also known as an ensemble code, isdefined as a mapping of all the possiblesequences of symbols of S (message

ensemble) into the sequences of symbols inA.

In binary coding, the number of codesymbols r is equal to 2, since there are only

two code symbols available: the binary

digits 0 and 1.

Example 5.1

Consider an English article and ASCII code.

In this context, the source alphabet consists

of all the English letters in both lower andupper cases andall the punctuation marks.

The code alphabet consists of the binary 1and 0.

There are a total of 128 7-bit binary

codewords.From Table 5.1, we see that codeword

assigned to capital letter A (a source symbol)is 1000001 (the codeword).


4/45


5/45


Example 5.2

Table 5.2 lists what is known as the (5,2)

code.It is a linear blockcode.

In this example, the source alphabet consists

of the four )2( 2 source symbols listed in the

left column of the table: 00, 01, 10 and 11.

The code alphabet consists of the binary 1and 0.

There are four codewords listed in the rightcolumnof the table.

From the table, we see that the code assigns

a 5-bit codeword to eachsource symbol.

Table 5. 2 A (5,2) linear block code

Source Symbol Codeword

S1 ( 0 0 )

S2 ( 0 1 )S3 ( 1 0 )

S4 ( 1 1 )

0 0 0 0 0

1 0 1 0 00 1 1 1 1

1 1 0 1 1


6/45


5.1.2Some Desired Characteristics

5.1.2.1Block Code

A code is said to be a block code if it mapseach source symbol in S into a fixedcodeword in A.

Hence, the codes listed in the above two examples are block codes.

5.1.2.2Uniquely Decodable Code

A code is uniquely decodable if it can beunambiguously decoded.

Obviously, a code has to be uniquelydecodableif it is to be in use.

Example 5.3


7/45


Table 5. 3 A not uniquely decodable code


S1

S2

S3

S4

0 0

1 0

0 0

1 1

Nonsingular Code

A block code is nonsingular if all thecodewordsare distinct.

Table 5. 4 A nonsingular code


S1

S2S3

S4

1

1 10 0

0 1


8/45


Example 5.4

Table 5.4 gives a nonsingular code since all

four codewordsare distinct.If a code is not a nonsingular code, i.e., at

least two codewords are identical, then the

code is not uniquelydecodable.

Notice that, however, a nonsingular code

does not guarantee unique decodability.

The code shown in Table 5.4 is such anexample in that it is nonsingular while it is

not uniquely decodable (because once the

binary string 11 is received, we do notknow if the source symbols transmitted are

1s followed by 1s orsimply 2s .)

The nth Extension of a Block Code

The nth extension of a block code, whichmaps the source symbol is into thecodeword

i, is a block code that maps the

sequences of source symbols iniisss L

21


9/45


10/45


11/45


In decoding we can immediately tell

which source symbols are transmittedsince each codeword has the same length.

q In the second code, code symbol 1functions like a comma. Whenever we see a

1, we know it is the end of the codeword.

q The third code is different from the previoustwo codes in that if we see a 10 string we

are not sure if it corresponds to 2s until wesee a succeeding 1. Specifically, if thenext code symbol is 0, we still cannot tell

if it is 3s since the next one may be 0

(hence 4s ) or 1 (hence 3s ). In this

example, the next 1 belongs to thesucceeding codeword. Therefore we see thatcode 3 is uniquely decodable. It is not

instantaneous,however.

Definition of the jth Prefix

Assume a codeword ikiiiaaaA L

21= . Then

the sequences of code symbols ijii aaa L21


12/45


with kj 1 is the jth order prefix of the

codeword iA .

Example 5.7

q If a codeword is 11001, it has the followingfive prefixes: 11001, 1100, 110,11, 1.

q The first order prefix is 1, while the fifth

order prefixis 11001.

A Necessary and Sufficient Condition of

Being Instantaneous Codes

v A code is instantaneous if and only if nocodeword is a prefix of some othercodeword.

v This condition is called the prefixcondition. Hence, the instantaneous codeis also called the prefix condition code or

sometimes simply the prefix code.

v In many applications, we need a blockcode that is nonsingular, uniquely

decodable, and instantaneous.


13/45


5.1.2.4Compact Code

A uniquely decodable code is said to becompact if its average length is the

minimum among all other uniquelydecodable codes based on the same

source alphabet Sand code alphabetA.

A compact code is also referred to as aminimum-redundancy code, or anoptimum code.

5.1.3Discrete Memoryless Sources

The simplest model of an information source.

In this model, the symbols generated by thesource are independent of each other. That

is, the source is memoryless or it has a zero-memory.


14/45


15/45


16/45


)()( SHnnSH = . (5. 4)

5.1.4.3Noiseless Source Coding Theorem

v For a discrete zero-memory informationsource S, Shannons noiseless coding

theorem can be expressedas

1)()( +


17/45


v Since )()( nHnH = andavgavg nL

nL = , we have

nSHavgLSH

1)()( +


18/45


5.2 Huffman Codes

In many cases, we need a direct encodingmethod that is optimum and instantaneous(hence uniquely decodable) for an

information source with finite source

symbols in source alphabet S.

Huffman code is the first such optimumcode [huffman 1952].

The most frequently used at present.It can be used for r-ary encoding as r>2. For

the notational brevity, however, we discuss

only the Huffman coding used in the binary

casepresented here.

5.2.1Required Rules for Optimum

Instantaneous Codes

Consider an information source:),,,( 21 msss

L= . (5. 9)

WLOG, assume the occurrence probabilitiesof the source symbols are as follows:


19/45

Index 19

)()()()(121 mm

spspspsp

L

Since we are seeking the optimum code forS, the lengths of codewords assigned to the

source symbolsshould be

mmllll

121L . (5. 10)

Based on the requirements of the optimumand instantaneous code, Huffman derived

the following rules(restrictions):

1. mm llll = 121 L . (5. 11)2.The codewords of the two least

probable source symbols should be thesame except for theirlast bits.

3.Each possible sequence of length1ml bits must be used either as a

codeword or must have one of its

prefixes used as a codeword.


20/45


21/45

Index 21

q The same procedure can be applied to thisnewly createdsource alphabet.

qThe second auxiliary source alphabet willagain have one source symbol less than thefirst auxiliary source alphabet.

q The procedure continues. In some step, theresultant source alphabet will have only twosource symbols. At this time, we combine

them to form a single source symbol with aprobability of 1. The coding is thencomplete.

Example 5.9


22/45


23/45

Index 23

5.2.2.1Applications

q Recall that it has been used in differentialcoding and transformcoding.

q In TC, the magnitude of the quantizednonzero transform coefficients and the run-

length of zeros in the zigzag scan are

encoded by using the Huffmancode.

5.2.3 Modified Huffman Codes

when the occurrence probabilities are skewed,

and the number of less probable source

symbols is large, the required codebookmemory will be large.

The modified Huffman code can reduce the

memory requirement while keeping almost the

same optimality.


24/45


5.3 Arithmetic Codes

Arithmetic coding:

quite different from Huffmancoding

gaining increasingpopularity

5.3.1Limitations of Huffman Coding

Huffman coding is optimum for block-encoding a source alphabet, with eachsource symbol having an occurrence probability.

The average codeword length achieved byHuffman coding satisfies the followinginequality [gallagher 1978].

086.0)()( max ++


25/45

Index 25

In the case where the probability distributionamong source symbols is skewed (someprobabilities are small, while some are quitelarge), the upper bound may be large,

implying that the coding redundancy may

not be small.

An extreme situation. There are only twosource symbols. One has a very smallprobability, while the other has a very large

probability (very close to1).

Entropy of source alphabet is close to0 since the uncertainty is verysmall.

Using Huffman coding, however, weneed two bits: onefor each.

average codeword length is1

redundancy 1

This agrees with Equation5.22.

This inefficiency is due to the

fact that Huffman coding alwaysencodes a source symbol with an

integer numberof bits.


26/45


27/45


28/45


29/45


30/45


Figure 5. 2 Arithmetic coding working on the samesource alphabet as that in Example 5.9. The encodedsymbol string is S1 S2 S3 S4 S5 S6.

0.3 0.4 0.6 0.65 0.75 1.00

[ 0, 0.3) [ 0.3, 0.4) [ 0.4, 0.6)

[ 0.6, 0.65) [ 0.65, 0.75)

[ 0.75, 1.0)S1

0.3

0.09 0.12 0.18 0.195 0.225

0

S2

[ 0.09, 0.12)

0.12

0.099 0.102 0.108 0.1095 0.1125

0.09

S3

[ 0.102, 0.108)

0.108

0.1083 0.1044 0.1056 0.1059 0.1065

0.102 S4

[ 0.1056, 0.1059)

0.1059

0.10569 0.10572 0.10578 0.105795 0.105825

0.1056 S5

[ 0.105795, 0.105825)

0.1058250

0.105804 0.105807 0.105813 0.1058145 0.1058175

0.105795 S6

[ 0.1058175, 0.1058250)

(a)

(b)

(c)

(d)

(e)

(f)


31/45

Index 31

5.3.2.2Encoding

Encoding the First Source Symbol

Refer to Part (a) of Figure 5.3. Since the first

symbol is 1s , we pick up its subinterval [0,

0.3). Picking up the subinterval [0, 0.3) meansthat any real number in the subinterval, i.e.,any real number equal to or greater than 0 and

smaller than 0.3, can be a pointer to thesubinterval, thus representing the source

symbol 1s . This can be justified by considering

that all the six subintervals are disjoined.

Encoding the Second Source Symbol

Refer to Part (b) of Figure 5.3. We use the

same procedure as used in Part (a) to divide

the interval [0, 0.3) into six subintervals.

Since the second symbol to be encoded is 2s ,we pick up its subinterval [0.09, 0.12).

Notice that the subintervals are recursively

generated from Part (a) to Part (b). It isknown that an interval may be completelyspecified by its lower end point and width.

Hence, the subinterval recursion in the


32/45


arithmetic coding procedure is equivalent to

the following two recursions: end pointrecursion and width recursion.

The lower end pointrecursion:

newCPcurrentWcurrentLnewL +=

: the lower end points

W: the width

new: thenewrecursion

current: thecurrent recursion

The width recursionis

)(i

scurrent

WnewW = , (5. 14)

Encoding the Third Source Symbol

Encoding the Fourth, Fifth and Sixth Source Symbols

The resulting subinterval [0.1058175,0.1058250) can represent the source symbol

string654321

ssssss .


33/45


34/45


That is, the lower end is contained in the

subinterval corresponding tothe symbol 2s .

As a result, 2s is the second decodedsymbol.

The procedure repeats itself until all six

symbols are decoded.

Note that a terminal symbol is necessary toinform the decoder to stop decoding.

q The above procedure gives us an idea ofhowdecoding works.

q The decoding process, however, does notneed to construct Parts (b), (c), (d), (e) and

(f) of Figure 5.3.

q Instead, the decoder only needs theinformation contained in Part (a) of Figure 5.3.

q Decoding can be split into the followingthree steps: comparison, readjustment(subtraction), and scaling [langdon 1984].


35/45

Index 35

q In summary, considering the way in whichParts (b), (c), (d), (e) and (f) of Figure 5.3are constructed, we see that the three steps

discussed in the decoding process:comparison, readjustment and scalingexactly undo what the encoding procedure

has done.

q Back to Example:

oAfter symbol 1s is firstdecoded,othe lower end of the corresponding

interval, 0, is subtracted from 0.1058175,

resulting in 0.1058175.

oThis value is then divided by p1=0.3,resulting in 0.352725. Compare this valuewith Part (a), we decode 2

s since0.3


36/45


37/45

Index 37

when the length of the source symbol string

becomes larger and larger. This causes whatis known as the precisionproblem.

It is this problem that prohibited arithmeticcoding from practical usage for quite a long

period oftime.

Only after this problem was solved in thelate 1970s, did arithmetic coding become an

increasingly important codingtechnique.

It is necessary to have a termination symbolat the end of an input source symbol string.In this way, an arithmetic coding system is

able to know when to terminatedecoding.

To encode the same source symbol string,Huffman coding can be implemented in two

different ways.

One way is shown in Example 5.9. We

construct a fixed codeword for each sourcesymbol. Since Huffman coding is

instantaneous, we can cascade the

corresponding codewords to form the

output, a 17-bit code string00.101.11.1001.1000.01, where the five


38/45


periods are used to indicate different

codewords for easy reading.

As we see that for the same source symbolstring, the final subinterval obtained byusing arithmetic coding is [0.1058175,

0.1058250). It is noted that the 15-bit binary

decimal, 0.000110111111111, is equal to the

decimal 0.1058211962, which falls into the

final subinterval representing the string654321

ssssss .

This indicates that the arithmetic coding ismore efficient than the Huffamn coding in

thisexample.

Another way is to form a 6th extension of thesource alphabet as discussed in Section5.1.4: treat each group of six source symbols

as a new source symbol; calculate its

occurrence probability by multiplying therelated six probabilities; then apply the

Huffman coding algorithm to the 6th

extension of the discrete memoryless source.

This is called the 6th extension of Huffman

block code. In other words, in order to


39/45

Index 39

encode the source string 654321ssssss , (the

6th extension of) Huffman coding encodes

all of the 4665666 = codewords in the 6th

extension of the source alphabet. This

implies a high complexity in implementationand a large codebook, hence notefficient.

Similar to the case of Huffman coding,arithmetic coding is also applicable to r-ary

encodingwith r>2.

5.3.3Implementation Issues

q The growing precisionproblem.

q This problem has been resolved and thefinite precision arithmetic is now used in

arithmeticcoding.

q This advance is due to the incrementalimplementation of arithmeticcoding.

5.3.3.1 Incremental Implementation


40/45


41/45


42/45


fraction was introduced in Shannons

celebrated paper [shannon1948].

The recursive implementation of arithmeticcoding was devised by Elias (anothermember in Fanos first information theory

class at MIT).

This unpublished result was first introducedby Abramson as a note in his book on

information theory and coding [abramson 1963].

The result was further developed by Jelinekin his book on information theory [jelinek

1968].

The growing precision problem preventedarithmetic coding from practical usage,

however. The proposal of using finiteprecision arithmetic was made

independently by Pasco [pasco 1976] and

Rissanen [rissanen1976].

Practical arithmetic coding was developedby several independent groups [rissanen

1979, rubin 1979,guazzo 1980].


43/45

Index 43

A well-known tutorial paper on arithmeticcoding appeared in[langdon 1984].

The tremendous efforts made in IBM lead toa new form of adaptive binary arithmeticcoding known as the Q-coder [pennebaker1988].

Based on the Q-coder, the activities ofJPEG and JBIG combined the best features

of the various existing arithmetic coders anddeveloped the binary arithmetic coding

procedure known as the QM-coder[pennebaker1992].

5.3.5Applications

Arithmetic coding is becomingpopular.

Note that in text and bilevel image

applications there are only two source

symbols (black and while), and theoccurrence probability is skewed. Thereforebinary arithmetic coding achieves high

codingefficiency.


44/45


It has been successfully applied to bilevel

image coding [langdon 1981] and adoptedby the international standards for bilevel

image compression JBIG.

It has also been adopted by the JPEG.

5.4 References

[abramson 1963] N. Abramson, Information Theory and Coding, New York: McGraw-

Hill, 1963.

[bell 1990] T. C. Bell, J. G. Cleary and I. H. Witten, Text Compression, Englewood, NJ:Prentice Hall, 1990.

[blahut 1986] R. E. Blahut,Principles and Practice of Information Theory, Reading MA:Addison-Wesley, 1986.

[fano 1949] R. M. Fano, The transmission of information, Technical Report 65,Research Laboratory of Electronics, MIT, Cambridge, MA, 1949.

[gallagher 1978] R. G. Gallagher, Variations on a theme by Huffman, IEEE

Transactions on Information Theory, vol. IT-24, no. 6, pp. 668-674, November1978.

[guazzo 1980] M. Guazzo, A general minimum-redundacy source-coding algorithm,IEEE Transactions on Information Theory, vol. IT-26, no. 1, pp. 15-25, January 1980.

[hankamer 1979] M. Hankamer, A modified Huffman procedure with reducedmemory requirement, IEEE Transactions on Communications, vol. COM-27, no.6, pp. 930-932, June 1979.

[huffman 1952] D. A. Huffman, A method for the construction of minimum-

redundancy codes, Proceedings of The IRE, vol. 40, pp. 1098-1101, September1952.


45/45

Index 45

[jelinek 1968] F. Jelinek, Probabilistic Information Theory, New York: McGraw-Hill, 1968.

[langdon 1981] G. G. Langdon, Jr. and J. Rissanen, Compression of black-white

images with arithmetic coding, IEEE Transactions on Communications, vol.COM-29, no. 6, pp. 858-867, June 1981.

[langdon 1982] G. G. Langdon, Jr. and J. Rissanen, A simple general binary

source code,IEEE Transactions on Information Theory, IT-28, 800 (1982).

[langdon 1984] G. G. Langdon, Jr., An introduction to arithmetic coding, IBMJournal of Research and Development, vol. 28, no. 2, pp. 135-149, March 1984.

[nelson 1996] M. Nelson and J. Gailly, The Data Compression Book, second edition,New York: M&T Books, 1996.

[pasco 1976] R. Pasco, Source Coding Algorithms for Fast Data Compression, Ph.D.

dissertation, Stanford University, 1976.

[pennebaker 1988] W. B. Pennebaker, J. L. Mitchell, G. G. Langdon, Jr. and R. B. Arps,

An overview of the basic principles of the Q-coder adaptive binary arithmetic Coder,IBM Journal of Research and Development, vol. 32, no. 6, pp. 717-726, November 1988.

[pennebaker 1992] W. B. Pennebaker and J. L. Mitchell, JPEG: Still Image DataCompression Standard, New York: Van Nostrand Reinhold, 1992.

[rissanen 1976] J. J. Rissanen, Generalized Kraft inequality and arithmetic coding,IBMJournal of Research and Development, vol. 20, pp. 198-203, May 1976.

[rissanen 1979] J.J. Rissanen and G. G. Landon, Arithmetic coding, IBM Journal ofResearch and Development, vol. 23, no. 2, pp. 149-162, March 1979.

[rubin 1979] F. Rubin, Arithmetic stream coding using fixed precision registers,IEEETransactions on Information Theory, vol. IT-25, no. 6, pp. 672-675, November 1979.

[sayood 1996] K. Sayood, Introduction to Data Compression, San Francisco, CA:Morgan Kaufmann Publishers, 1996.

[shannon 1948] C. E. Shannon, A mathematical theory of communication, BellSystem Technical Journal, vol. 27, pp. 379-423 (Part I), July 1948, pp. 623-656

(Part II), October 1948. decodability

coding techs

Documents