Coding & Information theory

Coding & Information theoryBy: Shiva Navabi January, 29th 2011OverviewInformation theory and its importanceCoding theory and its subsectionsStatements of ShannonConclusion ReferencesQuestions & ambiguities !

What is Information theory mainly discussing?Information theory provided the possibility to measure the amount of information regardless of its content Information theory deals with the amount of information, not its content

Claude ShannonThe first one who brought up the idea of measuring the amount of information, was Claude ShannonShannon published his landmark paper A Mathematical Theory of Communication in 1948 and created a turning point in the field of digital communications and Information theory was first established on a firm basis

Significant aspects of Information TheoryIt sets bounds on what can be done but does little to aid in design of a particular systemDespite most bodies of knowledge, Information theory gives a central role to errors (noise) and consider it as an issue that already exists and we have to deal with thatPresents the ideal model of a communication system with considering the limits well face in practice and recognizes the maximum efficiency of the channel in the presence of noiseClarifies the concept of information and particularly discriminates theses two different concepts: amount & meaning of informationProvides mathematical tools for measuring the amount of informationRepresents Information in terms of bits and established the basis of digital communications and more generally, digitized all electronic devices

The conventional signaling system is modeled as bellow:1.An information source 2.An encoding of this source3.A channel (the medium through which the signal is transmitted)4.A noise (error) source that is added to the signal in the channel5.A decoding of the original information from the contaminated received signal6.A sink for the informationNoiseChannelDecodeEncodeSourceSink412356Elaboration of the blocks of the typical signaling system:1. Information sourceAny arbitrary information source is represented as a sequence of symbols in a source alphabet s1 , s2 , . . . , sq having q symbols.The power of both coding and information theory is that they do not define what information is; but try to measure the amount of it insteadContinuous forms of information (analog signals) are mostly sampled and digitized due to 2 main reasons: 1. storage, manipulation, transmission and generally processing of digital signals is much more reliable that analog signals 2. Integrated circuits and specially computers are all digital devices; therefore working with digital signals would be much easier

2&5. Encoding & Decoding Both encoding and decoding blocks implicitly should be divided into two parts.Encoding block consists of two main stages: 1. Source encoding 2. Channel encoding 1. Source encoding This stage of encoding basically deals with how to encode the mentioned source symbols more efficiently; The more complex the structure of the encoding method is, the more efficient the channel capacity would be utilized. Therefore Data Compression is highly influenced and controlled by the method we use to encode our information source

2. Channel encoding Strategies and methods in channel encoding, mainly are devised in order to compensate for the alterations and damages that may be applied to the original signal by the inevitable noise of the channel through which the signal is transmitted; Channel encoding methods on the other hand, expands the length of the sent message but dont contain any information; In conclusion, at the Encode block there will be two stages as following: 1. Source encoding 2. Channel encoding At the Decode block, of course, these two stages will be reversed as following: 1. Channel decoding 2. Source decoding

Coding Theory entryFrom the moment we start dealing with strategies and ideas of encoding the information source, coding theory is being utilized and implemented.Information theory and Coding theory are greatly related to each other; hence, the boundaries of these two are not very clear most of the times. Coding theory tries to bring Information theory into application using efficient methods of encoding.Coding theory leads to information theory, and information theory provides bounds on what can be done by suitable encoding of the information.Channel Coding Methods of channel coding are generally divided into two parts: 1. Error-Detecting codes 2. Error-Correcting codes also/ Escape charactersWhy Error-Detecting codes? We require a highly reliable transmission whether through a certain distance (transmission) or through time (storage); Therefore a reliable transmission definitely requires error-detection. Note/ It is obviously not possible to detect an error if every possible symbol, or set of symbols, that can be received is a legitimate message. It is possible to catch errors only if there are some restrictions on what is a proper messageAn example of Error-Detecting codesSimple Parity Checks

The simplest way of encoding a binary message to make it error detecting is to count the number of 1s in the message, and then append a final binary digit chosen so that the entire message has an even number of 1s in it. The entire message is therefore of even parity. Thus to (n-1) message positions we append an nth parity-check position.At the receiving end the count of the number of 1s is made, and an odd number of 1s in the entire n position indicates that at least one error has occurred.Note/ parity-check is only able to detect odd number of errors; Therefore in this code a double error cannot be detected. Nor can any even number of errors be detected. Need for Error CorrectionA first question is: if a computer can find out there is an error, why it can not find out where it is?A very first strategy: retransmission of the messageIf there was any error detected at receiving end, then we could request for retransmission of the message.The first and simplest error-correcting code: triplication codeIn this strategy every message is repeated three times and at the receiving end a majority vote is taken. It is obvious that this system will correct isolated single errors.However this strategy is very inefficient and requires expensive equipments.

Rectangular CodesThe next simplest error-correcting codes are the rectangular codes where the information is arranged in the form of an m-1 by n-1 rectangle. A parity-check bit is then added to each row of m-1 bits to make a total of m bits. Similarly, a check bit is added for each column.The redundancy will be smaller the more the rectangle approaches a square. For square codes of side n, we have (n-1)2 bits of information and 2n-1 bits of checking along the sides.

Source EncodingA better idea!Variable-Length CodesThe advantage of a code in which the message symbols are of variable length is that sometimes the code is more efficient in the sense that to represent the same information we can use fewer digits on the average.If every symbol is as likely as every other one, then the block codes are about as efficient as any code can be.But if some symbols are more probable than others, then we can take advantage of this to make the most frequent symbols correspond to the shorter encodings, and the less frequent symbols correspond to the longer encodings.Example: Morse code

Some fundamental problems with variable-length codesAt the receiving end how do you recognize each symbol of the code?

How do you recognize the end of one code word and the beginning of the next?

Note/ The message may have other larger structures (plus its source symbols having different probabilities) which could be used to make the code more efficient. Example: in English the letter Q is usually followed by the letter U

Unique DecodingThe received message must have a single, unique possible interpretation Consider a code in which the source alphabet S has four symbols and they are to be encoded in binary as follows: s1 = 0 s2 = 01 s3 = 11 s4 = 00 The particular received message 0011 could be one of these two: 0011 = s4 , s3 or 0011 = s1 , s1 , s3 Thus the code is not uniquely decodable. Condition/ Only if every distinct sequence of source symbols has a corresponding encoded sequence that is unique, can we have a uniquely decodable signaling system. Instantaneous CodesDecision Tree: Consider the following code: s1 = 0 s2 = 10 s3 = 110 s4 = 111 In order to decode any received message which is encoded using above code words in a logical way, wed better draw a decision tree.

Heres the Decoding tree:

In this example the decoding is instantaneous since no encoded symbol is a prefix of any other symbol. Therefore: A certain code can be instantaneously decoded only if none of the code words is the prefix of any of the others.

Example: the following code is uniquely decodable but is not instantaneous because you dont know when one symbol is over without looking further: s1 = 0 s2 = 01 s3 = 011 s4 = 111In this code, some code words are prefix of other ones; therefore the code cannot be decoded instantaneously. Consider this string : 0111 . . . 1111 s4 s4 It can only be decoded by first going to the end and then identifying each code word. Thus, you cannot decide that whether you have the word that corresponds to the prefix, or if you must wait until more is sent to complete the word. => this will cause a time delay!

Huffman CodesConsider an information source with q symbols probabilities of which are taken in decreasing order:p1 p2 p3 . . . pqFor length of corresponding code words, the relationship will run in the opposite way because of efficiency; therefore:

The two least frequent source symbols of an efficient code have the same encoded lengths(why?).Huffman encoding includes two stages:1. Reduction process2. Splitting process

Limitation on the efficiency of different codesAn important question/ How efficient could be the best encoding method?Obviously the most efficient coding method is the one which compress the information as great as possible.Therefore the problem is to find a limitation on the amount of efficiency and data compression.

InformationThe concept of information in the field of communication and information theory is different from the common expression of information between people and these two separate meanings should not be confused; since they are not equivalent.

Definition of information in mathematicsA signaling system is set up so that it could transmit a message to the receiver where they already dont know what the content of the message is. Therefore, if you already know what youre gonna receive at the receiving end, the whole process is a waste and you wont get any information.(Example: p1 = 1 and all other pi = 0)On the other hand, if the probabilities are all very different, then when a symbol with a low probability arrives, you feel more surprised, get more information than when a symbol with a higher probability arrives.Thus information is somewhat inversely related to the probability of occurrence.I(p) is the mathematical function which will measure the amount of information, surprise and uncertainty in the occurrence of an event of probability p.

I(p) is represented as bellow:

With the base 2 for the logarithm, the unit of information is called bit.If we use base e then the unit would be called nat.For base 10 the unit would be called hartley.The are three things assumed about I(p):

1. I(p) 0 (a real nonnegative measure)2. I(p1 , p2) = I(p1) + I(p2) for independent events 3. I(p) is a continuous function of p

Now, the question is / if we get I(si) units of information when we receive the symbol si , how much do we get on the average?

EntropyThe concept of entropy in mathematics is similar to the one we have in other fields.Common concept of entropy Entropy used to represent a quantitative measure of the disorder of a system and inversely related to the amount of energy available to do work in an isolated system. The more energy has become dispersed, the less work it can perform and the greater the entropy would be.Concept of entropy in Information theoryIt will put a lower bound on the amount of efficiency and data compression in a certain code and it will tell us that for a specific distribution, how efficient the best coding method would be. Entropy will represent the minimum amount of memory that is occupied by a certain code to convey a constant amount of information.Mathematical relationship of entropyThe mathematical relationship of entropy is represented as bellow:

Which will tell us that on the average, over the whole alphabet of symbols si , how much information we will get. (information per symbol of the alphabet S)

How to maximize entropy?If you have no idea which one of the source symbols are going to arrive, then youll become most surprise and youll get the most amount of information; hence, The entropy will be maximum. Therefore:Whenever all source symbols are equally probable, the entropy of the distribution would become the maximum of all other ones.There also seem to be another factor affecting entropy. Example: For a die youll have : { , , , , , }

For a fair coin youll have : { , }So, which distribution has a greater entropy?Generally, the greater the uncertainty is, the greater the entropy would be.

Back to Information theoryInformation theory, represented by Shannon, includes two important statements of conclusion:Source-coding theoremAccording to this theorem, entropy of the information source represents the minimum amount of bits (memory) that you need to encode the source symbols and no coding method could compress the data more than that. However, with a suitable encoding strategy it is possible to approach to the value given by the entropy of the information source. Noisy-channel coding theoremReliable transmission of the message through a noisy channel is possible. However there is a limitation on the transmission rate at which the signal is travelling into the channel. The Channel Capacity sets a threshold on the transmission rate of the signal and can be approached via suitable coding methods.Channel CapacityThe inequality of Channel Capacity is represented by Shannon:

WhereC is the capacity of the channel (transmission rate) which is represented in bits per second. It shows the amount of discrete information in bits that the medium (channel) can hold.W is the bandwidth of the channel in hertz.S/N is the signal-to-noise ratio of the communication channel Note/ SNR is expressed as a straight power ratio (not as decibels)

Some examplesExample 1:Assume that:SNR = 20 dB ; W = 4 kHz Hence => 26.63 kbit/secNote/ the value of 100 is appropriate for an SNR of 20 dBMeaning that the transmission rate could not exceeds over 26.63 bit/sec

Example 2:Assume that :C = 50 kbit/sec ; W = 1 MHzHence =>

Meaning that for these circumstances, the minimum SNR should be 0.035 corresponding to an SNR of -14.5 dB.

This example shows that it is possible to transmit the signal while the original signal (carrying information) is much weaker than the background noise. However the Information theory wont tell us how? It only convinces us that it is possible.

ConclusionInformation theory recognizes the limitations of communication systems in practice and sets bounds on ideal transmission in the presence of noise; However it does not tell us how to achieve them.Coding theory generally, control the reliability of transmission (channel encoding) and affects the efficiency and data compression of the transmitted signal.In fact, Coding theory will help us to approach the bounds determined by Information theory in practice. It should be emphasized that it is hard to specify clear and distinct boundaries for these two theories; Your idea ?ReferencesW.Hamming, R. (1986). Coding and Infomation theory. Prentice-Hall.Forchhammer, J. J. (2010). Two-Dimentional Information and Coding Theory. Cambridge University Press.Tilborg, H. C. (1993). Coding Theory a first course. Eindhoven. (n.d.). Retrieved from wikipedia: http://www.wikipedia.org

Questions & ambiguities ?!

Challenging points & discussion !Thanks for your attention & participation