INFORMATION THEORY

BYK.SWARAJA

ASSOCIATE PROFESSORMREC

Introduction

Communication : The purpose of communication system is to carry the

information bearing base band signals from one place to another on communication channel

This is done by number of modulation schemes

Information theory is a branch of probability theory which may be applied to the study of the communication system

Information theory deals with mathematical modelling and analysis of communication system rather than with physical sources and physical channels

Discrete memory less channels(DMC)

DMC is a statistical model with an input ‘x’ and an output ‘y’. During each unit of time ,the channel accepts an input symbol from ‘x’ and in response it generates an output symbol ‘y’.

The channel is ‘discrete ’when alphabets of x & y are both finite.

The channel is “memory less” when current output depends on only the current input and not on any of the previous input.

Each possible input-to-output path is indicated along with a conditional probability p(yi /xi)

INFORMATION:

Information contained in a message depends on its probability of occurrence.

If the probability of occurrence of a particular message is more, then it contains less amount of information and vice versa and approaches to zero for a probability of unity.

The probability occurrence of each possible message is known such that the given set of probabilities satisfies the condition given as

Information sources

A Practical source communication system is a device which produces messages(analog or discrete).

The set of source symbols is called the Source alphabet and the elements of the set are called Symbols or letters.

Information Sources can be classified as

a) memory

b) memory lessA source with memory is one for which current symbol

depends on the previous symbols.A memory less source is one for which each symbol

produced is independent of the previous symbols.

Amount of information

The amount of information contents contained in the signal is measured with the extent of uncertainty in the information

If pi=0 and pk=1 then there is no information generated. If source symbol occur with low probability then there

is more amount of information. If the source symbol occur with high probability then

there is less amount of information.

INFORMATION

Definition: If two independent messages are generated in

sequence , then total information should equal to the sum of individual information contents of two messages.

Total probability of composite message is the product of two individual probabilities

The probability of information must be such the when probability are multiplied together , the corresponding channels are added, the logarithmic function satisfies these requirements

The amount of information I(sk) gained after the event s=sk occurs with probability pk is defined as the logarithmic function.

I(sk) = log(1/pk) for k=0,1,2,3…….(k-1) I(sk) = -log2(pk)

The unit of information is a bit , its base is 2The amount of information can be expressed as

Entropy(Average information)

Entropy of a source is a measure of the average amount of information per source symbol in a long message.

Units: bits/symbol.Let the amount of information I(sk) generated

by a discrete source takes on the values I(so),I(s1),I(s2)…….I(Sk-1) with respective probabilities P0 ,P1 ,P2 , . . .P(SK-1 ).

The entropy of source alphabet ‘s’ is given as H(S)=

H(S)=

Since the entropy is a measure of uncertainty ,the probability distribution that generates the maximum uncertainty will have the maximum entropy.

Entropy of binary & multilevel memory less source

Consider a binary source for which symbol ‘0’ occurs with probability p0 and symbol ‘1’ occurs with probability ‘1’=(1-p0 ) i.e. the case of two equiprobable events.

The source entropy,

H(s) = p0 log2 (1/p0 )+ p1 log2 (1/p1 ) In case of binary level, the entropy becomes maximum (1bit/symbol) when

both the symbols are equiprobable. Similarly in case of multilevel, the entropy becomes maximum when all the

symbols are equiprobable. For equiprobable symbols,

H(S)=log2M ,

which gives minimum number of bits needed to encode the symbol.This ensures the minimum transmission bandwidth.

The source entropy H(S) satisfies the following relation

0≤ H(S) ≤ log2M

Lower bound occurs when Pk =1 and

Upper bound occurs when p(Si)=1/m i.e. for equiprobable events.

Various specific channels

1. Lossless channels: A channel described by a channel matrix with only one non-zero element in each column is called a “lossless channel”

2. Deterministic channel: A channel described by a channel matrix with only one non-zero element in each row.

3. Noiseless channel: A channel is called noiseless if it is both lossless and deterministic.

4. Binary symmetric channel(BSC):This channel is symmetric because the probability of receiving ‘1’if ‘o’ is send is same as probability of receiving a ‘0’ if a ‘1’ is sent.

p[y/x]=

Joint entropy

Definition : H(x, y) is the average uncertainty of the communication channel as a whole considering the entropy due to channel input as well as channel output.

Where p(xi ,yj ) is the joint probability of the average uncertainty of the channel input H(x) and the average uncertainty of the channel output H(y).

The entropy functions for a channel with ‘m’ inputs and ‘n’ outputs are given as

Where p(xi ) represents the input probabilities and

p(yj )represents the output probabilities.

Conditional entropy

H(x/y) and H(y/x),is a measure of the average uncertainty remaining about the channel input after the channel output has been observed and the channel output after the channel input has been observed respectively.

Mutual information

The mutual information I(x;y) of a DMC with the input alphabet x, the output alphabet y and conditional probabilities p(yk/xj),

where j=0,1,2,…….(j-1) and k=0,1,2, ……(k-1) is defined as

I{x;y}= where p(xj ,yk ) is the joint probability distribution of random

variables x and y and is defined as p(xj,yk)=p(yk/xj )p(xj)

where p(yk /xj ) is the conditional probabilities and given as

p(yk /xj )=p(y=yk /x=xj); for all values of ‘j’ and ‘k’.

P(yk ) is the marginal probability distribution of output random variable y and is given as

p(yk ) =In order to calculate mutual information, it

is necessary to know the input probability distribution, p(xj) where j=0,1,2,3,…..(j-1)

Mutual information of a channel depends not only on the channel but also on the way in which the channel is used.

Mutual information in terms of entropy

I(x;y)=H(x)-H(x/y) b/symbol,• where H(x) is the entropy of channel input and H(x/y) is the conditional entropy of channel input given the channel output.

llly , I(y;x)=H(y)-H(y/x)•The difference between entropy and conditional entropy must represent uncertainty about the channel output which is nothing but mutual information of the channel.

Properties

I(x;y)=I(y;x) -----symmetric

I(x;y)≥ 0 -------non negative property

I(x;y)=H(y)-H(y/x)

I(x;y)=H(y)+H(y)-H(x,y) -----joint entropy of input/output channel

Information rate If the time rate at which source ‘s’ emits symbols is

r(symbol, s) the information rate ‘R’ of the source is given by

R= r H(s) b/second

here, R=information rate

H(s) is entropy

and ‘r’ is the rate at which symbols are generated.

Units:

‘R’ is represented in average number of bits of information per second.it is calculated as

R=[r in symbols/sec]*[H(s) in information bits/symbol]

R=information bits/second

Shannon limit for channel capacity

Definition:

The maximum data rate at which information can be transmitted over a given communication channel, under given condition is referred to as the channel capacity

C=Blog2(1+S/N) bps for a given level of noise, the data rate or capacity

can be increased by increasing either bandwidth or signal strength.

If bandwidth B is increased , more noise is introduced to the system. Since noise is assumed to be white noise (N=KTB). Thus as B increases, SNR decreases.

Shannon’s theorem says that if R≤C, it is possible to transmit information without any error and if noise present coding techniques are used to correct and detect the errors.

The negative statement of shannon’s theorem says that if R>C then every message will be in error.

Noise less channel has infinite capacity i..e, if N=0 then S/N∞ then c=∞

Infinite bandwidth channel has finite capacity. This is because as bandwidth increases noise power also increases i.e. N=NOB

Due to this increase in noise power, signal to noise ratio decreases. Hence even if B∞ capacity does not approaches infinity.

Bandwidth and channel capacity

In a noise-free communication channel, the limitation on data rate of the transmitted signal is mainly constrained by the available bandwidth of the communication channel.

A communication channel cannot propagate a signal that contains a frequency which is changing at a rate greater than bandwidth of the channel.

The greater the bandwidth available the higher the information carrying capacity.

For binary information , the highest signal rate that can be supported is 2B bps in a given bandwidth of BHz.