16548 notes 1

8/15/2019 16548 Notes 1

1/61

1

16.548

Coding, Information Theory (and

Advanced Modulation

!rof. "ay #eit$en

%all 411

"ay&'eit$enuml.edu

8/15/2019 16548 Notes 1

2/61

)

Cla** Coverage

+ undamental* of Information Theory (4'ee-*

+ %loc- Coding ( 'ee-*+ Advanced Coding and modulation a* a 'ay

of achieving the /hannon Ca0acity ound2

Convolutional coding, trelli* modulation,and turo modulation, *0ace time coding (3'ee-*

8/15/2019 16548 Notes 1

3/61

Cour*e #e /ite

+ htt02faculty.uml.edu'eit$en16.548

Cla** note*, a**ignment*, other material* on

'e *ite

!lea*e chec- at lea*t t'ice 0er 'ee-

7ecture* 'ill e *treamed, *ee cour*e 'e*ite

http://faculty.uml.edu/jweitzen/16.548http://faculty.uml.edu/jweitzen/16.548

8/15/2019 16548 Notes 1

4/61

4

!rereui*ite* (#hat you need to

-no' to thrive in thi* cla**+ 16.6 or 16.584 (A !roaility cla**

+ /ome !rogramming (C, 9%, Matla

+ :igital Communication Theory

8/15/2019 16548 Notes 1

5/61

5

;rading !olicy

+ 4 Mini

+7em0el $iv com0re**or

+ Cyclic >edundancy Chec-

+ Convolutional Coder:ecoder *oft

deci*ion

+ Trelli* Modulator:emodulator

8/15/2019 16548 Notes 1

6/61

6

Cour*e Information and Te?t

%oo-*+ Coding and Information Theory y #ell*,

0lu* hi* note* from @niver*ity of Idaho

+ :igital Communication y /-lar, or !roa-i*

%oo-

+ /hannon* original !a0er (1B48

+ ther material on #e *ite

8/15/2019 16548 Notes 1

7/61

3

Claude /hannon ound* /cience

of Information theory in 1B48In hi* 1B48 0a0er, DD

A Mathematical Theory of Communication,EE

Claude F. /hannon formulated the theory of datacom0re**ion. /hannon e*tali*hed that there i* a

fundamental limit to lo**le** data com0re**ion.

Thi* limit, called the entro0y rate, i* denoted y

H . The e?act value of H de0end* on the

information *ource

8/15/2019 16548 Notes 1

8/61

8

8/15/2019 16548 Notes 1

9/61

B

Thi* i*

Im0ortant

8/15/2019 16548 Notes 1

10/61

1G

/ource Modeling

8/15/2019 16548 Notes 1

11/61

11

Hero order model*

It ha* een *aid, that if you get enough mon-ey*, and *it them do'n at enough

ty0e'riter*, eventually they 'ill com0lete the 'or-* of /ha-e*0eare

8/15/2019 16548 Notes 1

12/61

1)

ir*t rder Model

8/15/2019 16548 Notes 1

13/61

1

igher rder Model*

8/15/2019 16548 Notes 1

14/61

14

8/15/2019 16548 Notes 1

15/61

15

8/15/2019 16548 Notes 1

16/61

16

8/15/2019 16548 Notes 1

17/61

13

8/15/2019 16548 Notes 1

18/61

18

8/15/2019 16548 Notes 1

19/61

1B

8/15/2019 16548 Notes 1

20/61

)G

Heroth rder Model

8/15/2019 16548 Notes 1

21/61

)1

8/15/2019 16548 Notes 1

22/61

))

:efinition of Fntro0y

/hannon u*ed the idea* of randomne** and entro0y from the

*tudy of thermodynamic* to e*timate the randomne** (e.g.

information content or entro0y of a 0roce**

8/15/2019 16548 Notes 1

23/61

)

Juic- >evie'2 #or-ing 'ith

7ogarithm*

8/15/2019 16548 Notes 1

24/61

8/15/2019 16548 Notes 1

25/61

)5

8/15/2019 16548 Notes 1

26/61

)6

8/15/2019 16548 Notes 1

27/61

)3

8/15/2019 16548 Notes 1

28/61

)8

Fntro0y of Fngli*h Al0haet

8/15/2019 16548 Notes 1

29/61

)B

8/15/2019 16548 Notes 1

30/61

G

8/15/2019 16548 Notes 1

31/61

1

8/15/2019 16548 Notes 1

32/61

)

8/15/2019 16548 Notes 1

33/61

8/15/2019 16548 Notes 1

34/61

4

Kind of

Intuitive,

ut hard to

0rove

8/15/2019 16548 Notes 1

35/61

5

8/15/2019 16548 Notes 1

36/61

3

%ound* on Fntro0y

8/15/2019 16548 Notes 1

37/61

8/15/2019 16548 Notes 1

38/61

B

In thi* 0re*entation, 'ell di*cu** the

joint density of t'o random variale*.Thi* i* a mathematical tool for

re0re*enting the interde0endence of

t'o event*.

ir*t, 'e need *ome random

variale*.

7ot* of tho*e in %edroc-.

8/15/2019 16548 Notes 1

39/61

4G

7et N e the numer of day* red

lint*tone i* late to 'or- in a given'ee-. Then N i* a random varialeO

here i* it* den*ity function2

Ama$ingly, another re*ident of %edroc- i* late 'ith

e?actly the *ame di*triution. It*...

red* o**, Mr. /lateP

L 1 )

(L .5 . .)

L 1 )

8/15/2019 16548 Notes 1

40/61

41

L 1 )

(L .5 . .) >ememer thi* mean* that!(NQ Q .).

7et e the numer of day* 'hen /late i* late. /u00o*e 'e

'ant to record %T N and for a given 'ee-. o' li-ely

are different 0air*R

#ere tal-ing aout the joint density of N and , and 'e recordthi* information a* a function of t'o variale*, li-e thi*2

1 ) 1 .5 .1 .G5

) .15 .1 .G5

G .1 .1

Thi* mean* that

!(NQ and Q) Q .G5.

#e lael it f(,).

L 1 )

8/15/2019 16548 Notes 1

41/61

4)

L 1 )

(L .5 . .)

1 )

1 .5 .1 .G5) .15 .1 .G5

G .1 .1

The fir*t o*ervation to ma-e i* that

thi* oint 0roaility function

contain* all the information from

the den*ity function* for N and

('hich are the *ame here.

or e?am0le, to recover !(NQ, 'e

can add f(,1Sf(,)Sf(,.

.) The individual 0roaility function*recovered in thi* 'ay are calledmarginal .

Another o*ervation here i* that /late i* never late three day* in

a 'ee- 'hen red i* only late once.

L 1 )

8/15/2019 16548 Notes 1

42/61

4

L 1 )

(L .5 . .)

/ince he ride* to 'or- 'ith red (at lea*t until the directing career'or-* out, %arney >ule i* late to 'or- 'ith the *ame 0roaility

function too. #hat do you thin- the oint 0roaility function for

red and %arney loo-* li-eR

1 )

1 .5 G G

) G . G

G G .)

It* diagonalP

Thi* *hould ma-e *en*e, *ince in any

'ee- red and %arney are late the

*ame numer of day*.

Thi* i*, in *ome *en*e, a ma?imum

amount of interaction2 if you -no'

one, you -no' the other.

L 1 )

8/15/2019 16548 Notes 1

43/61

44

L 1 )

(L .5 . .)

A little

8/15/2019 16548 Notes 1

44/61

45

L 1 )

(L .5 . .)

1 )

1 .)5 .15 .1) .15 .GB .G6

.1 .G6 .G4

/ince 'e -no' the variale* N

and H (for /0oc- are

inde0endent, 'e can calculateeach of the oint 0roailitie*

y multi0lying.

or e?am0le, f(), Q !(NQ) and HQ

Q !(NQ)!(HQ Q (.(.) Q .G6.

Thi* re0re*ent* a minimal amount of

interaction.

8/15/2019 16548 Notes 1

45/61

46

:e0endence of t'o event* mean* that -no'ledge of one give*

information aout the other.

Lo' 'eve *een that the oint den*ity of t'o variale* i* ale to

reveal that t'o event* are inde0endent ( and , com0letely

de0endent ( and , or *ome'here in the middle ( and .

7ater in the cour*e 'e 'ill learn 'ay* to uantify de0endence.

/tay tuned.

A%%A :A%%A :P

8/15/2019 16548 Notes 1

46/61

4B

8/15/2019 16548 Notes 1

47/61

5G

8/15/2019 16548 Notes 1

48/61

8/15/2019 16548 Notes 1

49/61

5)

Conditional !roaility

another event

(

,(U(

B P

B A P B A P =

8/15/2019 16548 Notes 1

50/61

5

Conditional !roaility (contd

!(%UA

8/15/2019 16548 Notes 1

51/61

54

:efinition of conditional 0roaility

+ If !(% i* not eual to $ero, then the conditional 0roaility of A

relative to %, namely, the 0roaility of A given %, i*

!(AU% Q !(A %!(%

∩

!(A % Q !(% !(AU%

or

!(A % Q !(A !(%UA

∩ •

∩ •

8/15/2019 16548 Notes 1

52/61

55

Conditional !roaility

G.45G.)5 G.)5

A %

!(A Q G.)5 S G.)5 Q G.5G

!(% Q G.45 S G.)5 Q G.3G

!(A Q 1 < G.5G QG.5G

!(%Q 1

8/15/2019 16548 Notes 1

53/61

56

7a' of Total !roaility

If % % ,......, and % are mutually e?clu*ive event* of 'hich

one mu*t occur, then for any event A

!(A Q !(% !(AU% S !(%

1 ) -

1 1 )

,

( U ...... ( ( U ⋅ ⋅ + + ⋅

P A B P B P A Bk k )

P A P B P A B P B P A B( ( ( U ( ( U E E= ⋅ + ⋅

Special case of rule of Total Probability

8/15/2019 16548 Notes 1

54/61

53

%aye* Theorem

8/15/2019 16548 Notes 1

55/61

58

8/15/2019 16548 Notes 1

56/61

5B

;enerali$ed %aye* theorem

If % % and % are mutually e?clu*ive event* of 'hich

one mu*t occur, then

1 ) - , ,....

4((.......4((4((

4((4(

))11 k k

ii

i

B A P B P B A P B P B A P B P

B A P B P A B P

⋅++⋅+⋅

⋅=

-.),......,1,Qi for

8/15/2019 16548 Notes 1

57/61

6G

@rn !rolem*

+ A00lication* of %aye* Theorem

+ %egin to thin- aout conce0t* of Ma?imum

li-elihood and MA! detection*, 'hich 'e'ill u*e throughout codind theory

8/15/2019 16548 Notes 1

58/61

61

8/15/2019 16548 Notes 1

59/61

6)

8/15/2019 16548 Notes 1

60/61

6

8/15/2019 16548 Notes 1

61/61

Fnd of Lote* 1

16548 notes 1

Documents