Binomial Random Variable Approximations,Conditional Probability Density Functionsand Stirling’s Formula

Let X represent a Binomial r.v ,Then from

for large n. In this context, two approximations are extremely useful.

2

1

2

1

.)(21

k

kk

knkk

kkn qp

k

nkPkXkP

.)()(

) " and between is of sOccurrence ("

2

1

2

1

211 1

21

k

kk

knkk

kkkkkk qp

k

nXPXXXP

kkAP

The Normal Approximation (Demoivre-Laplace Theorem)

Suppose with p held fixed. Then for k in the neighborhood of np, we can approximate

And we have:

where

n npq

.2

1 2/)( 2 npqnpkknk enpq

qpk

n

,2

1

2

1 2/

2/)(

21

22

1

22

1

dyedxenpq

kXkP yx

x

npqnpxk

k

. , 22

11

npq

npkx

npq

npkx

As we know,

If and are within with approximation:

where

We can express this formula in terms of the normalized integral

that has been tabulated extensively.

2

1

2

1

)(21

k

kk

knkk

kkn qp

k

nkPkXkP

1k 2k , , npqnpnpqnp

,2

1

2

1 2/

2/)(

21

22

1

22

1

dyedxenpq

kXkP yx

x

npqnpxk

k

)(2

1)(

0

2/2

xerfdyexerfx y

. , 22

11

npq

npkx

npq

npkx

x erf(x) x erf(x) x erf(x) x erf(x)

0.05

0.10 0.15 0.20 0.25

0.30 0.35 0.40 0.45 0.50

0.55 0.60 0.65 0.70 0.75

0.01994 0.03983 0.05962 0.07926 0.09871 0.11791 0.13683 0.15542 0.17364 0.19146 0.20884 0.22575 0.24215 0.25804 0.27337

0.80 0.85 0.90 0.95 1.00

1.05 1.10 1.15 1.20 1.25

1.30 1.35 1.40 1.45 1.50

0.28814 0.30234 0.31594 0.32894 0.34134 0.35314 0.36433 0.37493 0.38493 0.39435 0.40320 0.41149 0.41924 0.42647 0.43319

1.55

1.60 1.65 1.70 1.75

1.80 1.85 1.90 1.95 2.00

2.05 2.10 2.15 2.20 2.25

0.43943 0.44520 0.45053 0.45543 0.45994 0.46407 0.46784 0.47128 0.47441 0.47726 0.47982 0.48214 0.48422 0.48610 0.48778

2.30

2.35 2.40 2.45 2.50

2.55 2.60 2.65 2.70 2.75

2.80 2.85 2.90 2.95 3.00

0.48928 0.49061 0.49180 0.49286 0.49379 0.49461 0.49534 0.49597 0.49653 0.49702 0.49744 0.49781 0.49813 0.49841 0.49865

A fair coin is tossed 5,000 times.

Find the probability that the number of heads is between 2,475 to 2,525.

We need

Since n is large we can use the normal approximation.

so that and

and

So the approximation is valid for and

Example

Solution

).525,2475,2( XP

,2

1p 500,2np .35npq

,465,2 npqnp

475,21 k .525,22 k

,535,2 npqnp

Here,

Using the table,

Example - continued

2

1

2

.2

1 2/21

x

x

y dyekXkP

.7

5 ,

7

5 22

11

npq

npkx

npq

npkx

516.07

5erf2

|)(|erf)(erf

)(erf)(erf525,2475,2

12

12

xx

xxXP

. 258.0)7.0(erf

The Poisson Approximation

For large n, the Gaussian approximation of a binomial r.v is valid only

if p is fixed, i.e., only if and

What if is small, or if it does not increase with n?

- for example, as such that is a fixed number.

1np .1npq

np

0p ,n np

The Poisson Approximation

Consider random arrivals such as telephone calls over a line.

n: total number of calls in the interval

as we have

Suppose

Δ : a small interval of duration

).,0( T

T n

.Tn

0 T

1 2 n

The Poisson Approximation

p: probability of a single call (during 0 to T) occurring in Δ:

as

Normal approximation is invalid here.

Suppose the interval Δ in the figure:

- (H) “success” : A call inside Δ,

- (T ) “failure” : A call outside Δ

: probability of obtaining k calls (in any order) in an interval of duration Δ ,

.T

p

0p .T

T

Tnp

)(kPn

knkn pp

kkn

nkP

)1(

!)!(

!)(

0 T

1 2 n

The Poisson Approximation

Thus,

.)/1(

)/1(

!

11

21

11

)/1(!

)()1()1()(

k

nk

knk

kn

n

n

kn

k

nn

nnpk

np

n

knnnkP

e

kkP

k

nnppn !

)(lim ,0 ,

the Poisson p.m.f

Suppose

- two million lottery tickets are issued

- with 100 winning tickets among them.

a) If a person purchases 100 tickets, what is the probability of winning?

Example: Winning a Lottery

Solution

.105102

100

ticketsof no. Total

tickets winningof No. 56

pThe probability of buying a winning ticket

X: number of winning tickets

n: number of purchased tickets ,

P: an approximate Poisson distribution with parameter

So, The Probability of winning is:

Winning a Lottery - continued

.005.0105100 5 np

,!

)(k

ekXPk

.005.01)0(1)1( eXPXP

b) How many tickets should one buy to be 95% confident of having a winning ticket?

we need

But or

Thus one needs to buy about 60,000 tickets to be 95% confident of having a winning ticket!

Winning a Lottery - continued

Solution

.95.0)1( XP

.320ln implies 95.01)1( eXP

3105 5 nnp .000,60n

A space craft has 100,000 components

The probability of any one component being defective is

The mission will be in danger if five or more components become defective.

Find the probability of such an event.

n is large and p is small

Poisson Approximation with parameter

Example: Danger in Space Mission

Solution

n

).0(102 5 p

2102000,100 5 np

.052.03

2

3

42211

!1

!1)4(1)5(

2

4

0

24

0

e

ke

keXPXP

k

kk

k

Conditional Probability Density Function

, )( )( xXPxFX

.0)( ,)(

)()|(

BP

BP

BAPBAP

.

)(

)( |)( )|(

BP

BxXPBxXPBxFX

.0

)(

)(

)(

)( )|(

,1)(

)(

)(

)( )|(

BP

P

BP

BXPBF

BP

BP

BP

BXPBF

X

X

Conditional Probability Density Function

Further,

Since for

),|()|(

)(

)( )|)((

12

2121

BxFBxF

BP

BxXxPBxXxP

XX

,12 xx

. )()()( 2112 xXxxXxX

,

)|()|(

dx

BxdFBxf X

X

2

1

21 .)|(|)(x

x X dxBxfBxXxP

x

XX duBufBxF

.)|()|(

Toss a coin and X(T)=0, X(H)=1.

Suppose

Determine

has the following form.

We need for all x.

For so that

and

Example

Solution

}.{HB ).|( BxFX

)(xFX

)|( BxFX

,)( ,0 xXx ,)( BxX

.0)|( BxFX

)(xFX

x

(a)

q1

1

( | )XF x B

x

(b)

1

1

For so that

For and

Example - continued

, )( ,10 TxXx

HTBxX )( .0)|( and BxFX

,)( ,1 xXx

}{ )( BBBxX 1)(

)()|( and

BP

BPBxFX

( | )XF x B

x

1

1

Given suppose Find

We will first determine

For so that

For so that

Example

Solution

),(xFX .)( aXB ).|( Bxf X

).|( BxFX

.

)|(

aXP

aXxXPBxFX

xXaXxXax ,

.

)(

)()|(

aF

xF

aXP

xXPBxF

X

XX

)( , aXaXxXax .1)|( BxFX

Thus

and hence

Example - continued

, ,1

, ,)(

)()|(

ax

axaF

xFBxF

X

X

X

otherwise. ,0

,,)(

)()|()|(

axaF

xfBxF

dx

dBxf

X

X

XX

)|( BxFX

)(xFX

xa

1

(a)

)|( Bxf X

)(xf X

xa(b)

Let B represent the event with

For a given determine and

Example

Solution

.)()(

)( )(

)(

)()(

|)( )|(

aFbF

bXaxXP

bXaP

bXaxXP

BxXPBxF

XX

X

bXa )( .ab ),(xFX )|( BxFX ).|( Bxf X

For we have

and hence

For we have

and hence

For we have

so that

Thus,

Example - continued

,bxa })({ )( )( xXabXaxX

.

)()(

)()(

)()(

)()|(

aFbF

aFxF

aFbF

xXaPBxF

XX

XX

XXX

,bx bXabXaxX )( )( )( .1)|( BxFX

,ax ,)( )( bXaxX

.0)|( BxFX

otherwise.,0

,,)()(

)()|(

bxaaFbF

xfBxf

XX

X

X

)|( Bxf X

)(xf X

xa b

Conditional p.d.f & Bayes’ Theorem

First, we extend the conditional probability results to random variables:

We know that If is a partition of S and B is an arbitrary event, then:

By setting we obtain:

)()|()()|()( 11 nn APABPAPABPBP

)],,,[ 21 nAAAU

Son partition a form ,

)()|()()|()(

)()|()()|()(

Hence

)()|()()|()(

1

11

11

11

n

nn

nn

nn

AA

APAxfAPAxfxf

APAxFAPAxFxF

APAxXPAPAxXPxXP

}{ xXB

Conditional p.d.f & Bayes’ Theorem

Using:

We obtain:

For ,

.)(

)()|()|(

BP

APABPBAP

)(

| )(

|)| AP

xF

AxFAP

xXP

AxXPxXAP

21 )( xXxB

).()(

)|()(

)()(

)|()|(

)(|

|

2

1

2

1

12

12

21

2121

APdxxf

dxAxfAP

xFxF

AxFAxF

APxXxP

AxXxPxXxAP

x

x X

x

x X

XX

XX

Conditional p.d.f & Bayes’ Theorem

Let so that in the limit as

or

we also get

or

,0 , , 21 xxxx ,0

).()(

)|(|)(|lim

0AP

xf

AxfxXAPxXxAP

X

X

.)(

)()|()|(| AP

xfxXAPAxf X

AX

,)()|()|()(

1

dxxfxXAPdxAxfAP XX

dxxfxXAPAP X )()|()(

(Total Probability Theorem)

Bayes’ Theorem (continuous version)

using total probability theorem in

We get the desired result

,)(

)()|()|(| AP

xfxXAPAxf X

AX

.)()|(

)()|()|(|

dxxfxXAP

xfxXAPAxf

X

XAX

probability of obtaining a head in a toss.

For a given coin, a-priori p can possess any value in (0,1).

: A uniform in the absence of any additional information

After tossing the coin n times, k heads are observed.

How can we update this is new information?

Let A= “k heads in n specific tosses”.

Since these tosses result in a specific sequence,

and using Total Probability Theorem we get

Example: Coin Tossing Problem Revisited

Solution

)(HPp

)( pfP

)( pfP

,)|( knkqppPAP

)( pfP

p0 1

.)!1(

!)!()1()()|()(

1

0

1

0

n

kkndpppdppfpPAPAP knk

P

The a-posteriori p.d.f represents the updated information given the event A,

Using

This is a beta distribution.

We can use this a-posteriori p.d.f to make further predictions.

For example, in the light of the above experiment, what can we say about the probability of a head occurring in the next (n+1)th toss?

Example - continued

| ( | )P Af p A

,)(

)()|()|(| AP

xfxXAPAxf X

AX

|

( | ) ( )( | )

( )

( 1)! , 0 1 ( , ).

( )! !

PP A

k n k

P A P p f pf p A

P A

np q p n k

n k k

)|(| Apf AP

p10

Let B= “head occurring in the (n+1)th toss, given that k heads have occurred in n previous tosses”.

Clearly

From Total Probability Theorem,

Using (1) in (2), we get:

Thus, if n =10, and k = 6, then

which is more realistic compare to p = 0.5.

Example - continued

,)|( ppPBP

1

0 .)|()|()( dpApfpPBPBP P

,58.012

7)( BP

1

0

( 1)! 1( ) .

( )! ! 2k n kn k

P B p p q dpn k k n