probability theory 2011 convergence concepts in probability theory definitions and relations...

Probability theory 2011

Convergence concepts in probability theory

Definitions and relations between convergence concepts Sufficient conditions for almost sure convergence Convergence via transforms The law of large numbers and the central limit theorem


Coin-tossing: relative frequency of heads

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 50 100 150 200 250 300 350 400

Re

lati

ve

fre

qu

en

cy

of

he

ad

s

Series1

Series2

Series3

Convergence of each trajectory?

Convergence in probability?


Convergence to a constant

The sequence {Xn} of random variables converges almost

surely to the constant c if and only if

P({ ; Xn() c as n }) = 1

The sequence {Xn} of random variables converges in

probability to the constant c if and only if, for all > 0,

P({ ; | Xn() – c| > }) 0 as n


An (artificial) example

Let X1, X2,… be a sequence of independent binary random variables such

that

P(Xn = 1) = 1/n and P(Xn = 0) = 1 – 1/n

Does Xn converge to 0 in probability?

Does Xn converge to 0 almost surely?

Common exception set?


The law of large numbers for random variableswith finite variance

Let {Xn} be a sequence of independent and identically distributed random

variables with mean and variance 2, and set

Sn = X1 + … + Xn

Then

Proof: Assume that = 0. Then

.

0 ,0)|(| all foras nn

SP n

2

)()|(|

n

SVar

n

SP

n

n


Convergence to a random variable: definitions

The sequence {Xn} of random variables converges almost surely to the

random variable X if and only if

P({ ; Xn() X() as n }) = 1

Notation:

The sequence {Xn} of random variables converges in probability to the

random variable X if and only if, for all > 0,

P({ ; | Xn() – X()| > }) 0 as n

Notation:

nXX san .. as

nXX pn as


Convergence to a random variable: an example

Assume that the concentration of NO in air is continuously

recorded and let Xt, be the concentration at time t.

Consider the random variables:

Does Yn converge to Y in probability?

Does Yn converge to Y almost surely?

tt XY 10max

1/2/10 ,...,,,max XXXXY nnn


Convergence in distribution: an example

Let Xn Bin(n, c/n). Then the distribution of Xn converges to

a Po(c) distribution as n

.

Binomial and Poisson distributions (n = 20, c = 0.1)

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0 1 2 3 4 5 6 7 8 9 10

Pro

bab

ilit

y

BinomialPoisson

p = 0.1)


Convergence in distribution and in norm

The sequence Xn converges in distribution to the random variable X as

n iff

for all x where FX(x) is continuous.

Notation:

The sequence Xn converges in quadratic mean to the random variable X

as n iff

Notation:

nxFxF XX n )()( as

nXXE n 0 || 2 as

nXX dn as

nXX n as 2


Relations between the convergence concepts

Almost sure convergence

Convergence in r-mean

Convergence in probability

Convergence in distribution


Convergence in probability impliesconvergence in distribution

Note that, for all > 0,

||

||

XXxXP

XXxXP

xXP

nn

nn

n


Convergence almost surely -convergence in r-mean

Consider a branching process in which the offspring distribution has mean 1.

Does it converge to zero almost surely?Does it converge to zero in quadratic mean?

Let X1, X2,… be a sequence of independent random variables such that

P(Xn = n2) = 1/n2 and P(Xn = 0) = 1 – 1/n2

Does Xn converge to 0 in probability?

Does Xn converge to 0 almost surely?

Does Xn converge to 0 in quadratic mean?


Relations between different types ofconvergence to a constant

Almost sure convergence

Convergence in r-mean

Convergence in probability

Convergence in distribution


Convergence via generating functions

Let X, X1, X2, … be a sequence of nonnegative, integer-

valued random variables, and suppose that

Then

ntgtg XX n )()( as

nXX dn as

Is the limit function of a sequence of generating functions a generating function?


Convergence via moment generating functions

Let X, X1, X2, … be a sequence of random variables, and

suppose that

Then

htntt XX n ||, )()( foras

nXX dn as

Is the limit function of a sequence of moment generating functions a moment generating function?


Convergence via characteristic functions

Let X, X1, X2, … be a sequence of random variables, and

suppose that

Then

tntt XX nforas , )()(

nXX dn as

Is the limit function of a sequence of characteristic functions a characteristic function?


Convergence to a constantvia characteristic functions

Let X1, X2, … be a sequence of random variables, and

suppose that

Then

net itcX n

)( as

ncX pn as


The law of large numbers(for variables with finite expectation)

Let {Xn} be a sequence of independent and identically

distributed random variables with expectation , and set

Sn = X1 + … + Xn

Then

.

nn

SX pn

n as


The strong law of large numbers(for variables with finite expectation)

Let {Xn} be a sequence of independent and identically

distributed random variables with expectation , and set

Sn = X1 + … + Xn

Then

.

nn

SX san

n as..


The central limit theorem

Let {Xn} be a sequence of independent and identically distributed random

variables with mean and variance 2, and set

Sn = X1 + … + Xn

Then

Proof: If = 0, we get

.

nNn

nS dn as)1,0(

nn

XS

n

S n

to

n

t

n

t

n

tt

nn

)(2

1)()()(22


Rate of convergence in the central limit theorem

Example: XU(0,1)

.

n

XExxF

n

nSx n 3

3||7975.0|)()(|sup

nnn

XE 3.8

)1212/(1

4/17975.0

||7975.0

3

3


Sums of exponentially distributed random variables

0.00

0.20

0.40

0.60

0.80

1.00

1.20

0 5 10 15 20 25

Cu

mu

lati

ve d

istr

ibu

tio

n f

un

ctio

n

gamma(10;1) N(10;sqr(10))

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0 5 10 15 20 25

Pro

bab

ility

den

sity

gamma(10;1) N(10;sqr(10))


Convergence of empirical distribution functions

Proof: Write Fn(x) as a sum of indicator functions

Bootstrap techniques: The original distribution is replaced with the empirical distribution

))(1)((,0())()(( xFxFNxFxFn dn

n

xnsobservatioxFn

#)(


Resampling techniques- the bootstrap method

**2

*1 ,...,, Nxxx

3467

798839

41

8570

62

90 58 4460

73

22

587988

41

88

8570

90

22 34 4460

41

60Sampling with replacement

Resampled dataObserved data

x **2

*1 ...,,, Nxxx


Characteristics of infinite sequences of events

Let {An, n = 1, 2, …} be a sequence of events, and define

Example: Consider a queueing system and let

An = {the queueing system is empty at time n}

1* inflim

n nmmnn AAA

1

* suplimn nm

mnn AAA


The probability that an event occurs infinitely often - Borel-Cantelli’s first lemma

Let {An, n = 1, 2, …} be an arbitrary sequence of events. Then

Example: Consider a queueing system and let

An = {the queueing system is empty at time n}

0.).()(1

oiAPAP nn

n

Is the converse true?


The probability that an event occurs infinitely often- Borel-Cantelli’s second lemma

Let {An, n = 1, 2, …} be a sequence of independent events. Then

1.).()(1

oiAPAP nn

n


Necessary and sufficient conditions for almost sure convergence of independent random variables

Let X1, X2, … be a sequence of independent random variables. Then

)|(|01

..

nn

san XPnasX


Exercises: Chapter VI

6.1, 6.6, 6.9, 6.10, 6.17, 6.21, 6.25, 6.49

probability theory 2011 convergence concepts in probability theory definitions and relations...

Documents

sequence x n

distribution of x n

x n binn

probability convergence

n notation

random variable x

sure convergence convergence

n iff notation