Lecture 9: Weak Convergence - II

1. Convergence in distribution of real-valuedrandom variables

1.1 Convergence in distribution (weak convergence)1.2 Convergence in distribution of transformed random variables1.3 Subsequence approach to weak convergence

2. Convergence in distribution of random variableswith values in a metric space

2.1 Convergence in distribution (weak convergence)2.2 Weak convergence of transformed random variables2.3 Subsequence approach to weak convergence

3. Invariance principle

3.1 Process of step-summs of i.i.d. random variables3.2. Brownian motion (Wiener process)3.3 Central Limit Teorem and Donsker Invariance Principle3.4 Examples

1. Convergence in distribution

1.1 Convergence in distribution (weak convergence)

1

< Ωn,Fn, Pn > are probability spaces;

Xn is a real-valued random variables defined on a probabilityspace < Ωn,Fn, Pn > for every n = 0, 1, 2, . . ..Fn(x) = P (Xn ≤ x), x ∈ R1 is a distribution function of therandom variable Xn for every n = 0, 1, 2, . . ..CF is the set of continuity points for a distribution functionF (x).Fn(A) = P (Xn ∈ A), A ∈ B1 is the distribution of the randomvariable Xn for every n = 0, 1, 2, . . ..

Definition 9.1 . Random variables Xn converge in distri-

bution to X0 as n → ∞ (Xnd−→ X0 as n → ∞) or that is the

same their distribution functions Fn weakly converge to F0 asn→∞ (Fn ⇒ F0 as n→∞) iff

Fn(x)→ F0(x) as n→∞, x ∈ CF0.

(1) Alternatively, the term weak convergence is used instead ofterm convergence in distribution and the corresponding notationXn ⇒ X0 as n→∞ is applied directly to random variables Xn

instead of their distribution functions.

Theorem 9.1 (Skorokhod)*. Let random variablesXnd−→

X0 as n→∞. The it is possible to construct on some probabil-ity space < Ω,F , P > random variables Xn, n = 0, 1, 2, . . . suchthat (a) P (Xn ≤ x) = P (Xn ≤ x), x ∈ R1 for every n = 0, 1, . . .;(b) Xn

a.s−→ X0 as n→ 0.

1.3 Weak convergence via convergence of for trans-formed random variables

2

F (x) be some distribution function, F (A) be the correspondingprobability measure on Borel σ-algebra of subsets of real line;f(x) is a Borel function R1 → R1;Cf the set of continuity points of function f .

Theorem 9.2 (8.4). Let random variables Xnd−→ X0

as n → ∞ if and only if random variables Yn = f(Xn)d−→

Y0 = f(X0) as n→∞ for any real-valued measurable functionsf(x), x ∈ R1 a.s. continuous with respect to measure F0(A), i.e.,such that F0(Cf) = 1.

Theorem 9.2 (8.7). Random variables Xnd−→ X0 as n →

∞ if and only if Ef(Xn) → Ef(X0) as n → ∞ for any real-valued measurable bounded function f(x), x ∈ R1 a.s. continu-ous with respect to measure F0(A), i.e., such that F0(Cf) = 1.

A ∈ B1, ∂A is a boundary of the set A, i.e., the set of pointsx ∈ R1 such that (x−ε, x+ε)∩A 6= ∅ and (x−ε, x+ε)∩A 6= ∅for any ε > 0.

Theorem 9.3. Random variables Xnd−→ X0 as n → ∞ if

and only if P (Xn ∈ A)→ P (X0 ∈ A) as n→∞ for any A ∈ B1,such that F0(∂A) = 0.——————————-(a) P (Xn ∈ A) = EIA(Xn);

(b) ∂A is the set of discontinuity points of function IA(x);

(c) Fn(y) = EI(−∞,y](Xn);

(d) function I(−∞,y](x) has the boundary y;

3

(e) F0(y) = F0(y) − F0(y − 0) and, thus, F0(y) = 0 iffy ∈ CF0

.——————————-

1.3 Subsequence approach to weak convergence

Let an is a sequence of real numbers. Then an → a0 asn→∞ if and only if from an arbitrary subsequence nk →∞ ask →∞ one can select a subsequence n′r == nnr →∞ as r →∞such that (a) an′r → a as r → ∞, (b) a = a0 does not dependon choice of subsequences nk and n′r.

It follows from the definition of weak convergence that, in or-der to prove that distribution functions Fn(x) weakly convergeas n→ 0, one can use the following subsequence approach.

First, an arbitrary subsequence nk → ∞ as k → ∞ shouldbe selected. Second, it should be shown that a subsequencen′r = nnr can be selected from the first subsequence such thatFn′r(·) ⇒ F (·), where F (x) is a distribution function. Third, itshould be shown that the distribution function F (x) ≡ F0(x)does not depend on the choice of subsequences nk and n′r. ThenFn(·)⇒ F0(·) as n→∞.

Indeed, in this case for any x ∈ CF0we get Fn′r(x)→ F (x) =

F0(x) as r →∞. Thus, Fn(x)→ F0(x) as n→∞.

Let R be a subset of R1. A set S ⊆ R is dense in R1, ifinfy∈S |x− y| = 0 for every x ∈ R.

4

Let us choose a set S dense in R1. Note that S = s1, s2, . . .can be a countable subset of R1. Due to continuity from theright, any distribution function F (x) is completely determinedby its values in points of the set S In this sense, S can be referredto as a defining set.

Let Fn(x), n = 1, 2, . . . be an arbitrary sequence of distribu-tion functions.

(1) Let now nk ≥ 0 be an arbitrary sequence such that nk →∞ as k → ∞. Using Cantor’s diagonal method it is alwayspossible to find a subsequence n′r = nnr such that Fn′r(x)→ F (x)as r →∞ for all x ∈ S, where the limits F (x) ∈ [0, 1].——————————-(a) F

n(1)1

(s1), Fn(1)2(s1), Fn(1)3

(s1), . . .;

(b) Fn(1)k

(s1)→ F (s1) as k →∞;

(c) n(2)r = n(1)kr, r = 0, 1, . . .;

(d) Fn(2)r

(s2)→ F (s2) as r →∞;

(e) . . .;

(f) n′l = n(l)l , l = 1, 2, . . .;

(g) Fn′l(sm)→ F (sm) as l→∞ for every m = 1, 2, . . ..

——————————-(2) The function F (x), defined on the set S, is non-decreasing.

Using this fact one can always define this function at every pointx ∈ R1 \ S as the right limit of the values F (xk) for some se-quence of points xk ∈ S, xk > x, xk → x. The function F (x),defined on R1 in this way, is non-decreasing, continuous from theright, and F (x) ∈ [0, 1] for every x ∈ R1. So, it is a distributionfunction.

5

(3) However, it can be an improper distribution function,i.e., it can be such that F (+∞)−F (−∞) < 1, where F (±∞) =limx→±∞ F (x).

(4) For example, let Fn(x) = I[an,∞)(x), where an → a0 = ∞as n→∞. In this case, Fn(x)→ F0(x) ≡ 0 as n→∞.

(5) In order to prove that F (x) is a proper distribution func-tion (for any subsequences nk and n′r chosen as is describedabove), one should require that the initial family of distributionfunctions Fn(x), n = 0, 1, . . . be t ight that means that the fol-lowing condition holds:

T’: limK→∞maxn≥0(Fn(−K − 0) + 1− Fn(K))

= limK→∞maxn≥0 Fn([−K,K]) = 0.

(6) Condition T’ implies that, for any subsequence nk →∞as k → ∞, a subsequence n′r = nkr can be selected from thefirst subsequence in such a way that the distribution functionsFn′r(·) ⇒ F (·) as k → ∞, where F (x) is a proper distributionfunction.

(7) The family of distribution functions Fn(x), n = 0, 1, . . .is tight if and only if it is relatively compact, i.e., for any sub-sequence nk → ∞ as k → ∞, a subsequence n′r = nkr can beselected from the first subsequence in such a way that the dis-tribution functions Fn′r(·) ⇒ F (·) as k → ∞, where F (x) is aproper distribution function.

6

(8) Now, let us also require convergence of distribution func-tions Fn(x) in points of the defining set S:

W’: Fn(x)→ F0(x) as n→∞, for x ∈ S.

(9) Note that limits in are some numbers from the interval[0, 1]. The function F0(x), defined in W’, is automatically non-decreasing. But it is not required that the corresponding limitsof F0(x), as x tends to −∞ or +∞, be equal to 0 and 1, re-spectively. The function F0(x) can be continued to the wholereal line, as it was described above, by using right limits. It is aproper or improper distribution function.

(10) Condition W’ implies also that Fn(x) → F0(x) as n →∞, for x ∈ CF0

.

(11) If condition T’ holds, then, according (7) the family ofdistribution functions Fn(x), n = 0, 1, . . . is relatively compact.Condition W’ implies, in this case, that all limits F (x) = F0(x),x ∈ S and also that F (x) are a proper distribution function.Since S is a defining set, F (x) = F0(x), x ∈ R1. So, the dis-tribution function F (x) does not depend on the choice of thesubsequences nk and n′r.

(12) Summarizing the remarks made above one can concludethat, in order to prove weak convergence of distribution func-tions Fn(·) ⇒ F0(·) as n → ∞, it is sufficient to assume thatboth conditions T’ and W’ hold.

(13) Moreover, it can be easily shown that conditions T’ and

7

W’ are not only sufficient but also necessary for weak conver-gence.——————————-(a) Sufficiency statement was prove above;

(b) Let Fn(·)⇒ F0(·) as n→∞;

(c) Then condition W’ holds for any countable dense in R1 setS = sk such that sk ∈ CF0

, k = 1, 2, . . .;

(d) Obviously, the family of distribution functions is relativelycompact and, therefore, is is tight, i.e. condition T’ also holds.——————————-

2. Convergence in distribution of random variableswith values in a metric space

2.1 Convergence in distribution (weak convergence)

Let X be a metric space with a metric d(x, y) ((a) 0 ≤d(x, y) = d(y, x); (b) d(x, y) = 0→ x = y; (c) d(x, y)+d(y, z) ≥d(x, z)).

The space X is complete if for any fundamental sequence ofpoints xn ∈ X , i.e., a sequence such that d(xn, xm) → 0 asn,m → ∞, there exists a point x ∈ X such that d(xn, x) → 0as n→∞.

The space X is separable if there exists a countable subsetY = y1, y2, . . . ⊆ X such that mink≤n d(yk, x) → 0 as n → ∞for any point x ∈ X .

8

The term Polish space is used to indicate that X is a com-plete separable metric space. Below, X is always a Polish space.

A set K is a compact (set) in a Polish space X if there existsa countable set Y = y1, y2, . . . ⊆ X such that

mink≤n

supx∈K

d(yk, x)→ 0 as n→∞.

Let BX be the Borel σ-algebra of subsets of X (the minimalσ-algebra containing any ball Br(x) = y: d(x, y) ≤ r in thespace X ).

The space Rm is a particular example of a Polish space wit4hthe metric

d(x, y) = |x− y| =√

(x1 − y1)2 + · · ·+ (xm − ym)2.

Other examples that we will be interested in are the func-tional spaces of continuous functions, C[0, 1] of real valued ffunc-tions x =< x(t), t ∈ [0, 1] > with the uniform metric

dU(x, y) = supt∈[0,t]

|x(t)− y(t)|.

A random variable X = X(ω) defined on a probability space< Ω,F , P > with values in the Polish space X is a measurablefunction acting Ω→ X such that X−1(A) ∈ F for any A ∈ BX .

The distribution of a random variable X is a probability mea-sure

FX(A) = P (X ∈ A) = P (X−1(A)), A ∈ BX .In the case of X = Rm a random variable X = (X1, . . . , Xm)

is a random vector.

9

In the case of X = C([0, 1]), a random variable X =< X(t), t ∈ [0, T ] is a real-valued continuous stochastic process.

Let Xn, n = 0, 1, . . . be a sequence of random variables (de-fined on probability spaces< Ωn,Fn, Pn > are probability spaces)that take values in X . We denote by Fn(A) = PXn ∈ A,A ∈ BX , the distribution of the random variable Xn.

Let ∂A denote the boundary of the set A, i.e., the set ofpoints x such that every ball Br(x), with centre in x and a ra-dius r > 0, has non-empty intersections with both sets A andA. If F0(∂A) = 0, then A is called a set of continuity for thedistribution F0. The class of such sets, B(F0), is a σ-algebra ofsubsets from BX .

Definition 9.2. Random variables Xn converge in distribu-

tion to X0 as n → ∞ (Xnd−→ X0 as n → ∞) or equivalently,

distributions Fn weakly converge to F0 as n → ∞ (Fn ⇒ F0 asn→∞) iff

Fn(A)→ F0(A) as n→∞, A ∈ B(F0).

Definition 9.3. Random variables Xn a.s. converge to X0

as n→∞ (Xna.s.−→ X0 as n→∞) iff

P ( limn→∞Xn(ω) = X0(ω)) = 1.

Lemma 9.1. If Xna.s.−→ X0 as n → ∞ then Xn

d−→ X0 asn→∞.

Theorem 9.4 (Skorokhod)**. Let random variablesXnd−→

X0 as n → ∞. The it is possible to construct on some prob-ability space < Ω,F , P > random variables Xn, n = 0, 1, 2, . . .

10

such that (a) P (Xn ∈ A) = P (Xn ∈ A), A ∈ BX for everyn = 0, 1, . . .; (b) Xn

a.s−→ X0 as n→∞, i.e.,

2.2. Convergence in distribution of transformed ran-dom variables

If f(x) be a measurable real-valued function defined on aspace Polish space X (the inverse image of any Borel set in R1

is a Borel set in X ), and X is a random variable with values inX ) then f(X) is a real-valued random variable.F (A) be the corresponding probability measure on Borel σ-algebra of subsets of real line;f(x) is a Borel function R1 → R1;Cf the set of continuity points of function f .

Theorem 9.5. Random variables Xnd−→ X0 as n → ∞ if

and only if Ef(Xn) → Ef(X0) as n → ∞ for any real-valuedmeasurable bounded function f(x), x ∈ R1 a.s. continuous withrespect to measure F0(A), i.e., such that F0(Cf) = 1.

——————————-(a) Let assume that Ef(Xn)→ Ef(X0) as n→∞ for any real-valued bounded Borel function f defined on X and such thatF0(Cf) = 1.

(b) The indicator function IA(x) of a Borel set A is a measur-able function and it has the set of discontinuity points, ∂A. Thecondition F0(∂A) = 0 means that IA(x) is an a.s. continuousfunction with respect to the probability measure F0.

(c) Thus, EIA(Xn) = Fn(A)→ EIA(X0) = F0(A) as n→∞ forall sets of continuity for the limiting distribution F0.

11

(d) Let Xnd−→ X0 as n → ∞. Construct on some probabil-

ity space < Ω,F , P > random variables Xn, n = 0, 1, 2, . . . suchthat P (Xn ≤ x) = P (Xn ≤ x), x ∈ R1 for every n = 0, 1, . . .;and Xn

a.s−→ X0 as n→ 0.

(d) f(Xn)a.s−→ f(X0) as n→∞.

(e) By Lebesgue theorem Ef(Xn)→ Ef(X0) as n→∞.

(f) P (f(Xn) ≤ x) = P (f(Xn) ≤ x), x ∈ R1 for every n =0, 1, . . .;

(g) Ef(Xn)→ Ef(X0) as n→∞.——————————-

Theorem 9.6. Let random variables Xnd−→ X0 as n → ∞

if and only if random variables Yn = f(Xn)d−→ Y0 = f(X0) as

n → ∞ for any real-valued measurable functions f(x), x ∈ R1

a.s. continuous with respect to measure F0(A), i.e., such thatF0(Cf) = 1.

——————————-(a) Let assume that f(Xn)

d−→ f(X0) as n → ∞ for any real-valued Borel function f defined on X and such that such thatF0(Cf) = 1.

(b) Then f(Xn)d−→ f(X0) as n → ∞ for any real-valued

bounded Borel function f defined on X and such that such thatF0(Cf) = 1.

(c) Then, by Helly theorem, Ef(Xn)→ Ef(X0) as n→∞.

(d) Let Xnd−→ X0 as n → ∞. Construct on some probabil-

ity space < Ω,F , P > random variables Xn, n = 0, 1, 2, . . . such

12

that P (Xn ≤ x) = P (Xn ≤ x), x ∈ R1 for every n = 0, 1, . . .;and Xn

a.s−→ X0 as n→ 0.

(b) Let Yn = f(Xn), n = 0, 1, . . ..

(c) P (Yn ≤ x) = P (Xn ∈ f−1((−∞, x])

= P (Xn ∈ f−1((−∞, x]) = P (Yn ≤ x).

(d) A = ω : Xn(ω) → X0(ω), B = ω : X0(ω) ∈ Cf. ThenP (A) = 1, P (B) = 1 and, therefore, P (A ∩B) = 1.

(d) If ω ∈ A∩B then Yn(ω) = f(Xn(ω))→ Y0(ω) = f(X0(ω)).

(e) Thus, Yna.s.−→ Y0 and, therefore, Yn

d−→ Y0.

(f) Thus Ynd−→ Y0.

——————————-

2.3 Subsequence approach to weak convergence

Theorem 9.7**. Distributions Fn(·) ⇒ F0(·) as n → ∞ ifand only any subsequence nk → ∞ as k → ∞ contains a sub-sequence n′r = nkr → ∞ as r → ∞ such that Fn′r(·) ⇒ F0(·) ask →∞.

The notions of tightness and relative compactness for a fam-ily of distributions play a principle role in the theory. Let usintroduce the following condition:

T: There exists a sequence of compact sets Km ⊆ X , m =1, 2, . . ., such that limm→∞maxn≥0 Fn(Km) = 0.

Definition 9.4. A family of distributions Fn, n ≥ 0, is tightif condition K holds.

13

Definition 9.5. A family of distributions Fn, n ≥ 0, is rela-tively compact, if any subsequence nk →∞ as k →∞ containsa subsequence n′r = nkr →∞ as r →∞, such that distributionsFn′r weakly converge to some probability measure F as r → ∞(possibly depending on the subsequences nk and n′r).

The following theorem plays a fundamental role in the theory.

Theorem 9.8 (Prokhorov)**. A family of probabilitymeasures Fn, n ≥ 0, is relatively compact if and only if it istight.

Also, the notion of defining class for a distribution is also im-portant.

Definition 9.6. A class of sets DF from the σ-algebra BX isa defining class for a probability measure F , if any probabilitymeasure F ′ that takes the same values as F on sets from theclass DF coincides with F .

Let us introduce the following condition:

W: Fn(A) → F0(A) as n → ∞ for A ∈ DF0, where DF0

issome defining class for the distribution F0.

Theorem 9.9 (Prokhorov)**. Conditions T and W arenecessary and sufficient for the weak convergence Fn ⇒ F0 asn→∞.

14

——————————-(a) It follows from Theorem 9.5 that the family of distributionsFn is relatively compact;

(b) By condition W, any weakly converging subsequences Fn′rhas the same limiting distribution F ≡ F0;

(c) Thus, by Theorem 9.4, Fn ⇒ F0 as n→∞;

(d) If Fn ⇒ F0 as n → 0, then the family of distributions Fn,n ≥ 0, is relatively compact;

(e) Therefore, due to Theorem 9.5, this family of distributionsis also tight;

(f) Also, the class of sets of continuity for the distribution F0

is a defining class for this distribution (DF0is a σ-algebra and

σ(DF0) = DX ;

(g) Thus, condition W holds.——————————-

3. Invariance principle

3.1 Process of step-summs of i.i.d. random variables

< Ω,F , P > is a probability space;

Xn, n = 0, 1, 2, . . . are independent identically distributed (i.i.d.)random variables defined on a probability space < Ω,F , P >;Z = N(0, 1) is standard normal random variable with the dis-

tribution function F (x) = 1√nσ

∫ x−∞ e

−y2

2 dy,−∞ < x <∞.

CLT: If E|X1|2 < ∞, EX1 = a, V arX1 = σ2 > 0, then

15

random variables

Wn =X1 + · · ·+Xn − an√

nσd−→ Z as n→∞ (1)

that means

P (Wn ≤ x)→ F (x) as n→∞, x ∈ R1. (2)

Let us construct the continuous stochastic process Xn =<Wn(t), t ∈ [0, 1] > based on sums Wn, n = 0, 1, . . . in the follow-ing way:

(1) Define the values Wn(kn) = Wk, k = 0, 1, . . . , n;

(2) Define the values Wn(t) = Wk(kn)+n(t− k

n)(Wk+1−Wk),kn ≤

t ≤ k+1n , k = 0, . . . , n − 1, i.e., by connecting points (kn ,Wk)

and (k+1n ,Wk+1) by the segment of strait line, for every k =

0, 1, . . . , n− 1.

One can consider the continuous stochastic process Xn asrandom variable taking value in the Polish space X = C[0, 1].

3.1 Brownian motion (Wiener process)

Let us consider a stochastic X0 =< W0(t), t ∈ [0, 1] >, which:(a) is a continuous process;(b) has independent increments, i.e., for any 0 ≤ t0 ≤ t1 ≤ · · · ≤tm ≤ 1 random variables W0(tk+1) −W0(tk), k = 0, 1, . . . , tm−1are independent for every m ≥ 1;(c) increments W0(t + s) − W0(t) has a Gaussian distributionwith expectation zero and variance s, for 0 ≤ t ≤ t+ s ≤ 1.

16

The process X0 is called a Brownian motion (or Wiener pro-cess), It can be considered as a random variable taking valuesin the Polish space X = C[0, 1].

The distribution of this random variable F0(A) = P (X0 ∈ A)on the Borel σ-algebra of space X = C[0, 1] is called the Wienermeasure.

3.3 Central Limit Teorem and Donsker InvariancePrinciple

Let f(·) be a measurable real-valued function (functional)acting from space C[0, 1]→ R1;

Cf is the set of continuity of the functional f(·) in the uni-form metrics, i.e., the set of continuous functions x0(·) =<x0(t), t ∈ [0, 1] > such that f(xn(·)) → f(x0) as n → ∞ ifdU(xn(·), x0(·)) = supt∈[0,1] |xn(t)− x0(t)| → 0 as n→∞.

Theorem 9.10 (Donsker Invariance Principle). If E|X1|2 <∞, EX1 = a, V arX1 = σ2 > 0, then random variables

f(Xn)d−→ f(X0) as n→∞. (3)

for any functional f(·) a.s. continuous with respect to Wienermeasure, i.e., such that F0(Cf) = 1.——————————-(a) (Wn(t1), . . . ,Wn(tm))

d−→ (W0(t1), . . . ,W0(tm)) as n → ∞for every 0 ≤ t0 ≤ t1 ≤ · · · ≤ tm ≤ 1,m ≥ 1;

(b) limc→∞maxn≥0 P (∆c(Wn(·)) > δ) = 0, δ > 0,

where ∆c(Wn(·)) = sup0≤t≤t+s≤t+c≤1 |Wn(t+ s)−Wn(t)|;(c) limN→∞maxn≥0 P (sup0≤t≤1 |Wn(t)| > N) = 0;

17

(d) Relation (a) implies that condition W holds for random vari-ables Xn;

(e) Relations (b) and (c) imply that condition K holds for ran-dom variables Xn;

——————————-

3.4 Examples

(1) f(x(·)) = mt′,t′′(x(·)) = supt′≤t≤t′′ x(t), 0 ≤ t′ ≤ t′′ ≤ 1;

Cf = C[0, 1];

mt′,t′′(Wn(·)) d−→ mt′,t′′(W0(·)) as n→∞.

(2) f(x(·)) = τa(x(·)) = inf(t : x(t) ≥ a) ∧ 1, a ≥ 0;

Cf = x(·) : m0,t′ 6= m0,t′′ = a, 0 ≤ t′ < t′′ ≤ 1;

τa(Wn(·)) d−→ τa(W0(·)) as n→∞.(3) f(x(·)) = Ig(x(·)) =

∫ T0 g(x(s))ds where g is a real valued

continuous function defined on a real line;

Cf = C[0, 1];

Ig(Wn(·)) d−→ Ig(W0(·)) as n→∞.

LN Problems

1. Prove that a family of distribution functions Fn(x), n =0, 1, . . . is tight if it is relatively compact.

2. Prove that weak convergenceXnd−→ X0 as n→∞ implies

condition T’ are not only sufficient but also necessary by directestimation of limK→∞maxn≥0 P (|Xn| ≥ K).

18

3. Prove that Cf = C[0, 1] in the Example (1) above.

4. Prove that Cf = x(·) : m0,t′ 6= m0,t′′ = a, 0 ≤ t′ < t′′ ≤1 in the Example (2) above.

5. Prove that Cf = C[0, 1] in the Example (3) above.

19