Karhunen-Loève decomposition of Gaussian
measures on Banach spaces
Jean-Charles Croix
GT APSSE - April 2017, the 13th
joint work with Xavier Bay.
1 / 29
Sommaire
1 Preliminaries on Gaussian processes
2 Gaussian measures on Banach spacesDenitionsKarhunen-loève decomposition in the Hilbert caseKarhunen-loève decomposition on Banach spaces
3 Examples on continuous functions
4 Conclusions
2 / 29
Preliminaries on Gaussian processes
Square integrable stochastic process
Consider:
T a non-empty set,
(Ω,F ,P) a probability space.
A square integrable stochastic process is a collection X = (Xt)t∈T ofsquare integrable real random variables:
∀t ∈ T , Xt : (Ω,F ,P)→ (R,B(R)) ∈ L2(Ω)
Mean function and covariance kernel
Let X be a square integrable process, then
∀t ∈ T , m(t) = E [Xt ],
∀(t, s) ∈ T 2, K (s, t) = E [(Xt −m(t))(Xs −m(s))].
From now on, all processes will be centered: m ≡ 0.
3 / 29
Preliminaries on Gaussian processes
Square integrable stochastic process
Consider:
T a non-empty set,
(Ω,F ,P) a probability space.
A square integrable stochastic process is a collection X = (Xt)t∈T ofsquare integrable real random variables:
∀t ∈ T , Xt : (Ω,F ,P)→ (R,B(R)) ∈ L2(Ω)
Mean function and covariance kernel
Let X be a square integrable process, then
∀t ∈ T , m(t) = E [Xt ],
∀(t, s) ∈ T 2, K (s, t) = E [(Xt −m(t))(Xs −m(s))].
From now on, all processes will be centered: m ≡ 0.
3 / 29
Preliminaries on Gaussian processes
The covariance kernel K is a symmetric kernel:
∀(s, t) ∈ T 2, K (s, t) = K (t, s),
and of positive type:
∀n ∈ N, ∀(t1, ..., tn) ∈ T , ∀(α1, ..., αn) ∈ Rn,
n∑i=1
αiαjK (ti , tj) ≥ 0.
In short, K is a positive denite kernel on T .
Gaussian process
A square integrable stochastic process X is Gaussian if and only if:
∀n ∈ N, ∀(t1, ..., tn) ∈ T n, (Xt1 , ...,Xtn) is a Gaussian vector.
Equivalently,
∀µ =n∑
i=1
αiδti , 〈X , µ〉 =n∑
i=1
αiXti is a real Gaussian variable.
4 / 29
Preliminaries on Gaussian processes
Gaussian space
Let X be a Gaussian process X on T , the associated Gaussian space is
HX = Xt , t ∈ T L2(Ω,F,P)
The mapping
U : Z ∈ HX → (t → 〈Z ,Xt〉L2) ∈ RT
is a linear and injective.
Reproducing Kernel Hilbert space (RKHS)
Let X be a Gaussian process on T , the associated RKHS is
HX = U(HX ).
It is a Hilbert space with 〈f , g〉HX= 〈U−1(f ),U−1(g)〉HX
as innerproduct.
5 / 29
Preliminaries on Gaussian processes
The previous mapping U is an isometric isomorphism, thus
HK ≡ HK as Hilbert spaces.
Moore-Aronszajn theorem
Let T a set and K a positive-denite kernel on T , then there exists aunique Hilbert subspace HK ⊂ RT such that:
1 HK = K (t, .), t ∈ T ,2 ∀t ∈ T , ∀h ∈ HK , h(t) = 〈h,Kt〉HK
.
Conversely, let T be a set and K a positive denite symmetric kernel,there exists a Gaussian process on T with K as covariance kernel(Kolmogorov extension theorem).
6 / 29
Preliminaries on Gaussian processes
Let T be a set and CT the cylindrical σ-algebra on RT . Any stochasticprocess X on T is a (CT ,F)-measurable mapping
X : (Ω,F ,P)→ (RT , CT ).
The process X denes a probability measure on (RT , CT )
µX = P X−1.
The dual space(RT)∗
= span(δt , t ∈ T ) and
∀µ ∈(RT), µX µ−1
is a Gaussian measure on (R,B(R)).
This property will be fundamental in the rest of this talk.
7 / 29
Gaussian measures on Banach spaces
From now on:
(E , ‖.‖E ) will always be a real (separable) Banach space,
B(E ) its Borel σ-algebra.
Gaussian measures on Banach spaces
A probability measure γ on (E ,B(E )) is (centered) Gaussian if and onlyif γ f −1 is a (centered) Gaussian measure on (R,B(R)) for all f ∈ E∗.
We will always consider centered Gaussian measure. The characteristicfunctional is well dened and
∀f ∈ E∗, γ(f ) :=
∫E
e i〈x,f 〉E,E∗γ(dx) = e−Cγ (f ,f )
2 ∈ R
with Cγ(f , g) =∫E〈x , f 〉E ,E∗〈x , g〉E ,E∗γ(dx) (covariance).
8 / 29
Gaussian measures on Banach spaces
The covariance is bilinear, symmetric, positive and continuous.
Covariance operator
Let γ be a Gaussian measure on (E ,B(E )) then
Rγ : f ∈ E∗ →∫E
〈x , f 〉E ,E∗xγ(dx) ∈ E
is the covariance operator associated to γ.
It is characterized by the following relation
∀(f , g) ∈ E∗ × E∗, 〈Rγ f , g〉E ,E∗ = Cγ(f , g).
Moreover, it is a positive symmetric kernel: ∀(f , g) ∈ E∗ × E∗
〈Rγ f , g〉E ,E∗ = 〈Rγg , f 〉E ,E∗ ,
〈Rγ f , f 〉E ,E∗ ≥ 0
and Rγ is a compact (nuclear) operator.
9 / 29
Gaussian measures on Banach spaces
Similarly to Gaussian processes, dene on Rγ(E∗)
〈h, g〉γ = 〈Rγ h, g〉E ,E∗
with Rγ h = h and Rγ g = g . The Cameron-Martin space is
H(γ) := Rγ f , f ∈ E∗ ⊂ E
and we have the reproducing property
∀h ∈ H(γ), ∀f ∈ E∗, 〈h, f 〉E ,E∗ = 〈h,Rγ f 〉γ .
(H(γ), < ., . >γ) is a Hilbertian subspace of E ,
Rγ(X ∗) is dense in H(γ),
Rγ may not be injective.
10 / 29
Gaussian measures on Banach spaces
As we just saw, Rγ : E∗ → E need not be injective.
However, Rγ f = 0 implies f = 0, γ-almost everywhere.
Gaussian space
Let γ be a Gaussian measure on (E ,B(E )) and
i : f ∈ E∗ → f ∈ L2(γ),
the Gaussian space is
E∗γ = i(E∗)L2(γ)
.
For all Gaussian measures, Gaussian and Cameron-Martin spaces areisometric
E∗γ ≡ H(γ).
11 / 29
Karhunen-loève decomposition on Hilbert spaces
Consider now:
(H, 〈., .〉H) a (separable) Hilbert space,
BH∗ is weakly compact,
H = H∗ (Riesz representation theorem),
γ Gaussian measure on (H,B(H)).
Continuity
h ∈ H → 〈Rγh, h〉H is weakly sequentially continuous.
As consequence, consider h0 ∈ BH such that:
sup‖h‖H≤1
〈Rγh, h〉H = 〈Rγh0, h0〉H = λ0,
then h0 is an eigenvector of Rγ associated to the eigenvalue λ0.
12 / 29
Karhunen-loève decomposition on Hilbert spaces
The previous h0 may be used to split H in two orthogonal subspaces
H = Rh0 ⊕⊥ H1.
From this, dene:
P0 : h ∈ H → 〈h, h0〉Hh0,γλ0 = γ P−1
0,
γ1 = γ (I − P0)−1.
Orthogonal split of the Gaussian measure
The two measures γλ0 and γ1 on (H,B(H)) are Gaussian and
γ = γλ0 ∗ γ1,Rγλ0 = P0RγP0 = λ0〈h0, .〉Hh0,Rγ1 = (I − P0)Rγ(I − P0) = Rγ − Rγλ0
with Rh0 = H(γλ0), H1 = (Rh0)⊥ both equipped with 〈., .〉γ .
13 / 29
Karhunen-loève decomposition on Hilbert spaces
Next, apply the splitting method to the (conditional) Gaussian measureγ1 and so on and so forth...
Properties
(hn) is orthonormal in H,
(hn) is dense in H(γ),
Rγ =∑
n≥0 λn〈., hn〉Hhn with∑
n≥0 λn <∞.
In particular, Rγ is compact (nuclear) and
h =∑n≥0
〈h, hn〉Hhn, γ a.e.
The standard theory uses the compacity of Rγ as an input and thespectral theorem. Moreover, the class of all Gaussian covariances is equalto symmetric, positive and nuclear operators.
14 / 29
Karhunen-loève decomposition on Banach spaces
The following result is already known (Theorem 3.5.1 p.112 in XX) and iscalled "Karhunen-loève" decomposition in some texts:
Decomposition theorem
Let γ be a Gaussian measure on a locally convex space E , H(γ) theCameron-Martin space, (en) an orthonormal basis, (ξn)n a sequence ofindependent standard Gaussian random variables on a probability space(Ω,F ,P) then ∑
n≥0
ξn(ω)en (1)
converges almost surely in E , and has distribution γ.
We will now construct a "particular" basis in Rγ(E∗) which correspondsto decomposition of the covariance operator.
15 / 29
Karhunen-loève decomposition on Banach spaces
Consider now the general case:
(E , ‖.‖E ) is a Banach space,
BE∗ is weakly compact (Banach alaoglu theorem),
γ Gaussian measure on (E ,B(E )).
Variance continuity
f ∈ E∗ → 〈Rγ f , f 〉E ,E∗ is weakly sequentially continuous.
∃f0 ∈ BE∗ , supf∈BX∗
〈Rγ f , f 〉E ,E∗ = 〈Rγ f0, f0〉E ,E∗ = λ0.
Now, write Rγ f0 = λ0x0 ∈ E .
Spectral theory is not dened in this context.
16 / 29
Karhunen-loève decomposition on Banach spaces
(f0, x0) will be useful to split "orthogonally" the Banach space
E = Rx0 + X1,
and similarly, dene
P0 : x ∈ E → 〈x , f0〉E ,E∗x0,
γλ0 = γ P−10
,
γ1 = γ (I − P0)−1.
Orthogonal split of the Gaussian measure
The two measures γλ0 and γ1 on (E ,B(E )) are Gaussian and
γ = γλ0 ∗ γ1,Rγλ0 = P0RγP
∗0
= λ0〈h0, .〉Hh0,Rγ1 = (I − P0)Rγ(I − P0)∗ = Rγ − Rγλ0
with Rh0 = H(γλ0), H1 = (Rh0)⊥ both equipped with 〈., .〉γ .
17 / 29
Karhunen-loève decomposition on Banach spaces
Again, repeat the splitting method on γ1.
Properties
‖xn‖E = ‖fn‖E∗ = 1,
(xn) is dense in H(γ),
Rγ =∑
n≥0 λn〈., fn〉E ,E∗xn.
Again, we have:
x =∑n≥0
〈x , f ∗n 〉E ,E∗xn, γ a.e.
andRγ =
∑n≥0
λn〈xn, .〉E ,E∗xn.
The class of Gaussian covariance operators is still an open problem.
18 / 29
Consequences
So far, we have the following remarks:
Approximation of covariance operator
In the previous decomposition, we have:∥∥∥∥∥Rγ −n∑
k=0
λk〈xk , .〉E ,E∗xk
∥∥∥∥∥ = λn+1
where the norm is in L(X ∗,X ).
Maximum variance
supf∈BE∗
〈Rγn f , f 〉E ,E∗ = supf∈BE∗
Var(〈PnX , f 〉E ,E∗) = λn
∑n≥0 λn may not be nite,
The decomposition may not be unique.
19 / 29
Specify a Gaussian measure in practice
Given a function space E , to specify a Gaussian measure on a functionspace:
choose a covariance kernel K such that:
P [(t → Xt) ∈ E ] = 1
choose directly a Covariance operator (ex: inverse of ellipticdierential operator)
Sample path from stochastic processes have properties linked to theKernel:
Continuity (Kolmogorov continuity theorem),
Integrability,
Dierentiability...
20 / 29
Example on the Wiener measure
Now, consider the standard Brownian motion with kernel K (s, t) = s ∧ t.On H = L2([0, 1],R), the decomposition is:
hn : t →√2 sin
((n +
π
2
)t),
λn =1(
n + π2
)2On E = C([0, 1],R), it becomes:
h′n : t →√2p(1 2k
2p+1 ,2k+12p+1 − 1 2k+1
2p+1 ,2k+22p+1
),
xn : t →√2p+2h′n(t),
λn =1
2p+2, n = 2p + k , k = 0, ..., 2p − 1, p ≥ 0
which represent "hat functions" on dyadic intervals.
21 / 29
Example with dierent kernels
0.0 0.2 0.4 0.6 0.8 1.0−0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2Basis functions
x0x1x2x3x4x5x6x7x8x9x10x11
Figure: Wiener measure basis functions.
22 / 29
Example with dierent kernels
0 2 4 6 8 1010-2
10-1
100
101 Variances
Variances
Figure: Wiener measure basis variances.
23 / 29
Example with dierent kernels
0.0 0.2 0.4 0.6 0.8 1.0−1.0
−0.5
0.0
0.5
1.0
1.5Basis functions
x0x1x2x3x4x5x6x7x8
Figure: Gaussian kernel basis functions.
24 / 29
Example with dierent kernels
0 1 2 3 4 5 6 7 810-13
10-12
10-11
10-10
10-9
10-8
10-7
10-6
10-5
10-4
10-3
10-2
10-1
100
101 Variances
Variances
Figure: Gaussian kernel basis variances.
25 / 29
Example with dierent kernels
0.0 0.2 0.4 0.6 0.8 1.0
−0.5
0.0
0.5
1.0
Basis functions
x0x1x2x3x4x5x6x7x8x9x10x11
Figure: Matern 3/2 kernel basis functions.
26 / 29
Example with dierent kernels
0 2 4 6 8 1010-4
10-3
10-2
10-1
100
101 Variances
Variances
Figure: Matern 3/2 kernel basis variances.
27 / 29
Brownian motion simulation
0.0 0.2 0.4 0.6 0.8 1.0−2.5
−2.0
−1.5
−1.0
−0.5
0.0
0.5Brownian motion simulation
Sin basis with n=1e4Sin basis with n=1e2Schauder basis with n=64
Figure: Simulation of brownian motion using dierent bases.
28 / 29
Conclusion
A new method to decompose Gaussian measures on Banach spaces.
The method can be applied to non-Gaussian square integrablemeasures,
Gaussian covariance characterization is still an open problem,
Links to optimal design in Kriging,
Links with Bayesian inverse problems,
Precise quantication of uncertainty.
29 / 29