Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

Hierarchical matrix approximation of

large covariance matricesA. Litvinenko1, M. Genton2, Ying Sun2, R. Tempone

1SRI-UQ Center and 2Spatio-Temporal Statistics & Data Analysis Groupat KAUST

alexander.litvinenko@kaust.edu.sa

Center for UncertaintyQuantification

Center for UncertaintyQuantification

Center for Uncertainty Quantification Logo Lock-up

Abstract

We approximate large non-structured covariance ma-trices in the H-matrix format with a log-linear com-putational cost and storage O(n log n). We computeinverse, Cholesky decomposition and determinant inH-format. As an example we consider the class ofMatern covariance functions, which are very popu-lar in spatial statistics, geostatistics, machine learningand image analysis. Applications are: kriging and op-timal design

1. Matern covariance

C(x, y) = C(|x−y|) = σ2 1

Γ(ν)2ν−1

(√2νr

L

)νKν

(√2νr

L

),

where Γ is the gamma function, Kν is the modifiedBessel function of the second kind, r = |x − y| andν, L are non-negative parameters of the covariance.

−2 −1.5 −1 −0.5 0 0.5 1 1.5 20

0.05

0.1

0.15

0.2

0.25

Matern covariance (nu=1)

σ=0.5, l=0.5

σ=0.5, l=0.3

σ=0.5, l=0.2

σ=0.5, l=0.1

−2 −1.5 −1 −0.5 0 0.5 1 1.5 20

0.05

0.1

0.15

0.2

0.25

nu=0.15

nu=0.3

nu=0.5

nu=1

nu=2

nu=30

As ν →∞ [4],

C(r) = σ2 exp(−r2/2L2).

When ν = 0.5, the Matern covariance is identical tothe exponential covariance function.

Cν=3/2(r) =

(1 +

√3r

L

)exp

(−√

3r

L

)

Cν=5/2(r) =

(1 +

√5r

L+

5r2

3L2

)exp

(−√

5r

L

).

Note: no need to assume neither C(x, y) = C(|x− y|) nor tensorgrid.

2. H-matrix approximation

25 20

20 20

20 16

20 16

20 20

16 16

20 16

16 16

19 20

20 19 32

19 19

16 16 32

19 20

20 19

19 16

19 16

32 32

20 20

20 20 32

32 32

20 19

19 19 32

20 19

16 16 32

32 20

32 32

20 32

32 32

32 20

32 32

20 19

19 19

20 16

19 16

32 32

20 32

32 32

32 32

20 32

32 32

20 32

20 20

20 20 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

20 2020 19

20 20 32

32 32 20

20 20

32 20

32 32 20

20 20

32 32

20 32 20

20 20

32 32

32 32 20

20

20 20

19 20 32

32 32

20 20

2032 20

32 32

20 20

2032 32

20 32

20 20

2032 32

32 32

20 20

20 20

20 20 32

32 32

20 19

20 19 32

32 32

20 20

19 19 32

32 32

20 20

20 20 32

32 32

32 20

32 32

32 20

32 32

32 20

32 32

32 20

32 32

32 32

20 32

32 32

20 32

32 32

20 32

32 32

20 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

20 20

20 20 2019 20

20 20 32

32 32

32 20

32 32

20 32

32 32

32 20

32 32

20 20

20 20

20 20

20 20 20

20 20

32 20

32 32 20

20 20

20 20

20 20

20 20 20

2032 20

32 32

20 20

20 20

20 20

20 20

20 20 2032 20

32 32

32 20

32 32

32 20

32 32

32 20

32 32

20 20

20 20

20 20

20 20

19 20

20 20 32

32 32

20 32

32 32

32 32

20 32

32 32

20 32

2020 20

20 20

20 20

20 20

20 20

32 32

20 32 20

2020 20

20 20

20 20

20 20

2032 32

20 32

20 20

2020 20

20 20

20 20

20 20

32 32

20 32

32 32

20 32

32 32

20 32

32 32

20 32

2020 20

20 20

20 20

20 20 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

19 19

20 20 32

32 32

32 20

32 32

20 32

32 32

32 20

32 32

19 20

19 20 32

32 32

20 32

32 32

32 32

20 32

32 32

20 32

20 20

20 20 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

20 20

32 32

32 32 20

20 20

32 20

32 32 20

20 20

32 32

20 32 20

20 20

32 32

32 32 20

2032 32

32 32

20 20

2032 20

32 32

20 20

2032 32

20 32

20 20

2032 32

32 32

20 20

32 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

32 20

32 32

32 20

32 20

32 20

32 32

32 20

32 20

32 32

20 32

32 32

20 32

32 32

20 20

32 32

20 20

32 32

32 32

32 32

32 32

32 32

32 32

32 32

32 32

25 9

9 20 9

920 7

7 169

9

20 9

9 20 9

9 329

9

20 9

9 20 9

9 32 9

932 9

9 32

9

9

20 9

9 20 9

9 32 9

9

20 9

9 20 9

9 329

9

32 9

9 32 9

932 9

9 32

9

9

20 9

9 20 9

9 32 9

932 9

9 329

9

20 9

9 20 9

9 32 9

932 9

9 32

9

9

32 9

9 32 9

932 9

9 329

9

32 9

9 32 9

932 9

9 32

Figure 2: Two approximation strategies [1]: fixed rank (left) andflexible rank (right) approximations, C ∈ Rn×n, n = 652.

I

I

I I

I

I

I I I I1

1

2

2

11 12 21 22

I11

I12

I21

I22

QQ t

S

dist

H=

t

s

1. Build cluster tree TI, I = {1, 2, ..., n}2. Build block cluster tree TI×I3. For each (t× s) ∈ TI×I, t, s ∈ TI, check admissibility

condition min{diam(Qt), diam(Qs)} ≤ η ·dist(Qt, Qs).

if(adm=true) then M |t×s is a rank-k matrix blockif(adm=false) then divide M |t×s further or define as adense matrix block, if small enough.

Grid → cluster tree (TI) + admissibility condition →block cluster tree (TI×I)→H-matrix→H-matrix arith-metics.

Operation Sequential Complexity Parallel Complexity(Hackbusch et al. ’99-’06) (Kriemann ’05)

storage(M) N = O(kn log n) NP

Mx N = O(kn log n) NP

M1 ⊕M2 N = O(k2n log n) NP

M1 �M2, M−1 N = O(k2n log2 n) NP +O(n)

H-LU N = O(k2n log2 n) NP +O(k

2n log2 nn1/d

)

Table 1: Computational cost of H-matrix arithmetics, sequentialand parallel.

Let ε =‖(C−CH)z‖2‖C‖2‖z‖2 , where z is a random vector.

n rank k size, MB t, sec. ε maxi=1..10

|λi − λi|, i ε2

for C C C C C4.0 · 103 10 48 3 0.8 0.08 7 · 10−3 7.0 · 10−2, 9 2.0 · 10−4

1.05 · 104 18 439 19 7.0 0.4 7 · 10−4 5.5 · 10−2, 2 1.0 · 10−4

2.1 · 104 25 2054 64 45.0 1.4 1 · 10−5 5.0 · 10−2, 9 4.4 · 10−6

Table 2: Accuracy of the H-matrix approx. exp. covariance function, l1 = l3 =0.1, l2 = 0.5.

l1 l2 ε

0.01 0.02 3 · 10−2

0.1 0.2 8 · 10−3

0.5 1 2.8 · 10−5

Table 3: Dependence of the H-matrix accuracy on the covari-ance lengths l1 and l2, n = 1292. The smaller cov. length the lessaccurate is H-approximation.

0

100

200

300

0

50

100

150

200

250

300

−1

0

1

2

−1

−0.5

0

0.5

1

1.5

2

2.5

3

0

100

200

300

0

50

100

150

200

250

300

−1

0

1

2

−3

−2

−1

0

1

2

Figure 4: Two realizations of random field generated viaCholesky decomposition of Matern covariance matrix, ν = 0.4.

3. Kullback-Leibler divergence

Measure of the information lost when distribution Q is used toapproximate P .

DKL(P‖Q) =∑i

P (i) lnP (i)

Q(i), DKL(P‖Q) =

∫ ∞−∞

p(x) lnp(x)

q(x)dx,

where p, q densities of P and Q. For miltivariate normal distribu-tions (µ0,C) and (µ1,C

H):

2DKL(N0‖N1) =

(tr((CH)−1C) + (µ1 − µ0)T (CH)−1(µ1 − µ0)− n− ln

(detC

detCH

)).

0 10 20 30 40 50 60 70 80 90 100−16

−14

−12

−10

−8

−6

−4

−2

0

rank k

log(r

el.

err

or)

Spectral norm, L=0.1, nu=0.5

Frob. norm, L=0.1

Spectral norm, L=0.2

Frob. norm, L=0.2

Spectral norm, L=0.5

Frob. norm, L=0.5

0 10 20 30 40 50 60 70 80 90 100−18

−16

−14

−12

−10

−8

−6

−4

−2

0

rank k

log(r

el.

err

or)

Spectral norm, L=0.1, ν=1.5

Frob. norm, L=0.1

Spectral norm, L=0.2

Frob. norm, L=0.2

Spectral norm, L=0.5

Frob. norm, L=0.5

Figure 5: RelativeH-matrix approx. error ‖C−CH‖2 for differentcov. lengths L = {0.1, 0.2, 0.5} and ν = {0.5, 1.5}

k KLD(C,CH) ‖C −CH‖2 ‖C(CH)−1 − I‖2L = 0.25 L = 0.75 L = 0.25 L = 0.75 L = 0.25 L = 0.75

5 0.51 2.3 4.0e-2 0.1 4.8 636 0.34 1.6 9.4e-3 0.02 3.4 228 5.3e-2 0.4 1.9e-3 0.003 1.2 810 2.6e-3 0.2 7.7e-4 7.0e-4 6.0e-2 3.112 5.0e-4 2e-2 9.7e-5 5.6e-5 1.6e-2 0.515 1.0e-5 9e-4 2.0e-5 1.1e-5 8.0e-4 0.0220 4.5e-7 4.8e-5 6.5e-7 2.8e-7 2.1e-5 1.2e-350 3.4e-13 5e-12 2.0e-13 2.4e-13 4e-11 2.7e-9

Table 4: Dependence of KLD on H-matrix rank k, Matern co-variance with L = {0.25, 0.75} and ν = 0.5, domain G = [0, 1]2,‖C(L=0.25,0.75)‖2 = {212, 568}.

For ν = 1.5 the KLD and the inverse (CH)−1 is hard to computenumerically. Results in Table 4 are better since covariance ma-trix with ν = 0.5 has smallest eigenvalues far enough from zero.The case ν = 1.5 is more smooth, the eigenvalues decay faster,but the smallest eigenvalues come much closer to zero than inν = 0.5 case.

4. Other applications

4.1 Low-rank approximation of Kriging and geo-statistical optimal designLet s ∈ Rn to be estimated, Css covariance matrix, y ∈ Rm isvector of measurements. The corresponding cross- and auto-covariance matrices are denoted by Csy and Cyy, respectively,sized n×m and m×m.

Kriging estimate s = CsyC−1yy y .

The estimation variance σ is the diagonal of the cond. cov. ma-trix Css|y: σs = diag(Css|y) = diag

(Css −CsyC

−1yyCys

)Geostatistical optimal design:

φA = n−1 trace[Css|y

]φC = cT

(Css −CsyC

−1yyCys

)c, c− a vector.

4.2 Weather forecast in Europa

180 24030

60

Figure 6: Europa weather stations (≈ 2500). Collected data setM ∈ R2500×365

.

0 50 100 150 200 250 300 350 400−20

−15

−10

−5

0

5

10

15

20

Figure 7: Truth temperature forecast and its low-rank approxi-mation (rank 50 approximation of matrix M ) in one station, rel.error=25%.

5. Open question

1. Compute the whole spectrum of large covariance matrix

2. Compute KLD for large matrices (det Σ ?)

3. How sensible is KLD to H-matrix accuracy ?

4. Derive/estimate KLD for non-Gaussian distributions.

Acknowledgements

A. Litvinenko is a member of the KAUST SRI UQ Center.

References

1. B. N. Khoromskij, A. Litvinenko, H. G. Matthies, Application ofhierarchical matrices for computing the Karhunen?Loeve expan-sion, Computing, Vol. 84, Issue 1-2, pp 49-67, 20082. R. Furrer, M. Genton, D. Nychka, Covariance tapering for in-terpolation of large spatial datasets, J. Comp. & Graph. Stat.,Vol.15, N3, pp502-523.3. M. Stein, Limitations on low rank approximations for covari-ance matrices of spatial data, Spat. Statistics, 20134. J. Castrillion-Candis, M. Genton, R. Yokota, Multi-Level Re-stricted Maximum Likelihood Cov. Estim. and Kriging for LargeNon-Gridded Datasets, 2014.