math 590: meshfree methods - chapter 3: positive definite functions

MATH 590: Meshfree MethodsChapter 3: Positive Definite Functions

Greg Fasshauer

Department of Applied MathematicsIllinois Institute of Technology

Fall 2010

[email protected] MATH 590 – Chapter 3 1

http://math.iit.edu/~fass

Outline

1 Positive Definite Matrices and Functions

2 Integral Characterizations for (Strictly) Positive Definite Functions

3 Positive Definite Radial Functions



We know that the solution of the scattered data interpolationproblem with RBFs and/or kernels amounts to solving a linearsystem

Ac = y ,

where Ajk = ϕ(‖x j − xk‖2), j , k = 1, . . . ,N.Linear algebra tells us that this system will have a unique solutionwhenever A is non-singular.Necessary and sufficient conditions to characterize this generalnon-singular case are still open.We will focus on basic functions ϕ that generate positive definitematrices.



Positive Definite Matrices and Functions

Definition

A real symmetric matrix A is called positive semi-definite if itsassociated quadratic form is non-negative, i.e.,

N∑j=1

N∑k=1

cjckAjk ≥ 0 (1)

for c = [c1, . . . , cN ]T ∈ RN .If the quadratic form (1) is zero only for c ≡ 0, then A is called positivedefinite.

Since the eigenvalues of a positive definite matrix are all positive apositive definite matrix is non-singular.We therefore look for basis functions Bk that generate a positivedefinite interpolation matrix.This leads to positive definite functions.




Definition

A complex-valued continuous function Φ : Rs → C is called positivedefinite on Rs if

N∑j=1

N∑k=1

cjck Φ(x j − xk ) ≥ 0 (2)

for any N pairwise different points x1, . . . ,xN ∈ Rs, andc = [c1, . . . , cN ]T ∈ CN .The function Φ is called strictly positive definite on Rs if the quadraticform (2) is zero only for c ≡ 0.

RemarkFor theoretical reasons the definition employs complex-valuedfunctions.All of our applications will only be real-valued.Because of the inequality in (2) we know that any (strictly) positivedefinite function must have a real-valued quadratic form.




RemarkHistory (i.e., analysts in the 1920s and 30s) defined positivedefinite functions in analogy to positive semi-definite matrices.This concept is not strong enough to guarantee a non-singularinterpolation matrix and so we need to define strictly positivedefinite functions as in [Micchelli (1986)].This leads to an unfortunate difference in terminology used in thecontext of matrices and functions.Sometimes the literature is confusing about this and you mightfind papers that use the term positive definite function in the strictsense, and others that use it in the way we defined it above.Hopefully, the authors are clear about their use of terminology.




Example

The functionΦB(x) = eix ·y , y ∈ Rs fixed,

is positive definite on Rs since its quadratic form is

N∑j=1

N∑k=1

cjck ΦB(x j − xk ) =N∑

j=1

N∑k=1

cjckei(x j−xk )·y

=N∑

j=1

cjeix j ·yN∑

k=1

cke−ixk ·y

=

∣∣∣∣∣∣N∑

j=1

cjeix j ·y

∣∣∣∣∣∣2

≥ 0.




Definition 2 suggests that we should use strictly positive definitefunctions as basis functions for our scattered data interpolationproblem, i.e.,

Pf (x) =N∑

k=1

ck Φ(x − xk ), x ∈ Rs. (3)

RemarkAt this point we don’t require Φ to be radial.In fact, the interpolant obtained via Pf of (3) is translation invariant.We will later specialize to strictly positive definite functions that arealso radial on Rs.One can also generalize this setup by introducing (strictly) positivedefinite kernels (a bit more later). Then we may even give uptranslation invariance.




Theorem (Properties of (strictly) positive definite functions)

(1) Non-negative finite linear combinations of positive definite functions are positivedefinite. If Φ1, . . . ,Φn are positive definite on Rs and cj ≥ 0, j = 1, . . . , n, then

Φ(x) =n∑

j=1

cj Φj (x), x ∈ Rs,

is also positive definite. Moreover, if at least one of the Φj is strictly positivedefinite and the corresponding cj > 0, then Φ is strictly positive definite.

(2) Φ(0) ≥ 0.

(3) Φ(−x) = Φ(x).

(4) Any positive definite function is bounded. In fact,

|Φ(x)| ≤ Φ(0).

(5) If Φ is positive definite with Φ(0) = 0 then Φ ≡ 0.

(6) The product of (strictly) positive definite functions is (strictly) positive definite.




Proof.Properties (1) and (2) follow immediately from Definition 2.

Property (5) follows immediately from (4).

Property (6) follows from Schur’s theorem in linear algebra whichensures that the elementwise (or Hadamard) product of positive(semi-)definite matrices is positive (semi-)definite.

We will now give details for Properties (3) and (4).

For more details see [Cheney and Light (1999)] or[Wendland (2005a)].




Proof (cont.)To show (3) we let N = 2, x1 = 0, x2 = x , and choose c1 = 1 andc2 = c.Then we have the quadratic form

2∑j=1

2∑k=1

cjck Φ(x j − xk ) = (1 + |c|2)Φ(0) + cΦ(x) + cΦ(−x) ≥ 0

for every c ∈ C.Now we can take c = 1 and c = i.These, respectively, imply that both

Φ(x) + Φ(−x) and i (Φ(x)− Φ(−x))

must be real.This, however, is only possible if Φ(−x) = Φ(x).




Proof (cont.)For the proof of (4) we let N = 2, x1 = 0, x2 = x , and choosec1 = |Φ(x)| and c2 = −Φ(x).Then we get the quadratic form

2∑j=1

2∑k=1

cjck Φ(x j − xk ) = 2Φ(0)|Φ(x)|2 − Φ(−x)Φ(x)|Φ(x)| − Φ2(x)|Φ(x)|

≥ 0.

Since Φ(−x) = Φ(x) by Property (3), this gives

2Φ(0)|Φ(x)|2 − 2|Φ(x)|3 ≥ 0.

If |Φ(x)| > 0, we divide by |Φ(x)|2 and get |Φ(x)| ≤ Φ(0) as claimed.If |Φ(x)| ≡ 0, then the statement holds trivially.




Example

The cosine function is positive definite on R.

First we remember that for any x ∈ R

cos x =12

(eix + e−ix

).

By our earlier example the function ΦB(x) = eix ·y is positive definite forany fixed y ∈ Rs.Therefore, Φ1(x) = eix and Φ2(x) = e−ix are positive definite, and thecosine function is positive definite by Property (1).




Property (3) shows that any real-valued (strictly) positive definitefunction has to be even.We even have

Theorem

A real-valued continuous function Φ is positive definite on Rs if andonly if it is even and

N∑j=1

N∑k=1

cjck Φ(x j − xk ) ≥ 0 (4)

for any N pairwise different points x1, . . . ,xN ∈ Rs, andc = [c1, . . . , cN ]T ∈ RN .The function Φ is strictly positive definite on Rs if the quadratic form (4)is zero only for c ≡ 0.

See [Wendland (2005a)] for a proof.



Integral Characterizations for (Strictly) Positive Definite Functions

We summarize some facts about integral characterizations of positivedefinite functions.

They were established in the 1930s by Bochner and Schoenberg.

We will also mention more recent extensions to strictly positive definitefunctions that are needed for the scattered data interpolation problem.

Many more details in [Wendland (2005a)].

We start with a very brief discussion of concepts from measure theoryand integral transforms that appear in the results later on.



Integral Characterizations for (Strictly) Positive Definite Functions Some Important Concepts from Measure Theory

Bochner’s theorem below is formulated in terms of Borel measures.

DefinitionLet X be an arbitrary set and denote by P(x) the set of all subsets ofX . A subset A of P(X ) is called a σ-algebra on X if(1) X ∈ A,(2) A ∈ A implies that its complement (in X ) is also contained in A,(3) Ai ∈ A, i ∈ N, implies that the union of these sets belongs to A.

DefinitionGiven an arbitrary set X and a σ-algebra A of subsets of X , a measureon A is a function µ : A → [0,∞] such that(1) µ(∅) = 0,(2) for any sequence Ai of disjoint sets in A we have

µ(∞⋃

i=1

Ai) =∞∑

i=1

µ(Ai).



Integral Characterizations for (Strictly) Positive Definite Functions Some Important Concepts from Measure Theory

DefinitionIf X is a topological space, and O is the collection of open sets in X ,then the σ-algebra generated by O is called the Borel σ-algebra anddenoted by B(X ).

If in addition X is a Hausdorff spacea, then a measure µ defined onB(X ) that satisfies µ(K ) <∞ for all compact sets K ⊆ X is called aBorel measure.

The carrier of a Borel measure is given by the set

X \ O : O ∈ O and µ(O) = 0.aX is a Hausdorff space if any two distinct points of X can be separated by

open sets



Integral Characterizations for (Strictly) Positive Definite Functions A few important integral transforms

Definition

The Fourier transform of f ∈ L1(Rs) is given by

f (ω) =1√

(2π)s

∫Rs

f (x)e−iω·xdx , ω ∈ Rs,

and its inverse Fourier transform is given by

f (x) =1√

(2π)s

∫Rs

f (ω)eix ·ωdω, x ∈ Rs.

This definition from [Rudin (1973)] is based on angular frequency andis used in [Wendland (2005a), Schölkopf and Smola (2002)].Another, just as common, definition from [Stein and Weiss (1971)]uses ordinary frequency, i.e.,

fSW (ω) =

∫Rs

f (x)e−2πiω·xdx =√

(2π)s f (2πω).

It appears in [Buhmann (2003), Cheney and Light (1999)][email protected] MATH 590 – Chapter 3 20



Similarly, we can define the Fourier transform of a finite (signed)measure µ on Rs by

µ(ω) =1√

(2π)s

∫Rs

e−iω·xdµ(x), ω ∈ Rs.




Since we are mostly interested in positive definite radial functions, wenote that the Fourier transform of a radial function is again radial:

Theorem

Let Φ ∈ L1(Rs) be continuous and radial, i.e., Φ(x) = ϕ(‖x‖). Then itsFourier transform Φ is also radial, i.e., Φ(ω) = Fsϕ(‖ω‖) with

Fsϕ(r) =1√r s−2

∫ ∞0

ϕ(t)ts2 J s−2

2(rt)dt ,

where J s−22

is the classical Bessel function of the first kind of order s−22 .

Proof in [Wendland (2005a)]. This transform is also calledFourier-Bessel transform or Hankel transform.




RemarkThe Hankel inversion theorem [Sneddon (1972)] ensures that theFourier transform for radial functions is its own inverse, i.e., for radialfunctions ϕ we have

Fs [Fsϕ] = ϕ.



Integral Characterizations for (Strictly) Positive Definite Functions Bochner’s Theorem

One of the most celebrated results on positive definite functions is theircharacterization in terms of Fourier transforms established by Bochnerin 1932 (for s = 1) and 1933 (for general s).

Theorem (Bochner)

A (complex-valued) function Φ ∈ C(Rs) is positive definite on Rs if andonly if it is the Fourier transform of a finite non-negative Borel measureµ on Rs, i.e.,

Φ(x) = µ(x) =1√

(2π)s

∫Rs

e−ix ·ydµ(y), x ∈ Rs.




Proof.There are many proofs of this theorem.

Bochner’s original proof can be found in [Bochner (1933)]. Otherproofs can be found, e.g., in [Cuppens (1975)] or[Gel’fand and Vilenkin (1964)].

A proof using the Riesz representation theorem to interpret the Borelmeasure as a distribution, and then taking advantage of distributionalFourier transforms can be found in [Wendland (2005a)].

We will prove only the one (easy) direction that is important for theapplication to scattered data interpolation.




Proof (cont.)We assume Φ is the Fourier transform of a finite non-negative Borelmeasure and show Φ is positive definite.Thus,

N∑j=1

N∑k=1

cjck Φ(x j − xk ) =1√

(2π)s

N∑j=1

N∑k=1

[cjck

∫Rs

e−i(x j−xk )·ydµ(y)

]

=1√

(2π)s

∫Rs

N∑j=1

cje−ix j ·yN∑

k=1

ckeixk ·y

dµ(y)

=1√

(2π)s

∫Rs

∣∣∣∣∣∣N∑

j=1

cje−ix j ·y

∣∣∣∣∣∣2

dµ(y) ≥ 0.

The last inequality holds because of the conditions imposed on themeasure µ.




RemarkBochner’s theorem shows that for any fixed y ∈ Rs the function

ΦB(x) = eix ·y , x ∈ Rs,

of our earlier example can be considered as the fundamental positivedefinite function since all other positive definite functions are obtainedas (infinite) linear combinations of this function.

While Property (1) of Theorem 4 implies that linear combinations of ΦBwill again be positive definite, the remarkable content of Bochner’sTheorem is the fact that indeed all positive definite functions aregenerated by ΦB.



Integral Characterizations for (Strictly) Positive Definite Functions Extensions to Strictly Positive Definite Functions

In order to guarantee a well-posed interpolation problem we have toextend (if possible) Bochner’s characterization to strictly positivedefinite functions.

We give a sufficient condition for a function to be strictly positivedefinite on Rs.

Theorem

Let µ be a non-negative finite Borel measure on Rs whose carrier is aset of nonzero Lebesgue measure. Then the Fourier transform of µ isstrictly positive definite on Rs.




Proof.As in the proof of Bochner’s theorem we have

N∑j=1

N∑k=1

cjck Φ(x j − xk ) =1√

(2π)s

N∑j=1

N∑k=1

[cjck

∫Rs

e−i(x j−xk )·ydµ(y)

]

=1√

(2π)s

∫Rs

N∑j=1

cje−ix j ·yN∑

k=1

ckeixk ·y

dµ(y)

=1√

(2π)s

∫Rs

∣∣∣∣∣∣N∑

j=1

cje−ix j ·y

∣∣∣∣∣∣2

dµ(y) ≥ 0.

Now let

g(y) =N∑

j=1

cje−ix j ·y ,

and assume that the points x j are all distinct and c 6= [email protected] MATH 590 – Chapter 3 29



Proof (cont.)Since the x j are distinct the functions

y 7→ e−ix j ·y

are linearly independent, and so g 6= 0.Since g is an entire function its zero set, i.e., y ∈ Rs : g(y) = 0 canhave no accumulation point and therefore it has Lebesgue measurezero (see, e.g., [Cheney and Light (1999)]).Now, the only remaining way to make the inequality

1√(2π)s

∫Rs

∣∣∣∣∣∣N∑

j=1

cje−ix j ·y

∣∣∣∣∣∣2

dµ(y) =1√

(2π)s

∫Rs|g(y)|2 dµ(y) ≥ 0

an equality is if the carrier of µ is contained in the zero set of g, i.e.,has Lebesgue measure zero.The hypothesis of the theorem does not allow this.




RemarkAnother sufficient condition is given in [zu Castell et al. (2005)].

A complete integral characterization of functions that are strictlypositive definite on Rs is to my knowledge still missing. The cases = 1 is covered in [Chang (1996)].

Complete characterizations of strictly positive definite functions oncompact spaces (such as spheres) do exist in the literature.




The following corollary gives us a way to construct strictly positivedefinite functions.

Corollary

Let f be a continuous non-negative function in L1(Rs) which is notidentically zero. Then the Fourier transform of f is strictly positivedefinite on Rs.

Proof.This is a special case of the previous theorem in which the measure µhas Lebesgue density f . Thus, we use the measure µ defined for anyBorel set B by

µ(B) =

∫B

f (x)dx .

Then the carrier of µ is equal to the (closed) support of f . However,since f is non-negative and not identically equal to zero, its supporthas positive Lebesgue measure, and hence the Fourier transform of fis strictly positive definite by the preceding theorem.




A very useful criterion to check whether a given function is strictlypositive definite is given by

Theorem

Let Φ be a continuous function in L1(Rs). Φ is strictly positive definite ifand only if Φ is bounded and its Fourier transform is non-negative andnot identically equal to zero.

Proof in [Wendland (2005a)].

If Φ 6≡ 0 (which implies that then also Φ 6≡ 0) we need to ensure onlythat Φ be non-negative in order for Φ to be strictly positive definite.



Positive Definite Radial Functions

We now focus on positive definite radial functions.

Earlier we characterized (strictly) positive definite functions in terms ofmultivariate functions Φ.

When working with radial functions Φ(x) = ϕ(‖x‖) it is convenient tocall the univariate function ϕ a positive definite radial function.

This is a small abuse of our terminology for positive definite functions,but commonly done in the literature.

If follows immediately that

Lemma

If Φ = ϕ(‖ · ‖) is (strictly) positive definite and radial on Rs then Φ isalso (strictly) positive definite and radial on Rσ for any σ ≤ s.




We now return to integral characterizations and begin with a theoremdue to Schoenberg (see, e.g., [Schoenberg (1938a), p.816], or[Wells and Williams (1975), p.27]).

Theorem

A continuous function ϕ : [0,∞)→ R is positive definite and radial onRs if and only if it is the Bessel transform of a finite non-negative Borelmeasure µ on [0,∞), i.e.,

ϕ(r) =

∫ ∞0

Ωs(rt)dµ(t).

Here

Ωs(r) =

cos r for s = 1,

Γ( s

2

) (2r

) s−22 J s−2

2(r) for s ≥ 2,

and J s−22

is the classical Bessel function of the first kind of order s−22 .




RemarkNow we can view the function Φ(x) = cos(x) as the fundamentalpositive definite radial function on R.

We will see in the next chapter that the characterization of theprevious theorem immediately suggests a class of (even strictly)positive definite radial functions.




A Fourier transform characterization of strictly positive definite radialfunctions on Rs can be found in [Wendland (2005a)]. This theorem isbased on the Fourier transform formula of radial functions given earlierand the check for strictly positive definite functions.

Theorem

A continuous function ϕ : [0,∞)→ R such thatr 7→ r s−1ϕ(r) ∈ L1[0,∞) is strictly positive definite and radial on Rs ifand only if the s-dimensional Fourier transform

Fsϕ(r) =1√r s−2

∫ ∞0

ϕ(t)ts2 J(s−2)/2(rt)dt

is non-negative and not identically equal to zero.




Above we saw that any function that is (strictly) positive definite andradial on Rs is also (strictly) positive definite and radial on Rσ for anyσ ≤ s.Therefore, we are interested in those functions which are (strictly)positive definite and radial on Rs for all s.The characterization for positive definite functions is from[Schoenberg (1938a), pp. 817–821] and the strictly positive definitecase from [Micchelli (1986)]:

Theorem (Schoenberg)

A continuous function ϕ : [0,∞)→ R is strictly positive definite andradial on Rs for all s if and only if it is of the form

ϕ(r) =

∫ ∞0

e−r2t2dµ(t),

where µ is a finite non-negative Borel measure on [0,∞) notconcentrated at the origin.




Letting µ be a point evaluation measure in Schoenberg’s theoremshows us that the Gaussian

ϕ(r) = e−ε2r2

is strictly positive definite and radial on Rs for all s.

Moreover, we can view the Gaussian as the fundamental member ofthe family of functions that are strictly positive definite and radial on Rs

for all s.

By Schoenberg’s theorem all such functions are given as infinite linearcombinations of Gaussians.




Schoenberg’s theorem employs a finite non-negative Borel measure µon [0,∞) such that

ϕ(r) =

∫ ∞0

e−r2t2dµ(t).

If we want to find a zero r0 of ϕ then we have to solve

ϕ(r0) =

∫ ∞0

e−r20 t2

dµ(t) = 0.

Since the exponential function is positive and the measure isnon-negative, it follows that µ must be the zero measure.However, then ϕ is identically equal to zero.Therefore, a non-trivial function ϕ that is positive definite and radial onRs for all s can have no zeros.




The discussion on the previous slide implies in particular that

Theorem

There are no oscillatory univariate continuous functions that arestrictly positive definite and radial on Rs for all s.There are no compactly supported univariate continuous functionsthat are strictly positive definite and radial on Rs for all s.



Appendix References

References I

Buhmann, M. D. (2003).Radial Basis Functions: Theory and Implementations.Cambridge University Press.

Cheney, E. W. and Light, W. A. (1999).A Course in Approximation Theory.Brooks/Cole (Pacific Grove, CA).

Cuppens, R. (1975).Decomposition of Multivariate Probability.Academic Press (New York).

Fasshauer, G. E. (2007).Meshfree Approximation Methods with MATLAB.World Scientific Publishers.

Gel’fand, I. M. and Vilenkin, N. Ya. (1964).Generalized Functions Vol. 4.Academic Press (New York).



Appendix References

References II

Iske, A. (2004).Multiresolution Methods in Scattered Data Modelling.Lecture Notes in Computational Science and Engineering 37, Springer Verlag(Berlin).

Rudin, W. (1973).Functional Analysis.McGraw-Hill (New York).

Schölkopf, B. and Smola, A. J. (2002).Learning with kernels: Support vector machines, regularization, optimization, andbeyond.MIT Press, Cambridge, Massachussetts.

Sneddon, I. H. (1972).The Use of Integral Transforms.McGraw-Hill (New York).



Appendix References

References III

Stein, E. M. and Weiss, G. (1971).Introduction to Fourier Analysis in Euclidean Spaces.Princeton University Press (Princeton).

Wells, J. H. and Williams, R. L. (1975).Embeddings and Extensions in Analysis.Springer (Berlin).

Wendland, H. (2005a).Scattered Data Approximation.Cambridge University Press (Cambridge).

Bochner, S. (1933).Monotone Funktionen, Stieltjes Integrale und harmonische Analyse.Math. Ann. 108, pp. 378–410.

zu Castell, W., Filbir, F. Szwarc, R. (2005).Strictly positive definite functions in Rd .J. Approx. Theory 137 2, pp. 277–280.



Appendix References

References IV

Chang, K. F. (1996).Strictly positive definite functions.J. Approx. Theory 87, pp. 148–158. Addendum: J. Approx. Theory 88 1997,p. 384.

Micchelli, C. A. (1986).Interpolation of scattered data: distance matrices and conditionally positivedefinite functions.Constr. Approx. 2, pp. 11–22.

Schoenberg, I. J. (1938a).Metric spaces and completely monotone functions.Ann. of Math. 39, pp. 811–841.



math 590: meshfree methods - chapter 3: positive definite functions

Documents