computational complexity and compressed sensingpotts/cms/cms10/slides/slideshans… ·...

Computational Complexity and CompressedSensing

Anders C. Hansen, University of Cambridge

Chemnitz, October 4, 2010

Anders C. Hansen, University of Cambridge Computational Complexity and Compressed Sensing

Three Talks

(i) Generalized Sampling and Infinite Systems of Equations

(ii) Compressed Sensing in Infinite Dimensions

(iii) Spectral Problems and the Solvability Complexity Index


Computations with the Infinite Matrix

Let

A =

!

"#a11 . . . a1n...

......

an1 . . . ann

$

%& , T =

!

"""#

a11 a12 a13 . . .a21 a22 a23 . . .a31 a32 a33 . . ....

......

. . .

$

%%%&

All questions that may be asked about A can be asked about T .For example: How to find the spectrum, how to find potentialeigenvectors, how to solve linear systems of equations etc.


Why?

! Computer Science is discrete.

! Physics is continuous.

Let H denote the Schrodinger operator and F the FourierTransform. In particular,

(Hf )(x) = !!f (x) + V (x)f (x), (Ff )(!) =

'f (x)e!2!i"x

! Compute the spectrum of H (Quantum Mechanics).

! Solve the inverse problem f = Fg (Magnetic ResonanceImaging).


Applications

! Mathematical Physics (Quantum Mechanics)

! Signal Processing

! Inverse Problems (MRI, Tomography)

! Compressed Sensing

! Computational Biology


Generalized Sampling and Infinite Systems ofEquations

Anders C. Hansen, University of Cambridge

Chemnitz, October 4, 2010

Anders C. Hansen, University of Cambridge Generalized Sampling and Infinite Systems of Equations

Motivation

Magnetic Resonance Imaging (MRI)Let F denote the Fourier Transform. In particular,

(Fg)(!) =

!

g(x)e!2!i"x

Letf = Fg .

We want to recover g (completely) from samples of f .


The Shannon Sampling Theorem

Suppose thatf = Fg , g ! L2(R),

and supp(g) " [#T ,T ] for some T > 0. If " $ 12T (the Nyquist

rate) then

f (t) =""

k=!"

f (k")sinc

#

t + k"

"

$

, L2 and unif. conv., (1)

g = """

k=!"

f (k")e2!i#k·, L2 convergence. (2)

In practice, one forms the approximations

fN =N

"

k=!N

f (k")sinc

#

t + k"

"

$

, gN = "N

"

k=!N

f (k")e2!i#k·.


Question 1:

Are the approximations

fN =N

"

k=!N

f (k")sinc

#

t + k"

"

$

, gN = "N

"

k=!N

f (k")e2!i#k·

optimal given the samples

{f (k")}Nk=!N .


Question 2:

Could there be L2 functions {#k}k#N and coe!cients {$k}k#N

such that the series

f ="

k#N

$kF#k , g ="

k#N

$k#k

converge faster than the series

f (t) =""

k=!"

f (k")sinc

#

t + k"

"

$

,

g = ""

"

k=!"

f (k")e2!i#k·.


Question 3:

There are therefore two important questions to ask:

(i) Can one obtain the coe!cients {$k}k#N (or at leastapproximations to them), based on the same samplinginformation {f (k")}k#N, and will this yield betterapproximations to f and g?

(ii) Can one subsample from {f ("k)}k#N (e.g. not sampling atthe Nyquist rate) and still get recovery of {$k}k#N and hencef and g?


Example:

−1 −0.5 0 0.5 1−1.5

−1

−0.5

0

0.5

1

1.5

−1 −0.5 0 0.5 1−1.5

−1

−0.5

0

0.5

1

1.5

Figure: The figure shows gN = "%N

k=!N f (k")e2!i"k· for N = 500 and" = 0.5 (left) as well as g (right).


The Abstract Sampling Idea

Let H be a separable Hilbert space and let f ! H be an elementwe would like to reconstruct.

! Given m ! N linearly independent sampling vectors{s1, . . . , sm} that span a subspace S " H, we can access thesampled inner products ck = %sk , f &, k = 1, . . . ,m.

! Given linearly independent reconstruction vectors{w1, . . . ,wm} that span a subspace W " H.

! Construct an approximation f ! W to f based on the samples{ck}m

k=1. Want to find coe!cients {dk}mk=1 (that are

computed from the samples {ck}mk=1) such that

f =%m

k=1 dkwk .


The State of the Art

The reconstruction (Unser and Aldrobi) (Eldar) is

f =m

"

k=1

dkwk = W (S$W )!1S$f , (3)

where the operators S ,W : Cm ' H are defined by

Sx = x1s1 + . . . + xmsm, Wy = y1w1 + . . . + ymwm, (4)

and their adjoints S$,W $ : H ' Cm are easily seen to be

S$g = {%s1, g&, . . . , %sm, g&}, W $h = {%w1, h&, . . . , %wm, h&}.


The State of the Art

From this it is clear that we can express S$W : Cm ' Cm as thematrix

&

'

(

%s1,w1& . . . %sm,w1&...

......

%s1,wM& . . . %sm,wm&

)

*

+. (5)

Also, S$W is invertible if and only if

W ( S% = {0}. (6)

Thus, to construct f one simply solves a linear system ofequations. The error can now conveniently be bounded from aboveand below by

)f # PW f ) $ )f # f ) $1

cos(%WS))f # PW f ),

where PW is the projection onto W,

cos(%WS) = inf{)PSg) : g ! W, )g) = 1}.


Questions

(i) What if W ( S% *= {0} so that S$W is not invertible?

(ii) What if )(S$W )!1) is large? The stability of the methodmust clearly depend on )(S$W )!1). Thus, even if (S$W )!1

exists, one may not be able to use the method in practice asthere will likely be increased sensitivity to both round-o" errorand noise.


Example

As for (i), the simplest example is to let H = l2(Z) and let {ej}j#Z

be the natural basis (ej is the infinite sequence with 1 in its j-thcoordinate and zeros elsewhere). For m ! N, let the samplingvectors {sk}m

k=!m and the reconstruction vectors {wk}mk=!m be

defined by sk = ek and wk = ek+1. Then, clearly,W ( S% = {em+1}.


Example I

For an example of more practical interest, consider the following:Let, for 0 < " $ 1, H = L2([0, 1/"]) and, for odd m ! N, definethe sampling vectors

{s#,k}(m!1)/2k=!(m!1)/2, s#,k = e!2!i#k·&[0,1/#],

(this is exactly the type of measurement vector that will be used ifone models Magnetic Resonance Imaging) and let thereconstruction vectors {wk}m

k=1 denote the m first Haar waveletson [0, 1] (including the constant function, w1 = &[0,1]).


Example II

20 40 60 80 100

2468101214

20 40 60 80 100

5

10

15

20

Figure: This figure shows log10 )(S"" W )!1) as a function of m and " for

m = 1, 2, . . . , 150. The left plot corresponds to " = 1, whereas the rightplot corresponds to " = 7/8 (circles), " = 1/2 (crosses) and " = 1/8(diamonds).


Example III

10 20 30 40 50

10

20

30

40

100 200 300 400 500

!2

!1

1

2

Figure: The left figure shows log10 )(S"W )!1) as a function of m form = 2, 4, . . . , 50 and " = 1, 1

2 , 78 , 1

8 (squares, circles, crosses and

diamonds respectively). The right figure shows log10 )f # f ) form = 4, 8, . . . , 500, where f (x) = 1

1+16x2 .


Connection to the Finite Section Method

Let {sk}k#N and {wk}k#N be two sequences of linearly independentelements in a Hilbert space H. Define the infinite matrix U by

U =

&

'

'

'

(

u11 u12 u13 . . .u21 u22 u23 . . .u31 u32 u33 . . ....

......

. . .

)

*

*

*

+

, uij = %sj ,wi &. (7)

Thus, by (5) the operator S$W is simply the m by m finite sectionof U. In particular

S$W = PmUPm|Pml2(N).

The finite section method has been studied extensively in the lastdecades (Bottcher, Hagen, Lindner, Roch, Silbermann).


Connection to the Finite Section Method II

It is well known that even if U is invertible we may have that

! PmUPm|Pml2(N) may never be invertible for any m.

! PmUPm|Pml2(N) is invertible for all m ! N, but

)(PmUPm|Pml2(N))!1) ' +, m ' +.

! There exist x , y such that

x = U!1y , x , y ! l2(N), xm = (PmUPm|Pml2(N))!1Pmy

butxm ! x , m ' +.


The New Approach

One would like to have a completely general sampling theory thatcan be described as follows:

(i) We have a signal f ! H and a Riez basis {wk}k#N that spanssome closed subspace W " H, and

f =""

k=1

$kwk , $k ! C.

(ii) We have sampling vectors {sk}k#N that form a Riez basis fora closed subspace S " H, and we can access the samplingvalues {%sk , f &}k#N.


The New Approach

Goal: To reconstruct the best possible approximation f ! Wbased on a finite set of the sampling information {%sk , f &}k#N, say,we are given m ! N samples {%sk , f &}m

k=1.


The New Approach

Since {sj} and {wj} are Riesz bases, there exist constantsA,B ,C ,D > 0 such that

A"

k#N

|'k |2 $

,

,

,

,

,

"

k#N

'kwk

,

,

,

,

,

2

$ B"

k#N

|'k |2

C"

k#N

|'k |2 $

,

,

,

,

,

"

k#N

'ksk

,

,

,

,

,

2

$ D"

k#N

|'k |2, , {'1,'2, . . .} ! l2(N).

(8)


The New Approach

Now let U be defined as in (7), choose n ! N and compute thesolution {$1, . . . , $n} of the following equation:

A

&

'

'

'

(

$1

$2...

$n

)

*

*

*

+

= PnU$Pm

&

'

'

'

(

%s1, f &%s2, f &

...%sm, f &

)

*

*

*

+

, A = PnU$PmUPn|PnH, (9)

provided a solution exists (later we will provide estimates on thesize of n,m for (9) to have a unique solution). Finally we let

f =n

"

k=1

$kwk . (10)


Theoretical Results

PropositionLet H be a separable Hilbert space and S,W " H be closedsubspaces. Suppose that {sk}k#N and {wk}k#N are Riesz bases forS and W respectively. In particular, we have constantsA,B ,C ,D > 0 such that (8) is satisfied. Define U as in (7). ThenU is a bounded operator on l2(N) with )U) $

-BD. Also, U is

invertible if and only if W ( S% = {0} and W% ( S = {0}, with)U!1) $

-

(AC )!1.


Theoretical Results

Theorem[Adcock,H’10] Let H be a separable Hilbert space and S,W " Hbe closed subspaces such that W ( S% = {0}. Suppose that{sk}k#N and {wk}k#N are Riesz bases for S and W respectivelywith constants A,B ,C ,D > 0. Suppose that

f ="

k#N

$kwk , $ = {$1,$2, . . . , } ! l2(N).

Let n ! N. Then there is an M ! N such that for all m . M thenthe solution {$1, . . . , $n} to (9) is unique. Also, if f is as in (10)then

)f # f )H $-

B(1 + Kn,m))P%n $)l2(N), (11)

where

Kn,m =,

,

,(PnU

$PmUPn|PnH)!1PnU$PmUP%

n

,

,

,. (12)


Theoretical Results

CorollaryWith the same assumptions as in Theorem 2, for fixed n ! N then

,

,(PnU$PmUPn|PnH)!1

,

, #',

,(PnU$UPn|PnH)!1

,

, $,

,(U$U)!1,

, ,(13)

as m ' +. In addition, if U is an isometry onto its range (inparticular, when {wk}k#N, {sk}k#N are orthonormal) then, for fixedn ! N, it follows that

Kn,m #' 0, m ' +.


Theoretical Results

PropositionLet F denote the Fourier transform on L2(Rd). Suppose that{#j}j#N is a Riesz basis with constants A,B (as in (8)) for asubspace V " L2(Rd) such that there exists a T > 0 withsupp(#j) " [#T ,T ]d for all j ! N. For " > 0, let ( : N ' ("Z)d

be a bijection. Define the infinite matrix

U =

&

'

'

'

(

u11 u12 u13 . . .u21 u22 u23 . . .u31 u32 u33 . . ....

......

. . .

)

*

*

*

+

, uij = (F#j )(((i)).

Then, for " $ 12T , we have that U : l2(N) ' l2(N) is bounded

and invertible onto its range with )U) $-

"!dB and)U!1) $

-"dA!1 .


Theoretical Results

Theorem(The Generalized Sampling Theorem)[Adcock,H’10] With the same setupas in Proposition 4, set

f = Fg , g =#"

j=1

$j#j ! L2(Rd ),

and let Pn denote the projection onto span{e1, . . . , en}. Then, for everyn ! N there is an M ! N such that, for all m . M, the solution to

A

&

'

'

'

(

$1

$2...

$n

)

*

*

*

+

= PnU"Pm

&

'

'

'

(

f (((1))f (((2))

...f (((m))

)

*

*

*

+

, A = PnU"PmUPn|PnH, (14)

is unique.


Theoretical Results

TheoremAlso, if

gn,m =n

"

j=1

$j#j , fn,m =n

"

j=1

$jF#j ,

then

)g # gn,m)L2(Rd ) $-

B(1 + Kn,m))P$n $)l2(N), $ = {$1, $2, . . .}, (15)

and)f # fn,m)L!(Rd ) $ (2T )d/2

-B(1 + Kn,m))P$

n $)l2(N), (16)

where Kn,m is given in (12) and satisfies (13). Moreover, when {#j}j%N

is an orthonormal set, it follows that

Kn,m #' 0, m ' +,

for fixed n.


Norm Bounds

0 50 100 150 200 250 3000

200

400

600

800

1000

1200

1400

0 50 100 150 200 250 3000

200

400

600

800

1000

1200

1400

Figure: The left figure shows m (for the Haar wavelet) such thatKn,m $ 1 together with the functions (in black) x /' 4.1x . The rightfigure shows m such that Kn,m $ 2 together with the function x /' 3.75x .


Example (reconstruction from Fourier Transform) I

Suppose now that we consider the function

g(t) = cos(2)t)&[0.5,1](t).

In this case, due to the discontinuity, forming

gN = "N

"

n=!N

f (n") e2!in"·, " =1

2, N ! N, (17)

may be less than ideal, since the convergence gN ' g as N ' + may beslow.Suppose that we are given the finite collection of samples

*f = {f (#N"), f ((#N + 1)"), . . . , f ((N # 1)"), f (N")}, (18)

with N = 900 and " = 12 . Define $ = {$1, . . . , $n} by equation (14), and

let

gn,m =n

"

j=1

$j+j , m = 1801, n = 500.


Example (reconstruction from Fourier Transform) II

0 0.2 0.4 0.6 0.8 1

−1

−0.5

0

0.5

1

0 0.2 0.4 0.6 0.8 1

−1

−0.5

0

0.5

1

0 0.2 0.4 0.6 0.8 1

−1

−0.5

0

0.5

1

0.48 0.5 0.52 0.54 0.56−1.04

−1.02

−1

−0.98

−0.96

0.48 0.5 0.52 0.54 0.56−1.04

−1.02

−1

−0.98

−0.96

0.48 0.5 0.52 0.54 0.56−1.04

−1.02

−1

−0.98

−0.96

Figure: The upper figures show gN (left), gn,m (middle) and g (right) onthe interval [0, 1]. The lower figures show gN (left), gn,m (middle) and g(right) on the interval [0.47, 0.57].


Example (reconstruction from point samples) I

In this example we consider the following problem. Let f ! L2(R) suchthat

f = Fg , g(x) =K

"

j=1

'j+j(x) + sin(2)x)&[0.3,0.6](x),

for K = 400, where {+j} are Haar wavelets on [0, 1], and {'j}Kj=1 are

some arbitrarily chosen real coe!cients in [0, 10]. Suppose that we canaccess the following pointwise samples of f :

*f = {f (#N"), f ((#N + 1)"), . . . , f ((N # 1)"), f (N")},

with " = 12 and N = 600. We may form

fN(t) =N

"

k=!N

f

#

k

2

$

sinc(2t # k), N = 600.


Example (reconstruction from point samples) II

−5000 0 5000

−20−10

01020

−5000 0 5000

−20−10

01020

Figure: The figure shows Re(f ) (left) and Im(f ) (right) on the interval[#5000, 5000].


Example (reconstruction from point samples) III

Let

fn,m =n

"

j=1

$jF+j , n = 500, m = 1201

where {$1, . . . , $n} satisfies (14).In this case we have

,

,

,(PnU

"PmUPn|PnH)!1,

,

,$ 0.9022,

,

,

,(PnU

"PmUPn|PnH)!1 PnU"Pm

,

,

,$ 0.9498.


Example (reconstruction from point samples) IV

−5000 0 50000

5

10

−5000 0 50000

0.5

1

x 10−3

Figure: The figure shows the error |f # fN | (left) and |f # f | (right) onthe interval [#5000, 5000].


computational complexity and compressed sensingpotts/cms/cms10/slides/slideshans… ·...

Documents