flexible krylov methods for p regularization€¦ · introduction methods based on fgk sparsity...

Introduction Methods based on FGK Sparsity under transform Numerical experiments Conclusions

Flexible Krylov Methods for `p Regularization

Silvia GazzolaJoint work with Julianne Chung and Malena Sabaté Landman

Department of Mathematical Sciences

Modern Challenges in ImagingTufts University, August 6, 2019

S. Gazzola (UoB) Regularization by Flexible Krylov Methods August 6, 2019 1 / 28


What is this talk about?

Regularization of linear inverse problemsAxtrue + � = b ,

whereb ∈ RM observations or measurementsxtrue ∈ RN desired parametersA ∈ RM×N ill-conditioned matrix models forward process� ∈ RM additive Gaussian noise

image deblurring and denoising

computed tomography



What is this talk about?Regularization of linear inverse problems

Axtrue + � = b ,where

b ∈ RM observations or measurementsxtrue ∈ RN desired parametersA ∈ RM×N ill-conditioned matrix models forward process� ∈ RM additive Gaussian noise

image deblurring and denoising

computed tomography



Outline

1 Introduction`p variational regularizationIteratively Re-weighted Norm (IRN) methods

2 Methods based on the Flexible Golub-Kahan (FGK) algorithmFlexible Golub-Kahan (FGK) AlgortihmFLSQR and FLSMRHybrid FLSQR and Hybrid FLSMR

3 Sparsity under transformInvertible transforms (wavelets)Non-Invertible transforms (TV)

4 Numerical experiments

5 Conclusions



Applying variational regularization...

xreg = arg minx∈RN

‖Ax− b‖22 + λR(x) , λ > 0

R(x) = ‖Lx‖22Krylov methods are popular in this setting

xk ∈ Kk (C, d) = span{d,Cd, . . . ,Ck−1d}rk = b− Axk ⊥ Kk (C′, d′)

AZk = Wk+1Gk

xk = Zkyk , yk = arg miny∈Rk‖gk − Gky‖22+λk‖Lky‖22

Fast semi-convergence

Iteration 150

Iteration 0

20 40 60 80 100 120 1400.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1.1

C G LSHyB R

Iteration

Rela

tive

Erro

r

Hybrid Method Stabilizes the Error

Hanke (1995); Frommer and Maas (1999);O’Leary and Simmons (1981); Calvetti, Morigi, Reichel, Sgallari (2000);Kilmer, Hansen, Espanol (2007); Chung and Palmer (2015); G., Novati, Russo (2015)...





‖Ax− b‖22 + λR(x) , λ > 0

R(x) = ‖Lx‖22

Krylov methods are popular in this setting


AZk = Wk+1Gk



Iteration 150

Iteration 0

20 40 60 80 100 120 1400.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1.1

C G LSHyB R

Iteration

Rela

tive

Erro

r







‖Ax− b‖22 + λR(x) , λ > 0



AZk = Wk+1Gk



Iteration 150

Iteration 0

20 40 60 80 100 120 1400.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1.1

C G LSHyB R

Iteration

Rela

tive

Erro

r







‖Ax− b‖22 + λR(x) , λ > 0



AZk = Wk+1Gk

xk = Zkyk , yk = arg miny∈Rk‖gk − Gky‖22

+λk‖Lky‖22


Iteration 150

Iteration 0

20 40 60 80 100 120 1400.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1.1

C G LSHyB R

Iteration

Rela

tive

Erro

r







‖Ax− b‖22 + λR(x) , λ > 0



AZk = Wk+1Gk



Iteration 150

Iteration 0

20 40 60 80 100 120 1400.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1.1

C G LSHyB R

Iteration

Rela

tive

Erro

r







‖Ax− b‖22 + λR(x) , λ > 0

`p regularization(we will consider R(x) = ‖x‖pp , R(x) = ‖Ψx‖pp , R(x) = TVp(x); p ≥ 1, p > 0)

Sub-gradient strategiesShevade and Keerthi (2003), Perkins (2003), Andrew and Gao (2007), ...

Constrained optimizationChen et al (1999), Bertsekas (2004), Gafni and Bertsekas (1984), ...

Iterative shinkage-thresholding algorithms (ISTA)Bioucas-Dias and Figueiredo(2007), Giryes, Elad and Eldar (2011), Beck andTeboulle (2009), Goldstein and Osher (2009), Osher et al (2005) ....

Rodriguez and Wohlberg (2008), Renaut et al (2017), ...

Generalized Krylov methods for `p − `qLanza et al (2015), Huang, Lanza, Morigi, Reichel, Sgallari (2017),Buccini and Reichel (2019),...Gazzola and Nagy (2014)...





‖Ax− b‖22 + λR(x) , λ > 0

`p regularization(we will consider R(x) = ‖x‖pp , R(x) = ‖Ψx‖pp , R(x) = TVp(x); p ≥ 1, p > 0)

Sub-gradient strategiesShevade and Keerthi (2003), Perkins (2003), Andrew and Gao (2007), ...

Constrained optimizationChen et al (1999), Bertsekas (2004), Gafni and Bertsekas (1984), ...

Iterative shinkage-thresholding algorithms (ISTA)Bioucas-Dias and Figueiredo(2007), Giryes, Elad and Eldar (2011), Beck andTeboulle (2009), Goldstein and Osher (2009), Osher et al (2005) ....

Iteratively re-weighted normRodriguez and Wohlberg (2008), Renaut et al (2017), ...

Generalized Krylov methods for `p − `qLanza et al (2015), Huang, Lanza, Morigi, Reichel, Sgallari (2017),Buccini and Reichel (2019),...

Flexible Arnoldi methods (for square problems)Gazzola and Nagy (2014)...



A basic Iteratively Re-weighted Norm (IRN) strategy ...

[Rodriguez and Wohlberg (2008)]

Let Ψ = I, p = 1.

Turn `1-problems into a sequence of `2-problems:

‖x‖1 ≈ ‖L(x)x‖22

IRN algorithmInput: A, b, x0(= 0), L0 = L(x0)(= I)

For k = 1, . . . , till a stopping criterion is satisfiedTill a stopping criterion is satisfied: run an (iterative) solver for

xk = arg minx∈RN

‖b− Ax‖22 + λ‖Lk−1x‖22

Update Lk = diag(

1/√

fτ (|xk |))

, fτ (|[xk ]i |) ={|[xk ]i | if |[xk ]i | ≥ τ1τ2 if |[xk ]i | < τ1

Let Lk = L(xk), then xk+1 = L−1k yk+1 where



A basic Iteratively Re-weighted Norm (IRN) strategy ...

[Rodriguez and Wohlberg (2008)]

Let Ψ = I, p = 1. Turn `1-problems into a sequence of `2-problems:

‖x‖1 ≈ ‖L(x)x‖22



xk = arg minx∈RN

‖b− Ax‖22 + λ‖Lk−1x‖22

Update Lk = diag(

1/√

fτ (|xk |))





A basic Iteratively Re-weighted Norm (IRN) strategy ...[Rodriguez and Wohlberg (2008)]Let Ψ = I, p = 1. Turn `1-problems into a sequence of `2-problems:

‖x‖1 ≈ ‖L(x)x‖22


For k = 1, . . . , till a stopping criterion is satisfied

Till a stopping criterion is satisfied: run an (iterative) solver for

xk = arg minx∈RN

‖b− Ax‖22 + λ‖Lk−1x‖22

Update Lk = diag(

1/√

fτ (|xk |))






‖x‖1 ≈ ‖L(x)x‖22



xk = arg minx∈RN

‖b− Ax‖22 + λ‖Lk−1x‖22

Update Lk = diag(

1/√

fτ (|xk |))






‖x‖1 ≈ ‖L(x)x‖22



xk = arg minx∈RN

‖b− Ax‖22 + λ‖Lk−1x‖22

Update Lk = diag(

1/√

fτ (|xk |))



yk+1 = arg miny

∥∥∥AL−1k y− b∥∥∥22 + λ ‖y‖22S. Gazzola (UoB) Regularization by Flexible Krylov Methods August 6, 2019 6 / 28


... revisited within Flexible Krylov methods ...[G. and Nagy (2014)]

1 For A ∈ RN×N , use flexible Arnoldi to generate basis vectors:[Saad (1993, 2003)]

Zk =[L−11 v1 · · · L

−1k vk

]∈ RN×k

whereAZk = Vk+1Hk

Vk+1 =[v1 . . . vk+1

]∈ RN×(k+1) has orthonormal columns (ONC)

Hk ∈ R(k+1)×k is upper Hessenberg

2 Compute solution xk = x0 + Zkyk where

yk = arg miny

12 ‖Hky− ‖r0‖2 e1‖

22 + λ ‖y‖

22



Flexible Golub-Kahan (FGK) Process[Chung and G. (2018)]

A new flexible factorization

Given A ∈ RM×N ,b ∈ RM , initialize u1 = b/β1 where β1 = ‖b‖.After k iterations with changing preconditioners Lk , we haveRelated to inexact Krylov methods [Simoncini and Szyld (2007)]

Zk =[L−11 v1 · · · L

−1k vk

]∈ RN×k

Mk ∈ R(k+1)×k upper HessenbergTk ∈ Rk×k upper triangularUk+1 =

[u1 . . . uk+1

]∈ RM×(k+1) ONC

Vk =[v1 . . . vk

]∈ RN×k ONC

such thatAZk = Uk+1Mk and A>Uk+1 = Vk+1Tk+1

Remarks:If Lk = L, get right-preconditioned GK bidiagonalizationAdditional orthogonalizations and storage



Flexible Golub-Kahan (FGK) Process[Chung and G. (2018)]Given A ∈ RM×N ,b ∈ RM , initialize u1 = b/β1 where β1 = ‖b‖.After k iterations with changing preconditioners Lk , we haveRelated to inexact Krylov methods [Simoncini and Szyld (2007)]

Zk =[L−11 v1 · · · L

−1k vk

]∈ RN×k

Mk ∈ R(k+1)×k upper HessenbergTk ∈ Rk×k upper triangularUk+1 =

[u1 . . . uk+1

]∈ RM×(k+1) ONC

Vk =[v1 . . . vk

]∈ RN×k ONC

such thatAZk = Uk+1Mk and A>Uk+1 = Vk+1Tk+1

Remarks:If Lk = L, get right-preconditioned GK bidiagonalizationAdditional orthogonalizations and storage



Flexible LSQR and flexible LSMR

New flexible solvers

1 Use flexible GK to generate basis vectors:

Zk =[L−11 v1 · · · L

−1k vk

]∈ Rn×k

AZk = Uk+1Mk and A>Uk+1 = Vk+1Tk+1

2 Compute solution xk = Zkyk whereFlexible LSQR (FLSQR)

yk = arg miny∈Rk

‖Mky− β1e1‖22Optimality property:xk minimizes ‖Axk − b‖2 over x0 + span{Zk}.Flexible LSMR (FLSMR)

yk = arg miny∈Rk

‖Tk+1Mky− β1m11e1‖22Optimality property:xk minimizes

∥∥A>(Axk − b)∥∥2 over x0 + span{Zk}.Equivalency result:FLSMR is equivalent to FGMRES applied to the normal equations.



Flexible LSQR and flexible LSMR1 Use flexible GK to generate basis vectors:

Zk =[L−11 v1 · · · L

−1k vk

]∈ Rn×k


2 Compute solution xk = Zkyk whereFlexible LSQR (FLSQR)

yk = arg miny∈Rk


yk = arg miny∈Rk






Zk =[L−11 v1 · · · L

−1k vk

]∈ Rn×k

AZk = Uk+1Mk and A>Uk+1 = Vk+1Tk+12 Compute solution xk = Zkyk where

Flexible LSQR (FLSQR)

yk = arg miny∈Rk

‖Mky− β1e1‖22

Optimality property:xk minimizes ‖Axk − b‖2 over x0 + span{Zk}.

Flexible LSMR (FLSMR)yk = arg min

y∈Rk‖Tk+1Mky− β1m11e1‖22

Optimality property:xk minimizes





Zk =[L−11 v1 · · · L

−1k vk

]∈ Rn×k



yk = arg miny∈Rk


yk = arg miny∈Rk

‖Tk+1Mky− β1m11e1‖22

Optimality property:xk minimizes





Zk =[L−11 v1 · · · L

−1k vk

]∈ Rn×k



yk = arg miny∈Rk


yk = arg miny∈Rk





Flexible GK (FGK) hybrid methods

New flexible solversused in a hybrid framework


Zk =[L−11 v1 · · · L

−1k vk

]∈ RN×k


2 Compute solution xk = Zkyk , whereFlexible GK Tikhonov - R (FLSQR-R)

yk = arg miny∈Rk

‖Mky− β1e1‖22 + λk ‖Rky‖22 , Zk = QkRk

Flexible GK Tikhonov - I (FLSQR-I)

yk = arg miny∈Rk

‖Mky− β1e1‖22 + λk ‖y‖22





Zk =[L−11 v1 · · · L

−1k vk

]∈ RN×k



yk = arg miny∈Rk



yk = arg miny∈Rk

‖Mky− β1e1‖22 + λk ‖y‖22





Zk =[L−11 v1 · · · L

−1k vk

]∈ RN×k


2 Compute solution xk = Zkyk

, whereFlexible GK Tikhonov - R (FLSQR-R)

yk = arg miny∈Rk



yk = arg miny∈Rk

‖Mky− β1e1‖22 + λk ‖y‖22





Zk =[L−11 v1 · · · L

−1k vk

]∈ RN×k



yk = arg miny∈Rk



yk = arg miny∈Rk

‖Mky− β1e1‖22 + λk ‖y‖22



FLSQR-R: Approximate singular values of A

R−>k M>k MkR−1k = R

−>k M

>k U>k+1Uk+1MkR−1k = Q

>k A>AQk

50 100 150 200 250 300 350 400 450 500

iteration k

10-15

10-10

10-5

100

Singular values of A

FLSQR

FLSQR-R

Figure: This plot compares the singular values of A to the singular values of Mkfrom FLSQR and of MkR−1k from FLSQR-R, for iterations k between 20 and 420in increments of 100.



Solving the transformed problem

Let Ψ 6= I (invertible), p = 1 (e.g., Ψ : image domain → wavelet domain)

Equivalent problems (for Ψ̃ orthogonal):[Belge, Kilmer, Miller (2000)]

minx∈RN

‖Ax− b‖22 + λ ‖Ψx‖1 ⇔ minx∈RN‖Ψ̃AΨ−1︸︷︷︸

H

Ψx︸︷︷︸s− Ψ̃b︸︷︷︸

d

‖22 + λ‖Ψx︸︷︷︸s‖1

Solution subspace for flexible Arnoldi:

sk ∈ span{L−11 v̂1,L−12 v̂2, . . . ,L

−1k v̂k}, where

v̂1 = d/‖d‖2v̂2 = ONC(HL−11 v̂1)v̂3 = ONC(HL−12 v̂2)...xk = Ψ−1sk

Analogously for flexible Golub-Kahan (possibly without Ψ̃).




Let Ψ 6= I (invertible), p = 1 (e.g., Ψ : image domain → wavelet domain)Equivalent problems (for Ψ̃ orthogonal):[Belge, Kilmer, Miller (2000)]

minx∈RN


H

Ψx︸︷︷︸s− Ψ̃b︸︷︷︸

d

‖22 + λ‖Ψx︸︷︷︸s‖1


sk ∈ span{L−11 v̂1,L−12 v̂2, . . . ,L

−1k v̂k}, where

v̂1 = d/‖d‖2v̂2 = ONC(HL−11 v̂1)v̂3 = ONC(HL−12 v̂2)...

xk = Ψ−1skAnalogously for flexible Golub-Kahan (possibly without Ψ̃).





minx∈RN


H

Ψx︸︷︷︸s− Ψ̃b︸︷︷︸

d

‖22 + λ‖Ψx︸︷︷︸s‖1


sk ∈ span{L−11 v̂1,L−12 v̂2, . . . ,L

−1k v̂k}, where


Analogously for flexible Golub-Kahan (possibly without Ψ̃).





minx∈RN


H

Ψx︸︷︷︸s− Ψ̃b︸︷︷︸

d

‖22 + λ‖Ψx︸︷︷︸s‖1


sk ∈ span{L−11 v̂1,L−12 v̂2, . . . ,L

−1k v̂k}, where


Analogously for flexible Golub-Kahan (possibly without Ψ̃).S. Gazzola (UoB) Regularization by Flexible Krylov Methods August 6, 2019 12 / 28


An illustration: sparsity in a wavelet domainSignal Wavelet

10 20 30 40 50 60-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6exact

corrupted

10 20 30 40 50 60

-1

-0.5

0

0.5

1

10 20 30 40 50 60-1

-0.5

0

0.5

1

GMRES

FGMRES

10 20 30 40 50 60

-1

-0.5

0

0.5

1



An illustration: 2nd and 4th basis vectorsSignal Wavelet

10 20 30 40 50 60-0.4

-0.2

0

0.2

0.4

10 20 30 40 50 60-0.5

0

0.5

10 20 30 40 50 60-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

10 20 30 40 50 60

-0.4

-0.2

0

0.2

0.4



TV penalization

Let R(x) = TV(x).1d case:TV(x) = ‖D1dx‖1' ‖W1dDx‖22, where

D1d =

1 −1. . . . . .1 −1

∈ R(N−1)×N , W = diag (|D1dx|−1/2)2d case:[Wohlberg and Rodriguez. An iteratively reweighted norm algorithm for TV. IEEE, 2007]TV(x) = ‖

((Dhx)2 + (Dvx)2

)1/2 ‖1' ‖WD2dx‖22, whereD2d =

[DhDv]

=[

D1d ⊗ II⊗D1d

], Ŵ = diag

(((Dhx)2 + (Dvx)2

)−1/4),W =

[Ŵ 00 Ŵ

]



Smoothing Norm, A ∈ RN×N

Standard form transformation:

ȳL = arg minȳ‖Āȳ− b̄‖22 + λ‖ȳ‖22 , where

Ā = AL†A = A[I− (A(I− L†L))†A]

b̄ = b− Ax0xL = L†AȳL + x0 = x̄L + x0

.

[Hansen and Jensen. Smoothing-Norm Preconditioning for Reg. Min.-Res. SIMAX, 2007]

Write:

xL = x̄L + x0 = L†AȳL + x0 = L†AȳL + Kt0 , where R(K) = N (L) , L

†A rectangular .

Equivalently:

A[L†A, K

] [ ȳLt0

]= b ,

and, further: [(L†A)

T AL†A (L†A)

T AKKT AL†A K

T AK

][ȳLt0

]=[

(L†A)T b

KT b

].

Schur complement system:

(L†A)T PAL†Aȳ = (L

†A)

T Pb, where P = I− AK(KT AK)−1KT ∈ RN×N .



TV regularization, A ∈ RN×N[G. and Sabaté Landman (2019)]

Similar idea, with reweighting...

(D†)T PA(WD)†Aȳ = (D†)T Pb

Building a better approximation subspace for the solution!

L = WD (with W = W(xk)):flexible GMRES (instead of restarted GMRES);large-scale computations:

approximating L†(exploiting structure, and running preconditioned LSQR or LSMR)thresholding the weights







L = WD (with W = W(xk)):flexible GMRES (instead of restarted GMRES);

large-scale computations:approximating L†(exploiting structure, and running preconditioned LSQR or LSMR)thresholding the weights







L = WD (with W = W(xk)):flexible GMRES (instead of restarted GMRES);large-scale computations:

approximating L†(exploiting structure, and running preconditioned LSQR or LSMR)thresholding the weights



A simple 1D example...

50 100 150 200 250-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5exact

corrupted

50 100 150 200 250-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5GMRES

GMRES (D)

TV-FGMRES

basis vectors

50 100 150 200 250-0.3

-0.2

-0.1

0

0.1

0.2

0.3

v1

v2

v3

50 100 150 200 250-6

-4

-2

0

2

4

6

8

50 100 150 200 250-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

50 100 150 200 250-1.5

-1

-0.5

0

0.5

1

1.5

z1

z2

z6

z8

z20



Image deblurring example with Ψ = I[G., Hansen, Nagy. IR Tools (2018)]https://github.com/silviagazzola/IRtools

http://www2.compute.dtu.dk/ pcha/IRtools/

true PSF observed

Image is 256× 256 pixelsNoise level is 5× 10−2

Reflexive boundary conditions



Reconstruction errors

0 20 40 60 80 100

Iteration

0

0.5

1

1.5

2

Rela

tive E

rror

FLSQR

FLSQR-I

FLSQR-R

LSQR

HyBR DP

Reconstruction errors computed as ‖xk−xtrue‖2‖xtrue‖2λ for FLSQR-I and FLSQR-R use discrepancy principle



Basis imagesk=10

FLS

QR

-RLS

QR

k=20 k=100



Comparison to other methods

0 100 200 300 400 500

Iteration

0

0.2

0.4

0.6

0.8

1R

ela

tive

Err

or

FLSQR-R

GAT

PIRN

FISTA

SpaRSA

GAT = Generalized Arnoldi-TikhonovPIRN† = Preconditioned iteratively re-weighted normFISTA† = Fast iterative-shrinkage-thresholding algorithmSpaRSA† = Sparse Reconstruction by Separable Approximation

(† uses λ from FLSQR-R)S. Gazzola (UoB) Regularization by Flexible Krylov Methods August 6, 2019 22 / 28


Tomography example with Φ 6= I

[G., Hansen, Nagy. IR Tools (2018)]

n = 256; optn = PRtomo(‘defaults’);optn=PRset(optn,‘angles’,0:2:179,‘p’,round(sqrt(2)*n),‘d’,sqrt(2)*n);[A, b, x, ProbInfo] = PRtomo(n, optn);figure, PRshowx(x, ProbInfo)bn = PRnoise(b, 1e-2);

phantom is 256× 256 pixelsA has size 32580× 65536 (approx. 50% undersampling)Ψ is a 4-level 2D Haar wavelet transform



Tomography example with Φ 6= I[G., Hansen, Nagy. IR Tools (2018)]

n = 256; optn = PRtomo(‘defaults’);optn=PRset(optn,‘angles’,0:2:179,‘p’,round(sqrt(2)*n),‘d’,sqrt(2)*n);[A, b, x, ProbInfo] = PRtomo(n, optn);

figure, PRshowx(x, ProbInfo)bn = PRnoise(b, 1e-2);

phantom is 256× 256 pixelsA has size 32580× 65536 (approx. 50% undersampling)

Ψ is a 4-level 2D Haar wavelet transform




n = 256; optn = PRtomo(‘defaults’);optn=PRset(optn,‘angles’,0:2:179,‘p’,round(sqrt(2)*n),‘d’,sqrt(2)*n);[A, b, x, ProbInfo] = PRtomo(n, optn);figure, PRshowx(x, ProbInfo)

bn = PRnoise(b, 1e-2);







phantom is 256× 256 pixelsA has size 32580× 65536 (approx. 50% undersampling)Ψ is a 4-level 2D Haar wavelet transform



Reconstructed phantoms



Image deblurring example[G., Hansen, Nagy. IR Tools (2018)]Cameraman example: 256× 256 pixels.

blurred & noisy SN-GMRES

TV-FGMRES fast gradient-based method



Image deblurring example

0 20 40 60 8010

-1

100

GMRES(D)

FGMRES(1)

FGMRES(0.1)

FBTV

ReSt-GAT

ReSt-GKB

0 20 40 60 8010

4

106

108

1010

Relative Error History Total Variation History



Tomography example with flexible TV regularization

small PRtomo example: 32× 32 pixels; A ∈ R2025×1024... ongoing work

5 10 15 20 25 30

5

10

15

20

25

30

-1

40

-0.8

-0.6

30 40

-0.4

3020

-0.2

20

0

1010

0 0

Exact phantomnoisy (Gaussian white noise, ‖e‖/‖btrue‖ = 10−2) image





5 10 15 20 25 30

5

10

15

20

25

30

-1.2

40

-1

-0.8

-0.6

30 40

-0.4

-0.2

3020

0

20

0.2

1010

0 0

LSQR





5 10 15 20 25 30

5

10

15

20

25

30

-1.2

40

-1

-0.8

-0.6

30 40

-0.4

-0.2

3020

0

20

0.2

1010

0 0

LSQR (D)





5 10 15 20 25 30

5

10

15

20

25

30

-1.2

40

-1

-0.8

-0.6

30 40

-0.4

-0.2

3020

0

20

0.2

1010

0 0

TV-LSQR





5 10 15 20 25 30

5

10

15

20

25

30

-1.2

40

-1

-0.8

-0.6

30 40

-0.4

-0.2

3020

0

20

0.2

1010

0 0

TV-LSQR “0 norm”



Summary of benefits ...

Flexible Krylov methods4 Avoid inner-outer schemes (current solution immediately incorporated

in basis)4 Both square (flexible Arnoldi) non-square problems (flexible

Golub-Kahan)4 Optimality and equivalency results

Hybrid method4 Stabilize reconstruction errors4 Automatic choice of λ and stopping criteria

Transformed problem4 Enforce sparsity in a transform4 Connections to multi-parameter regularization



... and (hopefully) (much) more work to do ...

deeper theoretical analysis (convergence, recovery guarantees);extension to yet other regularization terms or constraints;parameter choice for nonlinear problems and solvers;

References

J. Chung and S. G. Flexible Krylov methods for `p regularization. SISC, 2019.S. G. and M. Sabaté Landman. Flexible GMRES for Total Variation regularization.BIT, 2019.S. G., P. C. Hansen, and J. Nagy. IR Tools: A MATLAB Package of IterativeRegularization Methods and Large-Scale Test Problems. Numer. Algorithms, 2018.https://github.com/silviagazzola/IRtools




deeper theoretical analysis (convergence, recovery guarantees);

extension to yet other regularization terms or constraints;parameter choice for nonlinear problems and solvers;

References





deeper theoretical analysis (convergence, recovery guarantees);extension to yet other regularization terms or constraints;

parameter choice for nonlinear problems and solvers;

References






References






References



Introductionp variational regularizationIteratively Re-weighted Norm (IRN) methods

Methods based on the Flexible Golub-Kahan (FGK) algorithmFlexible Golub-Kahan (FGK) AlgortihmFLSQR and FLSMRHybrid FLSQR and Hybrid FLSMR

Sparsity under transformInvertible transforms (wavelets)Non-Invertible transforms (TV)

Numerical experimentsConclusions

flexible krylov methods for p regularization€¦ · introduction methods based on fgk sparsity...

Documents