slides in pdf

/centre for analysis, scientific computing and applications

Introduction Theory of constrained optimization Numerical variational methods Iterative nonnegative regularization methods Numerical test results Conclusions

Optimization with nonnegativity constraints

Arie [email protected]

CASA Seminar, May 30, 2007



Seminar: Inverse problems

1 Introduction Yves van Gennip February 212 Regularization strategies Miguel Patricio March 33 Regularization by Hans Groot March 21

Galerkin methods4 Inverse eigenvalue problems Marco Veneroni April 45 Image deblurring Willem Dijkstra April 186 Parameter identification Nico van der Aa May 27 Total variation regularization Mark van Kraaij May 238 Optimization with Arie Verhoeven May 30

nonnegativity constraints9 ?? Martijn Slob June 13

10 ?? Marc Noot June 20



Outline

1 Introduction

2 Theory of constrained optimization

3 Numerical variational methods

4 Iterative nonnegative regularization methods

5 Numerical test results

6 Conclusions



Formulation of the problem

Discretized mathematical model:

f truei = ftrue(xi).

Discrete Fourier Transform:

gtrue = K · ftrue.

Measurements, e.g. from astronomical imaging, satisfy

di ∼ Poisson(gtruei ) + Normal(0, σ2).

Goal: find f for given d and K.



Least Squares method with Tikhonovregularization

The least squares solution minimizes the functional:

Jls(f) =12‖Kf − d‖2 +

α

2‖f‖2.

The minimizer is given by

f lsα =

[KT K + αI

]−1KT d.

The regularization parameter α is selected to minimize‖f ls

α − ftrue‖. But this approach does not work in general whenftrue is unknown.



The L-curve method

Define X (α) = log ‖Kfα − d‖2 and Y (α) = log ‖fα‖2. WithTikhonov regularization X , Y are smooth functions. Then α canbe selected which maximizes the curvature function

κ(α) =X (α)Y (α)− X (α)Y (α)

(X (α)2 + Y (α)2)32

.

Note that this selected point corresponds to the "corner" of theL-curve, which plots X against Y . Although this method isnonconvergent, it can be used to improve the value of α withoutknowledge of ftrue.



Nonnegatively constrained minimizationFor astronomical imaging it is well-known that fi ≥ 0. Thus it ismore accurate to solve instead

minf

Jls(f) subject to f ≥ 0.

Even better is to include the stochastic information of themeasurements. Then we solve the following constrainedPoisson likelihood minimization problem

minf

Jlhd(f) subject to f ≥ 0,

where

Jlhd(f) =n∑

i=1

(gi +σ2)+n∑

i=1

((max{di , 0}+σ2) log(gi +σ2))+α

2‖f‖2.



Example

ftrue(x) =

750, 0.1 < x < 0.25,250, 0.3 < x < 0.32,5 · 105(x − 0.75)(0.85− x), 0.75 < x < 0.85,0 otherwise.



Constrained likelihood minimization



Outline

1 Introduction





6 Conclusions



Optimization

Consider the inequality constrained minimization problem

minf∈Rn

J(f) subject to c(f) ≥ 0.

Active set of indices:

A(f) = {i | ci(f) = 0}.



KKT conditions

Karush-Kuhn-Tucker (first order necessary) conditions forinequality constrained minimization: There exists a vector λsuch that

gradJ(f∗)−m∑

i=1

λ∗i gradci(f∗) = 0

andλ∗i ≥ 0, ci(f

∗) ≥ 0, λ∗i ci(f∗) = 0.

We define the projection of f onto C as

PC(f) = arg minv∈C

‖f − v‖.

The operator PC is well defined and continuous.



Nonnegativity constraints

Now we consider the problem

minf∈Rn

J(f) subject to f ≥ 0.

If J is continuously differentiable and f∗ a local minimizer, itfollows that λ∗ = gradJ(f∗). Then we obtain

∂J∂fi

(f∗) ≥ 0, f ∗i ≥ 0, f ∗i∂J∂fi

(f∗) = 0.

A point which satisfies these conditions is a critical point, but itneeds not to be a minimizer if.e.g. J is not strictly convex,



Nonnegativity constraints II

For nonnegativity constraints, we define the feasible setC = {f | f ≥ 0}. The projected gradient ∇CJ : C → Rn satisfies

[∇CJ]i =

{∂J∂fi

(f) if fi > 0,

min{0, ∂J∂fi

(f)} if fi = 0.

Thus f∗ is a critical point if and only if ∇CJ(f∗) = 0. For given Cwe define

P(f) = arg minv≥0

‖v − f‖.

If f∗ is a local minimizer, then

f∗ = P(f∗ − τgradJ(f∗)) for any τ > 0.



Outline

1 Introduction





6 Conclusions



Gradient projection method

ν := 0;f0 := nonnegative initial guess;begin

pν := −grad J(fν);τν := arg minτ>0 J(P(fν + τpν));fν+1 := P(fν + τνpν);ν := ν + 1;

end

This generalized Steepest Descent method converges linearlyto the global minimizer if J is strictly convex, coercive, andLipschitz continuous.



Projected Newton method


gν := grad J(fν);Identify active set Aν ;HR := reduced Hessian at fν ;s := −H−1

R gν ;τν := arg minτ>0 J(P(fν + τs));fν+1 := P(fν + τνs);ν := ν + 1;

end

The reduced Hessian equals

[HR]ij =

{δij if i ∈ A(f) or j ∈ A(f),∂2J

∂fi∂fjotherwise.

If the active set can be correctlyidentified, this algorithm will belocally quadraticallyconvergent.



Gradient projection-reduced Newton method


Gradient Projection StagepGP := −grad J(fν);τGP := arg minτ>0 J(P(fν + τpGP));

fGPν := P(fν + τGPpGP);

Reduced Newton StageIdentify active set A(fGP

ν );

gR := reduced gradient at fGPν ;

HR := reduced Hessian at fGPν ;

s := −H−1R gR ;

τRN := arg minτ>0 J(P(fGPν + τs));

fν+1 := P(fGPν + τRNs);

ν := ν + 1;end

We need

[gR(f)]i =

{0 if i ∈ A(f),

∂J∂fi

(f) otherwise.

This algorithm combines theglobal convergence of GradientProjection with the locallyquadratic rate of ProjectedNewton. For large-scaleproblems the linear systemHRs = −gR could be solved byan iterative method like theConjugate Gradient method.



Outline

1 Introduction





6 Conclusions



Richardson-Lucy iteration

Iterative methods use iteration count as regularizationparameter.Consider

J(f) =m∑

i=1

di log[Kf ]i .

Approximation of maximizer by Richardson-Lucy iteration:

f ν+1j =

f νj

kj

m∑i=1

kij

(di∑n

l=1 kil f νl

), where kj =

m∑l=1

klj .

We get a sequence of approximations to the maximizer of J(f)subject to

m∑i=1

[Kf ]i =m∑

i=1

di .



Modified Reduced Newton Steepest Descent

ν := 0;f0 := nonnegative initial guess;g0 := KT (Kf0 − d);γ := (g0, f0. ∗ g0);begin

pν := −fν . ∗ gν ;u := Kpν ;τbndry := min{−[fν ]i/[pν ]i | [pν ]i < 0};fν+1 := fν + τνpν ;

gν+1 := gν + τνKT u;γ := (gν+1, fν+1. ∗ gν+1);ν := ν + 1;

end



Outline

1 Introduction





6 Conclusions



1D



2D



Outline

1 Introduction





6 Conclusions



Summary

Nonnegitivity constraints

Theory of constrained optimizationVariational methods

Gradient projection methodProjected Newton methodGradient projection-reduced Newton methodGradient projection-CG method

Iterative methodsRichardson-Lucy iterationModified Steepest Descent algorithm

Numerical test results



Conclusions

Optimization with nonnegativity constraints often leads tomore accurate reconstructions with e.g. less unwantedoscillations.

Optimizing Poisson likelihood is more accurate than LeastSquares.

Iterative methods are preferable if no good a priori value ofthe regularization parameter is available.

Variational regularization methods are more flexible,because they allow the use of prior information about thesolution and constraints.



Literature.

C.R. Vogel: Computational methods for inverse problems, SIAM,Philadelphia, 2002, pp. 151-171.

J. Nocedal and S.J. Wright: Numerical optimization,Springer-Verlag, New York, 1999.

S.G. Nash and A. Sofer: Linear and nonlinear programming,McGraw-Hill, New York, 1996.

H.W. Engl, M. Hanke and A. Neubauer: Regularization of inverseproblems, Kluwer Academic Publishers, Dordrecht, 1996.

W.H. Richardson: Bayesian-based iterative methods for imagerestoration, Journal of the Optical Society of America, 62 (1972),pp. 55-59.

B. Lucy: An iterative method for the rectification of observeddistributions, Astronomical Journal, 79 (1974), pp. 745-754.



Questions?

slides in pdf

Documents