an introduction and example of intervalneum/glopt/gicolag/talks/...local optimum. (see j. e. dennis...

(slide 1)

IntroductionTheoretical andPractical Considerations

Alternate Approaches toGlobal Optimization

General Branch andBound Algorithms

IntervalTechniquesBounding Ranges

Proving Existence andUniqueness

Practicalities

Use ofStructure: AnExampleObtaining the ObjectiveLower Bound

Gradient-Based BoxRejection

Structure in theKuhn–Tucker Matrix

An Introduction and Example of IntervalTechniques in Global Optimization

R. Baker Kearfott

Department of MathematicsUniversity of Louisiana at Lafayette

GICOLAG, December, 2006

(slide 2)






Practicalities




Outline

1 IntroductionTheoretical and Practical ConsiderationsAlternate Approaches to Global OptimizationGeneral Branch and Bound Algorithms

2 Interval TechniquesBounding RangesProving Existence and UniquenessPracticalities

3 Use of Structure: An ExampleObtaining the Objective Lower BoundGradient-Based Box RejectionStructure in the Kuhn–Tucker Matrix

(slide 3)






Practicalities




The General GlobalOptimization Problem

Mathematical Description

minimize ϕ(x)

subject to ci(x) = 0, i = 1, . . . , m1,

gi(x) ≤ 0, i = 1, . . . , m2,

where ϕ : Rn → R and ci , gi : Rn → R.

• We refer to the region defined by the constraints as D.

• Often bounds on the search region are given byx = ([x1, x1], . . . [xn, xn]).

(slide 4)






Practicalities




Local Versus GlobalOptimization

• Global Optimization is not to be confused with localoptimization, often based on a single execution of asteepest-descent-like algorithm.

• Local optimization algorithm developers speak of"globalization," but mean only the design of algorithmvariants that increase the domain of convergence to alocal optimum. (See J. E. Dennis and R. B. Schnabel,Numerical Methods for Unconstrained Optimizationand Nonlinear Least Squares, Prentice–Hall, 1983.)

• Local optimization algorithms contain many heuristics,and do not always work; however, many usefulimplementations exist.

• Repeated application of a local algorithm is typicallyused in practice within global optimization algorithms.

(slide 5)






Practicalities




Outline




(slide 6)






Practicalities




Do We Want Local or GlobalOptima?

Examples

• ϕ is the cost of running a (nominally) $50,000,000 permonth plant:The plant manager would like the smallest possibleoperating cost, but would be happy with a 5% lowercost than before.

• ϕ represents the potential energy of a particularconformation of a molecule:The globally lowest value for ϕ gives the mostinformation, but local minima give some information,and finding the global minimum may not be practical.

• A mathematician has reduced a proof to showing thatϕ ≥ 1 everywhere:The global minimum must not only be found, but alsomust be rigorously proven to be so.

(slide 7)






Practicalities





Examples




(slide 8)






Practicalities





Examples




(slide 9)






Practicalities




Local Versus GlobalOptimization

Global Algorithms

• Global optimization is NP hard, meaning that nogeneral algorithm is known that solves all largeproblems efficiently.

• Thus, barring monumental discoveries, any generalalgorithm will fail for some high-dimensional problems.

• Some low-dimensional problems are also difficult, butmany can be solved.

• Successes have occurred for large problems ofpractical interest by taking advantage of structure.

• Useful results have also been obtained by replacingguarantees by heuristics.

• Advances in computer speed and algorithmconstruction allow ever more practical problems to besolved.

(slide 10)






Practicalities




Outline




(slide 11)






Practicalities




Classes of Global OptimizationAlgorithms

• There are two types of algorithms: stochastic anddeterministic.

• Deterministic algorithms can be either rigorous orheuristic.

(slide 12)






Practicalities




Stochastic Algorithms

Monte-Carlo search: Random points are generated in thesearch space. The point with lowest objectivevalue is taken to be the global optimum.

Simulated annealing: is similar to a local optimizationmethod, although larger objective values areaccepted with a probability that decreases asthe algorithm progresses.

Genetic algorithms: Attributes, such as values of aparticular coordinate, correspond to particular“genes.” “Chromosomes” of these genes arerecombined randomly, and only the bestresults are kept. Random “mutations” areintroduced.

(slide 13)






Practicalities




Deterministic Algorithms

• These either• involve using problem-specific structure to be assured

that computed points are global optimizers, or else• involve some kind of systematic global search over the

domain, generally branch and bound processes.

• The various branch and bound algorithms rely onestimates of ranges of the objective and constraintsover subdomains.

• The estimates on the range can be eithermathematically exact or heuristic.

• The mathematically exact estimates can be eitherrigorous (taking account of roundoff error) ornon-rigorous (using floating point arithmetic and otherapproximation algorithms without taking account ofnumerical errors).

(slide 14)






Practicalities











(slide 15)






Practicalities











(slide 16)






Practicalities











(slide 17)






Practicalities











(slide 18)






Practicalities




Outline




(slide 19)






Practicalities




The Abstract Algorithm

Initialization: Establish an upper bound ϕ on the globaloptimum over the feasible set.

Branching: Subdivide the initial region D into two or moresubregions D̃. (branching step, where webranch into subregions.)

Bounding: Bound the range of ϕ below over each D̃, toobtain

ϕ(D̃) ≤{

ϕ(x) | x ∈ D̃, c(x) = 0, g(x) ≤ 0.}

.

Fathoming: IF ϕ > ϕ THEN discard D̃,ELSE IF the diameter of D̃ is smaller than aspecified tolerance THEN put D̃ onto a list ofboxes containing possible global optimizers,ELSE Put D̃ is put onto a list for furtherbranching and bounding.END IF

(slide 20)






Practicalities




Obtaining ϕ

• Some algorithms rely on Lipschitz constants to obtainϕ. These constants can be determined rigorously orheuristically.

• ϕ may also be obtained withoutwardly rounded interval arithmetic ornon-rigorous interval arithmetic, explained below.

• Recent algorithms (especially during the past decade)involve relaxations of the global optimization problemto obtain ϕ. For example, the relaxation can be alinear program, solvable by commercial technology.

(slide 21)






Practicalities




Outline




(slide 22)






Practicalities




Bounding Ranges with IntervalArithmetic

• Each basic interval operation ¯ ∈ {+,−,×,÷, etc.} isdefined by

x ¯ y = {x ¯ y | x ∈ x and y ∈ y } .

• This definition can be made operational; for example,for x = [x , x ] and y = [y , y ], x + y = [x + y , x + y ];similarly, ranges of functions such as sin, exp can becomputed.

• Evaluation of an expression with this intervalarithmetic gives bounds on the range of theexpression.

• With directed rounding (e.g. using IEEE standardarithmetic), the computer can give mathematicallyrigorous bounds on ranges.

(slide 23)






Practicalities






x ¯ y = {x ¯ y | x ∈ x and y ∈ y } .




(slide 24)






Practicalities






x ¯ y = {x ¯ y | x ∈ x and y ∈ y } .




(slide 25)






Practicalities






x ¯ y = {x ¯ y | x ∈ x and y ∈ y } .




(slide 26)






Practicalities




Outline




(slide 27)






Practicalities





Interval Newton Methods

For the system f (x) = 0, f : Rn → Rn, an interval Newtonmethod is an operator N : x → x̃ :

x̃ = N(f ; x , x̌) = x̌ + v , where Σ(A,−f (x̌)

) ⊂ v ,

where A is a Lipschitz matrix for f over x and Σ(A,−f (x̌)

)is that set {x ∈ Rn} such that there exists an A ∈ A withAx = −f (x̌).

Theorem

Suppose x̃ is the image of x under an interval Newtonmethod. If x̃ ⊆ x , it follows that there exists a uniquesolution of f (x) = 0 within x .

(See A. Neumaier, Interval Methods for Systems ofEquations, Cambridge University Press, 1990.)

(slide 28)






Practicalities




Interval Newton MethodsA Practical Summary

• N(f ; x , x̌) can be computed similarly to a classicalpoint multivariate Newton step, but with intervalarithmetic.

• Any solutions of f (x) = 0 in x must be in N(f ; x , x̌).(Hence, an interval Newton can be used to reduce thevolume of x , an acceleration procedure.)

• N(f ; x , x̌) ⊂ x implies there is a unique solution tof (x) = 0 in N(f ; x , x̌), and hence in x .

• Another consequence: N(f ; x , x̌) ∩ x = ∅ implies x isfathomed, and x may be discarded.

• Interval Newton methods are locally quadraticallyconvergent.

(slide 29)






Practicalities




Linear RelaxationsA Third “Interval" Technique

• If we replace ϕ by ϕl , ϕl ≤ ϕ on the feasible set, wereplace each equality constraint by a pair of inequalityconstraints, and we replace each inequality constraintg i ≤ 0 by g i,l ≤ 0, g i,l ≤ gi for x ∈ x . The resultingrelaxation has optimum less than or equal to theoptimum of the original problem.

• If the relaxation is linear, LP technology can be used,to obtain ϕ and, with a different objective, narrowerbounds on the coordinates.

• Due to a technique proposed by Neumaier andShcherbina (and independently by Janssen), thebounds computed by the approximate LP solver canbe made rigorous by evaluating a simple expressioninvolving the duality gap with interval arithmetic.Infeasibility can also be similarly detected.

(slide 30)






Practicalities








(slide 31)






Practicalities








(slide 32)






Practicalities




Constraint Propagation

In simple terms, constraint propagation is based on

• solving a relation for a selected variable in terms of theother variables;

• using the bounds on the other variables (e.g. byevaluating the expression with interval arithmetic) todetermine new bounds for the selected variable;

• iterating the process (in some order and with somestopping criteria) to narrow bounds on as manyvariables as is practical.

(slide 33)






Practicalities









(slide 34)






Practicalities









(slide 35)






Practicalities




Outline




(slide 36)






Practicalities




Use of Interval TechniquesPitfalls

• Interval Dependency typically leads to overestimationin the ranges of ϕ and the constraints, if intervalevaluation is applied naively.

• For a given width of x , the width of N(f ; x , x̌) isproportional to the condition number of the Jacobimatrix of f , thus limiting the usefulness of intervalNewton methods.

• Constraint propagation is analogous to a fixed pointiteration that only converges when the iteration is acontraction.

• Linear relaxations can suffer from use of a poor LPsolver; they also depend on which relaxation is used.

(slide 37)






Practicalities









(slide 38)






Practicalities









(slide 39)






Practicalities









(slide 40)






Practicalities




Use of Interval TechniquesAvoiding Pitfalls

• Design Algorithms with non-interval computations toget approximations, then use interval computationonly a posteriori to obtain rigorous bounds.

• Arrange expressions carefully to avoid overestimationin interval computations.

• Pose nonlinear systems carefully, taking advantage ofproblem structure, to avoid ill-conditioning in intervalNewton methods.

• Choose implementation details in constraintpropagation carefully, according to experience.

• Design stopping criteria for each of these processesrationally.

• Use experience to choose the best relaxations.• Use experience to develop heuristics on when and

how often to apply individual algorithm components.

(slide 41)






Practicalities












(slide 42)






Practicalities












(slide 43)






Practicalities












(slide 44)






Practicalities












(slide 45)






Practicalities












(slide 46)






Practicalities












(slide 47)






Practicalities




An Algorithm for VerifiedUnconstrained Minimax

Ph.D. Work of Siripawn Winter

We examine the special global optimization problem wherethe objective is

ϕ(x) = max1≤i≤m

fi(x) or ϕ(x) = max1≤i≤m

|fi(x)|,

where the fi : x → R, x ⊂ IRn and the fi are C2 functions.

• The objective is non-differentiable.

• If Lemaréchal’s technique is used to convert thesystem to a constrained system, the resultingKuhn–Tucker equations are not well-suited to intervalNewton methods.

• We present some simple ways in which the structurecan be used to advantage.

(slide 48)






Practicalities




Outline




(slide 49)






Practicalities




Obtaining ϕ

A rigorous lower bound on the objective is obtained as

ϕ(x ) = maxi{migx∈x (fi(x))},

as is illustrated here for n = 1, m = 2. (If the problem is tominimize the maximum over i of |fi |, then we replace fi by|fi | in the above.)

b

|f2| = mig(f2)

|f1| = mig(f1) = ϕ(x)

x x0

1

(slide 50)






Practicalities




Outline




(slide 51)






Practicalities




A Gradient-Based EliminationProcess

The basic idea is:• Begin with a point x̌ ∈ x .• Construct a box x r with one corner at x̌ , by

proceeding out from x̌ in coordinate directions inwhich the function fi with

fi(x̌) = max1≤j≤m

fj(x̌)

(the dominant function) is increasing.• Define x r with a point x̂ diametrically opposite x̌ , such

that we are assured that fi(x) ≥ fi(x̌) for every x ∈ x r .• Use a width-scaled version of the complementation

process in §4.3.1 of Rigorous Global Search:Continuous Problems (Kluwer, 1996) to remove x r

from the search region.

(slide 52)






Practicalities








fj(x̌)





(slide 53)






Practicalities








fj(x̌)





(slide 54)






Practicalities








fj(x̌)





(slide 55)






Practicalities





Some Implementation Details

• The direction in coordinate k to proceed from x̌ ,1 ≤ j ≤ n is chosen according to the sign of ∂fi/∂xk .

• The distances rk that we go in direction k aredetermined simultaneously by solving a linearminimax problem based on the degree-2 Taylorpolynomial of fi centered at x̌ . In particular, wemaximize the minimum rk for which the quadraticmodel predicts that ∂fi/∂xk (x̂) = 0.

• Once x r is constructed, it is verified with intervalarithmetic that ϕ(x) ≥ ϕ(x̌).

(slide 56)






Practicalities









(slide 57)






Practicalities









(slide 58)






Practicalities





Illustration

x1 x1

x2

x2

(x̌1, x̌2)

r2

r1

b

b

1

(slide 59)






Practicalities




Additional Techniques andDetails

• We also expand x r in the opposite direction (on theother side of x̌ to the extent possible.

• We compute approximate optima at various points inthe algorithm, to get better ϕ and to eliminate boxesaround such local optima.

• The algorithm was implemented in Matlab usingroutines from the optimization toolbox for localapproximate solutions to the minimax problem and forsolutions of the linear minimax problems.

(slide 60)






Practicalities








(slide 61)






Practicalities








(slide 62)






Practicalities





Illustration of Algorithm Performance

We consider problem “OET5":

minx

max1≤i≤21

|fi(x)|,

fi(x) = x4 − (x1t2i + x2ti + x3)

2 −√

ti ,

ti = 0.25 + 0.75(i − 1)/20.

For illustration purposes, we fixed x3 and x4 at the valuesat an approximate solution. On the next illustration,• red boxes were rejected by comparing ϕ to ϕ,• yellow boxes were rejected from the gradient-based

elimination process,• green boxes are constructed about approximate

optima.

(slide 63)






Practicalities




The Branch and BoundProcess for OET5

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

x1

x 2

Removing boxes Iteration:0

xs

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

x1

x 2


xsxs

xr

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

x1

x 2


xsxs

xr

xr

xs

xr

xr

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

x1

x 2


xsxs

xr

xr

xr xs

xr

xr

xr xs

xr

xr

xr

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

x1

x 2


xsxs

xr

xr

xr

xr

xs

xr

xr

xr

xr

xs

xr

xr

xr

xr

xs

xr

xr

xr

xr

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2−2

−1.5

−1

−0.5

0

0.5

1

1.5

2

x1

x 2


xsxs

xr

xr

xr

xr

xs

xr

xr

xr

xr

xs

xr

xr

xr

xr

xs

xr

xr

xr

xr

xs

xr

xr

xr

xr xrxr

xr

xr

xr

xs

(slide 64)






Practicalities




Outline




(slide 65)






Practicalities




Kuhn–Tucker versus Fritz John

• The normalized Fritz John equations have a-prioribounds on the multipliers, whereas the Kuhn–Tuckerequations do not; the a priori bounds are anadvantage in interval Newton methods inautomatically verified computations.

• On the other hand, the Fritz John equations tend topresent numerical difficulties in practice that theKuhn–Tucker equations do not.

• We describe a relationship between these twosystems for minimax problems.

(slide 66)






Practicalities








(slide 67)






Practicalities








(slide 68)






Practicalities




Lemaréchal Reformulation

minimize x n+1

subject to pi = fi(x 1, . . . , x n)− x n+1 ≤ 0, i = 1, . . . , m,pi+m − fi(x 1, . . . , x n)− x n+1 ≤ 0, i = 1, . . . , m.

has normalized Fritz John system

F (x , u) =

u0∇ϕ(x) +2m∑

j=1

uj∇pj

u1p1...

u2mp2m

u0 +2m∑

j=1

uj − 1

= 0,

where 0 ≤ ui ≤ 1, i = 0, 1, . . . , 2m

(slide 69)






Practicalities




Fritz John and Kuhn–TuckerEquivalence

Symbolic manipulation of the Fritz John system gives

F (X ) =

0.5∇ϕ(x) +2m∑

j=1

uj∇pj

u1p1...

u2mp2m

= 0.

Thus, the Fritz John system is equivalent to theKuhn–Tucker system with Kuhn–Tucker multipliers (i.e.Lagrange multipliers) double the Fritz John multipliers.

(slide 70)






Practicalities




Prospects for Interval NewtonMethods in Minimax Problems

• To date, we have been unable to effectively exploitinterval Newton methods as filters to reduce the sizeof boxes.

• However, we have used interval Newton methodseffectively to prove existence and uniqueness ofsolutions in boxes constructed around approximatesolutions.

(slide 71)






Practicalities




an introduction and example of intervalneum/glopt/gicolag/talks/...local optimum. (see j. e. dennis...

Documents