optimization with scilab · optimization with scilab ... genetic algorithm not 100% identical, but...

The free and open source software for numerical computation

Optimization with Scilab

June 29th 2011

Michaël BAUDIN & Vincent COUVERTScilab Consortium

The free and open source software for numerical computationThe free software for numerical computation

Part 1 - What's new in Scilab 5 ?

Focus on the Nelder-Mead component

Part 2 - Optimization in Scilab: Matlab® compatibility

Part 3 - OMD2 project: Scilab Platform Development

Part 4 - Conclusion

What is missing in Scilab ?

Outline


1. Introduction

1.1 What's in Scilab ?

1.2 What's new in Scilab v5 ?

1.3 What's new on Atoms ?

Part 1 – What's new in Scilab 5 ?

2. The Nelder-Mead Component

2.1 Introduction

2.2 The algorithm

2.3 Test cases

2.4 Conclusions


Objective Bounds Equality Inequalities Size GradientNeeded

Solver

Linear yes linear linear medium - linpro

Quadradic yes linear linear medium - quapro

Quadratic yes linear linear large - qpsolve

Quadratic yes linear linear medium - qld

Non-Linear yes large yes optim

Non-Linear small no fminsearch

Non-Linear yes small no neldermead

Non-Linear yes small no optim_ga

Non-Linear small no optim_sa

N.Li.Lea.Sq. large optional lsqrsolve

N.Li.Lea.Sq. large optional leastsq

Min-Max yes medium yes optim/nd

Multi-Obj yes small no optim_moga

Semi-Def. lin. (spectral) large no semidef

L.M.I. lin. (spectral) lin. (spectral) large no lmisolve



● Genetic Algorithms: nonlinear objective, bounds, global optimization

● Simulated Annealing: nonlinear objective, global optimization

● The Nelder-Mead component: nonlinear objective, unconstrained,

derivative-free, local optimization

● fminsearch: Matlab® compatible

1.2 What's new in Scilab 5 ?


● Optimization Solvers:

– Quapro: linear or quadratic objective, linear constraints (full matrices),

– SciIpopt: an interface to Ipopt. Nonlinear objective, nonlinear constraints (beta version),

– Fmincon: nonlinear objective, nonlinear constraints (alpha version) – Matlab® compatible,

– Other modules: Cobyla, Particle Swarm Optimization, Optkelley, …

● Test Problems:

– Uncprb: 35 unconstrained optimization problems,– AMPL: load AMPL problems into Scilab,– And also: CUTEr.

1.3 What's new on ATOMS ?


Outline

1. Introduction


1.2 What's new in Scilab v5 ?

1.3 What's new on Atoms ?

2. The Nelder-Mead Component

2.1 Introduction

2.2 The algorithm

2.3 Test cases

2.4 Conclusions


John Ashworth Nelder (8 October 1924 – 7 August 2010)Source: http://www.guardian.co.uk/technology/2010/sep/23/john-nelder-obituary

2. The Nelder-Mead Component2.1 Introduction


● We are interested in solving the unconstrained continuous optimization problem:

Minimize f(x)

with unbounded, real, multidimensional, variable x.

● A direct search algorithm:

– Uses only function values (no gradient needed),– Does not approximate the gradient.

● « A simplex method for function minimization », John Nelder, Roger Mead, Computer Journal, vol. 7, no 4, 1965, p. 308-313

2. The Nelder-Mead Algorithm2.1 Introduction


2. The Nelder-Mead Algorithm2.1 Introduction

Virginia Torczon (1989) writes:

"Margaret Wright has stated that over fifty percent of the calls received by the support group for the NAG software library concerned the version of the Nelder-Mead simplex algorithm to be found in that library."


A simplex: a set of n+1 vertices, in n dimensions.

In 2 dimensions.

In 3 dimensions.

2. The Nelder-Mead Algorithm2.2 The algorithm


● Steps in the Nelder-Mead algorithm

● Inputs:

– the n+1 vertices v(1), v(2), ..., v(n+1) of a nondegenerate simplex in n dimensions,

– the associated function values f(1),...,f(n+1), – the coefficients ρ (reflection), χ (expansion), γ (contraction),

and σ (shrinkage).● Standard Nelder-Mead: ρ=1, χ=2, γ=1/2, and σ=1/2.



f x1, x2=x12x2

2− x1 x2

function [ y , index ] = quadratic ( x , index )

y = x(1)^2 + x(2)^2 - x(1) * x(2);

endfunction

nm = neldermead_new ();

nm = neldermead_configure(nm,"-numberofvariables",2);

nm = neldermead_configure(nm,"-function",quadratic);

nm = neldermead_configure(nm,"-x0",[2 2]');

nm = neldermead_search(nm);

xopt = neldermead_get(nm,"-xopt");

nm = neldermead_destroy(nm);

2. The Nelder-Mead Algorithm2.3 Test cases


● Mc Kinnon, « Convergence of the neldermead simplex method to a nonstationary point ». SIAM J. on Optimization, 1998

● Failure by repeated inside contraction



● C. T. Kelley. « Detection and remediation of stagnation in the neldermead algorithm using a sufficient decrease condition » SIAM J. on Optimization, 1999

● Restart the algorithm...



● Some general facts:

– Memory requirement is O(n²)– Shrink steps are rare– Generally 1 or 2 function evaluations by iteration– Convergence is slow. Typical number of iterations is 100n,

where n is the number of dimensions– Hundreds of iterations are not rare– Convergence can be even slower when n > 10 (Han &

Neumann, 2006)– Restart the algorithm when in doubt for convergence (Kelley,

1999)– Convergence is guaranteed in 1 dimension (Lagarias et al.,

1999)

2. The Nelder-Mead Algorithm2.4 Conclusions


● We should not use this algorithm just because the gradient is not required:

– For example, if f is smooth, Quasi-Newton methods (optim) with numerical derivatives converge much faster.

● We may use this algorithm when:

– No other property of the problem can be used (e.g. non linear least squares can be solved by lsqrsolve),

– The objective function is nonsmooth or "noisy" (Kelley, 1999),– We do not need too much accuracy (Torzcon, 1989),– The number of parameters is moderate (Han & Neumann,

2006).

2. The Nelder-Mead Algorithm2.4 Conclusions


1. Introduction

2. Scilab Coverage

3. Overview

Part 3 – Optimization in Scilab: Matlab® compatibility


● Matlab® has many functions for optimization:

– Minimization,– Equation solving,– Datafitting and nonlinear least squares,– Global optimization.

● Scilab has often similar functions: let's see which ones.

Matlab is a registered trademark of The Mathworks, Inc.

1. Introduction


● For each Matlab® function, we search:

– Scilab function, if available,– Differences of features, differences of algorithms.

● (*) : Function will be reviewed at the end of the talk,

● For most functions, the match is not 100% identical,

– But some other functions can do it : which ones ?● We consider only Scilab Industrial Grade solvers:

– Scilab internal modules,– ATOMS modules,– Portables on all OS,– Well documented,– Tested.

1. Introduction


1. IntroductionMain differences

● Design:

– Matlab®: problem oriented (may be with several solvers),– Scilab: solver oriented (may be several solvers).

● Function arguments:

– Matlab® nearly always provides common options,– Scilab is less homogeneous.

● Management of the callbacks/extra-arguments:

– Matlab®: M-file or @

– Scilab: list


● Minimization:

– fminbnd Not 100% identical,

But optim can do it.

– fmincon ATOMS/fmincon (alpha version)– fminimax Not 100% identical,

but optim/''nd'' is designed for it.

– fminsearch fminsearch 90% identical in Scilab 5.3.2.

fminsearch 99% identical in 5.4.0

2. Scilab Coverage


– fminunc Not 100% identical,

but optim/''qn'' or optim/''gc'' are designed for it.

No sparsity pattern of Hessian in Scilab.

No PCG in optim: L-BFGS instead.

– linprog 100% for full matrices: karmarkar

ATOMS/quapro: linpro

No known solver for sparse matrices (*).

– quadprog 100% for full matrices: qpsolve, qp_solve

ATOMS/quapro: quapro

No known solver for sparse matrices.

2. Scilab Coverage


● Equation Solving:

– fsolve fsolve 100% for full matrices.

No known solver with sparse Jacobian (*).

– fzero No identical function.

But fsolve can do it.

● Least Squares (Curve Fitting):

– lsqcurvefit datafit

– lsqnonlin lsqrsolve (leastsq)

2. Scilab Coverage


● Global Optimization Toolbox:

● Genetic Algorithm Not 100% identical,

But optim_ga is built-in Scilab.

No linear equality and inequality in Scilab,

but bounds are managed.

● Simulated Annealing Not 100% identical,

But optim_sa is built-in Scilab

No bounds in Scilab SA, but user can

customize the neighbour function.

2. Scilab Coverage


Matlab Problem Scilab

bintprog Binary Integer Programming -

fgoalattain Multiobjective goal attainment -

fminbd Single-variable, on interval optim

fmincon Constrained, nonlinear, multivariable ATOMS/fmincon

fminimax Minimax, constrained optim/''nd''

fminsearch Unconstrained, multivariable, derivative-free fminsearch (100%)

fminunc Unconstrained, multivariable optim/''qn'',''gc''

fseminf Semi-infinitely constrained, multivariable, nonlinear

-

ktrlink Constrained or unconstrained, nonlinear, multivariable using Knitro

-

linprog Linear programming karmarkar, ATOMS/quapro

quadprog Quadratic programming qpsolve, ATOMS/quapro

3. Overview


Matlab Problem Scilab

fsolve Solve systems of nonlinear equations fsolve

fzero Root of continuous function of one variable -

lsqcurvefi t Nonlinear least squares curve fitting datafit

lsqlin Constrained linear least squares -

lsqnonlin Nonlinear least-squares (nonlinear data-fitting) lsqrsolve, leastsq

lsqnonneg Nonnegative least squares -

optimtool GUI to select solvers, options and run problems -

Global Search Solve GlobalSearch problems -

Multi Start Solve MultiStart problems -

Genetic Algorithm Genetic Algorithms optim_ga

Direct Search Pattern Search -

Simulated Annealing Simulated Annealing optim_sa

3. Overview


Part 4 -OMD2 project: Scilab Platform Development

1. Overview

2. Modules

2.1 Data Management

2.2 Modeling

2.3 Optimization


● OMD2 / CSDL projects collaboration

● Will be available on Scilab forge:

– http://forge.scilab.org/index.php/p/omd2/

– Private project up to first release.

● Scilab Optimization Platform:

– Batch mode (script edition, large scale execution),

– GUI mode (interactive edition, prototyping).

● Future Scilab external module available through ATOMS.

Overview

http://forge.scilab.org/index.php/p/omd2/


● Project management (Save & Load working data as HDF5 files)

● Wrappers:

– Scilab algorithms,– External tools,– Proactive.

● Mask complexity for users

● Modules:

– Data Management,– Modeling,– Optimization,– Visualization.

Main functionalities


● Factors / Parameters:

– Load existing Design Of Experiments (Isight .db files, …)– Generate Design Of Experiments:

● DoE generator wrappers (LHS, …),● DoE generator settings.

● Responses simulation using:

– External tool (openFOAM, Catia, CCM+, …),– Scilab function.

● 2-D visualization:

– Factor / Factor,– Response / Factor.

Data Management Module (1/2)


Data Management Module (2/2)


● Point selection:

– Learning points used for modeling,– Validation points used to validate model,– Bad points (simulation issue, …).

● Modeler:

– Selected among modeler wrappers (DACE, Lolimot, …),– Parameters configuration,– Multiple model management with best model user selection.

● Visualization:

– 2-D models,– Cross correlation,– Sensibility analysis.

Modeling module (1/2)


Modeling module (2/2)


● Responses coefficients values setting

● Optimizer:

– Selection among generic wrappers (optim, fmincon, genetic algorithms, …),

– Optimizer configuration,– Enable two chained optimizers.

● Visualization:

– Optimal point,– Paretos,– Robustness.

Optimization Module (1/2)


Optimization Module (2/2)


1. What is missing ?

2. Bibliography

Part 4 - Conclusion


● High Performance Optimization:

– Use BLAS/LAPACK within optim ?● Sparse Linear Programming:

– Update LIPSOL ?● Non Linear Programming:

– Improve fmincon ?● Non Linear Programming Test Cases:

– CUTEr requires a compiler on the test machine, – Connect the Hock-Schittkowski collection ?

Conclusion1. What is missing ?


● « Nelder-Mead User's Manual », Michaël Baudin, Consortium Scilab – DIGITEO, 2010

● « Optimization in Scilab », Baudin, Couvert, Steer, Consortium Scilab - DIGITEO – INRIA, 2010

● « Optimization with scilab, present and future », Michaël Baudin and Serge Steer, 2009 IEEE International Workshop on Open Source Software for Scientific Computation, pp.99-106, 18-20 Sept. 2009

● « Introduction to Optimization with Scilab », Michaël Baudin, Consortium Scilab – DIGITEO, 2010

● « Unconstrained Optimality Conditions with Scilab », Michaël Baudin, Consortium Scilab – DIGITEO, 2010

Conclusion2. Bibliography


Thanks for your attention

www.scilab.org


Some slides you won't see, unless you ask...

Extra-Slides


Some Historical References:

● Spendley, Hext, Himsworth (1962): fixed shape simplex algorithm

● Nelder, Mead (1965): variable shape algorithm

● Box (1965): simplex algo., with constraints

● O'Neill (1971): Fortran 77 implementation.

● Torczon (1989): Multi Directional Search.

● Mc Kinnon (1998): Counter examples of N-M.

● Lagarias, Reeds, Wright, Wright (1998): Proof of convergence in dimensions 1 and 2 for strictly convex functions.

● Han, Neumann (2006): More counter examples of N-M.

The Nelder-Mead Algorithm


In what softwares N-M can be found ?

● Matlab (fminsearch)

● NAG (E04CBF)

● Numerical Recipes (amoeba)

● IMSL (UMPOL)

● … and Scilab since v5.2.0 in 2009

● … and R after Sébastien Bihorel's port of Scilab's source code.



1. Sort by function value. Order the vertices: f(1) ≤ · · · ≤ f(n) ≤ f(n+1)

2. Calculate centroid. B = (v(1)+...+v(n))/n

3. Reflection. Compute R = (1+ρ)B − ρv(n+1) and evaluate f(R).

4. Expansion. If f(R)<f(1), compute E=(1+ρχ)B − ρχv(n+1) and evaluate f(E). If f(E)<f(R), accept E, else accept R and goto 1.

5. Accept R. If f(1) ≤ f(R) < f(n), accept R and goto 1.

6. Outside Contraction. If f(n)≤f(R)<f(n+1), compute Co=(1+ργ)B − ργv(n+1) and evaluate f(Co). If f(Co)<f(R), then accept Co and goto 1 else, goto 8.

7. Inside Contraction. If f(n+1)≤f(R), compute Ci=(1-γ)B +γv(n+1) and evaluate f(Ci). If f(Ci)<f(n+1), then accept Ci and goto 1 else, goto 8.

8. Shrink. Compute the points v(i)=v(1)+σ(v(i)-v(1)) and evaluate f(i)=f(x(i)), for i=2,3,...,n+1. Goto 1.



● Lagarias, Reeds, Wright, Wright (1998)

1. In dimension 1, the Nelder-Mead method converges to a minimizer, and convergence is eventually M-step linear, when the reflection parameter ρ = 1.

2. In dimension 2, the function values at all simplex vertices in the standard Nelder-Mead algorithm converge to the same value.

3. In dimension 2, the simplices in the standard Nelder-Mead algorithm have diameters converging to zero.

● Note that Result 3 does not implies that the simplices converge to a single point x*.



● Minimization:

– bintprog Solve binary integer programming problems– fgoalattain Solve multiobjective goal attainment problems– fminbnd Find minimum of single-variable function on

fixed interval

– fmincon Find minimum of constrained nonlinear

multivariable function

– fminimax Solve minimax constraint problem– fminsearch Find minimum of unconstrained multivariable

function using derivative-free method

What's in Matlab® ?


– fminunc Find minimum of unconstrained multivariable function

– fseminf Find minimum of semi-infinitely constrained

multivariable nonlinear function

– ktrlink Find minimum of constrained or unconstrained

nonlinear multivariable function using KNITRO

third-party libraries

– linprog Solve linear programming problems– quadprog Quadratic programming



● Equation Solving:

– fsolve Solve system of nonlinear equations– fzero Find root of continuous function of one variable

● Least Squares (Curve Fitting):

– lsqcurvefit Solve nonlinear curve-fitting (data-fitting) problems in least-squares sense

– lsqlin Solve constrained linear least-squares problems– lsqnonlin Solve nonlinear least-squares problems– lsqnonneg Solve nonnegative least-squares constraint

problem



● Utilities:

– optimtool GUI to select solver, optimization options, and

run problems

– optimget Optimization options values– optimset Create or edit optimization options structure



● Global Optimization Toolbox:

– GlobalSearch Create and solve GlobalSearch problems

– MultiStart Create and solve MultiStart problems– Genetic Algorithm Use genetic algorithm and Optimization

Tool, and modify genetic algorithm options

– Direct Search Use direct search and Optimization Tool,

and modify pattern search options

– Simulated Annealing Use simulated annealing and Optimization Tool, and modify simulated annealing options

What's in Matlab® toolboxes ?

optimization with scilab · optimization with scilab ... genetic algorithm not 100% identical, but...

Documents