non-linear optimization

Non-linear optimization

An overview, problems and a guide

w2

Optimization

)(min wEw

Unconstraint non-linear optimization

nE :

nw

E(w)

w1

Classes of Methods

Linear optimization Constraint <-> unconstraint Gradient based 1st order, 2nd order Genetic Algorithms,

Evolutionary Strategies Stochastic methods

(Simulated Annealing, Tabu Search, …)

Ellipsoid

Rosenbrock-function

Cross-Function

Canyon-function

Step-function

Performance criteria

Number of function evaluations Number of gradient calculation Time Number of fails Number of method params. Sensitivity of method params. Accuracy

Methods

Direct methods Successive variation Hooke-Jeeves

Gradient based methods Gradient decent Back-propagation Polak-Ribiere

Second order methods Newton-Raphson BFGS

Successive Variation

Successive Variation

Successive Variation

Successive Variation

Hooke-Jeeves

Hooke-Jeeves

Hooke-Jeeves

Gradient descent

)( )()()1( ttt wEww

Gradient descent

Gradient descent

Gradient Decent

Gradient descent

Gradient descent

Back-propagation

)1()()( )( ttt wwEw

)1()()1( ttt www

Gradient decent Momentum

Back-propagation

Back-propagationError E

Cycle

Conjugated gradients

cbwwAwwQ TT )(

Qn property

cbA nnn ,,

Beam search

))((min )()( tt wEwE

Polak-Ribiere

)1()()()( )( tttt dwEd

)1()()()1( tttt dww

)()(

)()]()([)1()1(

)()1()()(

tTt

tTttt

wEwE

wEwEwE

Beam search

Polak-Ribiere

Newton-method

Q1 property

)()( )(1)(2)()1( tttt wEwEww

BFGS

2)()(

)()()()(

)()(

)()()()()(

)]([

][][

][

][][tTt

TtttTt

tTt

TttTttt

w

wwv

w

vwwvB

)( )()()()()1( ttttt wEGww )()1()( ttt BGG

)1()()( ttt www

)()( )1()()( ttt wEwE)()1()()( tttt Gwv

BFGS

Comparison: Ellipsoid

0

5

10

15

20

25

30

SV HJ GD BP PR BFGS

Timen(E)n(grad E)

Comparison: Cross-Function

0

50

100

150

200

250Timen(E)n(grad E)

Comparison: Rosenbrock-Function

0

50

100

150

200

250

300

350Timen(E)n(grad E)

Comparison: Canyon-Function

0200400600800

100012001400160018002000

Timen(E)n(grad E)

n(E)=8983

Comparison: Step-Function

050

100150200250300350400450500

Timen(E)n(grad E)

n(E)=2487 n(E)=2448

Decision tree

#minima

MC / SA GA / ES Multi-start differentiable

aligned? elliptic?

channels?

#parameters

Complexity

Know

ledge

NM / LBFGS

#parameters coordinate axis

HJ / ROS ROS SV

PR / LBFGS BFGS

QP / RPROP BP

onefewsomemany

yesno

yesno yes

fewmany no yes

fewmany

no yes

flatcurved along axes

G / PR/ BFGS