non-linear optimization
DESCRIPTION
Non-linear optimization. An overview, problems and a guide. Optimization. Unconstraint non-linear optimization. E( w ). w 2. w 1. Classes of Methods. Linear optimization Constraint unconstraint Gradient based 1 st order, 2 nd order Genetic Algorithms, Evolutionary Strategies - PowerPoint PPT PresentationTRANSCRIPT
Non-linear optimization
An overview, problems and a guide
w2
Optimization
)(min wEw
Unconstraint non-linear optimization
nE :
nw
E(w)
w1
Classes of Methods
Linear optimization Constraint <-> unconstraint Gradient based 1st order, 2nd order Genetic Algorithms,
Evolutionary Strategies Stochastic methods
(Simulated Annealing, Tabu Search, …)
Ellipsoid
Rosenbrock-function
Cross-Function
Canyon-function
Step-function
Performance criteria
Number of function evaluations Number of gradient calculation Time Number of fails Number of method params. Sensitivity of method params. Accuracy
Methods
Direct methods Successive variation Hooke-Jeeves
Gradient based methods Gradient decent Back-propagation Polak-Ribiere
Second order methods Newton-Raphson BFGS
Successive Variation
Successive Variation
Successive Variation
Successive Variation
Hooke-Jeeves
Hooke-Jeeves
Hooke-Jeeves
Gradient descent
)( )()()1( ttt wEww
Gradient descent
Gradient descent
Gradient Decent
Gradient descent
Gradient descent
Back-propagation
)1()()( )( ttt wwEw
)1()()1( ttt www
Gradient decent Momentum
Back-propagation
Back-propagationError E
Cycle
Conjugated gradients
cbwwAwwQ TT )(
Qn property
cbA nnn ,,
Beam search
))((min )()( tt wEwE
Polak-Ribiere
)1()()()( )( tttt dwEd
)1()()()1( tttt dww
)()(
)()]()([)1()1(
)()1()()(
tTt
tTttt
wEwE
wEwEwE
Beam search
Polak-Ribiere
Newton-method
Q1 property
)()( )(1)(2)()1( tttt wEwEww
BFGS
2)()(
)()()()(
)()(
)()()()()(
)]([
][][
][
][][tTt
TtttTt
tTt
TttTttt
w
wwv
w
vwwvB
)( )()()()()1( ttttt wEGww )()1()( ttt BGG
)1()()( ttt www
)()( )1()()( ttt wEwE)()1()()( tttt Gwv
BFGS
Comparison: Ellipsoid
0
5
10
15
20
25
30
SV HJ GD BP PR BFGS
Timen(E)n(grad E)
Comparison: Cross-Function
0
50
100
150
200
250Timen(E)n(grad E)
Comparison: Rosenbrock-Function
0
50
100
150
200
250
300
350Timen(E)n(grad E)
Comparison: Canyon-Function
0200400600800
100012001400160018002000
Timen(E)n(grad E)
n(E)=8983
Comparison: Step-Function
050
100150200250300350400450500
Timen(E)n(grad E)
n(E)=2487 n(E)=2448
Decision tree
#minima
MC / SA GA / ES Multi-start differentiable
aligned? elliptic?
channels?
#parameters
Complexity
Know
ledge
NM / LBFGS
#parameters coordinate axis
HJ / ROS ROS SV
PR / LBFGS BFGS
QP / RPROP BP
onefewsomemany
yesno
yesno yes
fewmany no yes
fewmany
no yes
flatcurved along axes
G / PR/ BFGS