wb1440 engineering optimization – concepts and applications engineering optimization concepts and...

WB1440 Engineering Optimization – Concepts and Applications

Engineering OptimizationConcepts and Applications

Fred van Keulen

Matthijs Langelaar

CLA H21.1

[email protected]

http://www.tudelft.nl/


Contents

● Optimization problem checking and simplification

● Model simplification



Model simplification

● Basic idea:

Expensivemodel

Optimizer

Cheapmodel

Optimizer

● Motivation:

– Replacement of expensive function, evaluated many times

– Interaction between different disciplines

– Estimation of derivatives

– Noise



Model simplification (2)

● Drawback: loss of accuracy

● Different ranges: local, mid-range, global

● Synonyms:

– Approximation models

– Metamodels

– Surrogate models

– Compact models

– Reduced order models

Extractinformation

Constructapproximation

Procedure:



Model simplification (3)

● Information extraction: linked to techniques from physical experiments: “plan of experiments” / DoE

● Many approaches! Covered here:

– Taylor series expansions

– Exact fitting

– Least squares fitting (response surface techniques)

– Kriging

– Reduced basis methods

– Briefly: neural nets, genetic programming, simplified physical models

● Crucial: purpose, range and level of detail



Taylor series expansions

● Approximation based on local information:

21 )(''!2

1)('

!1

1)()( hxfhxfxfhxf

0

)( )(!

1

n

nn hxfn

N

n

nn hxfn0

)( )(!

1Truncation error!

● Use of derivative information!

● Valid in neighbourhood of x

)()(!

1

0

)( NN

n

nn hohxfn



Taylor approximation example

1st order2nd order3rd order4th order5th order20th order

3

)5/cos(

51

52/

xx

ef

x

FunctionApproximation(x = 20)

x



Exact fitting (interpolation)

● # datapoints = # fitting parameters

● Every datapoint reproduced exactly

● Example:

xaaf 10

2

1

1

0

2

1

1

1

f

f

a

a

x

x

x1 x2

f2

f1



Exact fitting (2)

● Easy for intrinsically linear functions:

● No smoothing / filtering / noise reduction

● Danger of oscillations with high-order polynomials

n

iii fafafafaf

1221100

● Often used: polynomials, generalized polynomials:

1 2 1 2log( ) log log logm nf a bx x f a b m x n x



9th orderpolynomial

Oscillations

● Referred to as “Runge phenomenon”

● In practice: use order 6 or less

2

1

1 25x5th order9th order



Least squares fitting

● Less fitting parameters than datapoints

● Smoothing / filtering behaviour

● “Best fit”? Minimize sum of deviations:

N

iii xfxf

1

|)(~

)(|mina

● “Best fit”? Minimize sum of squared deviations:

N

iii xfxf

1

2)(

~)(min

a

x

f

f~



Least squares fitting (2)

● Choose fitting function linear in parameters ai :

)()()()()(~

221100 xfaxfaxfaxfaxf mm

NmNmNNN

m

m

m

Na

a

a

a

xfxfxfxf

xfxfxfxf

xfxfxfxf

xfxfxfxf

xf

xf

xf

xf

2

1

0

2

1

0

210

2222120

1121110

0020100

2

1

0

)()()()(

)()()()(

)()()()(

)()()()(

)(~

)(~

)(~

)(~

εMaf ~

● Short notation:



LS fitting (3)

● Minimize sum of squared errors:

(Optimization problem!)

MafMafεεa

~~

minTTL

MaMfMMafM0a

TTTL2

~2

~2

fMMMafMMaM~~ 1 TTTT



Polynomial LS fitting

● Polynomial of degree m:

mmxaxaxaxaxf 2

21

10

0)(~

mmm

mmm

m

m

m

ma

a

a

a

xxx

xxx

xxx

xxx

xf

xf

xf

xf

2

1

0

2

1

0

2

22

22

12

11

02

00

2

1

0

1

1

1

1

)(~

)(~

)(~

)(~

mii

ii

ii

i

mm

imi

mi

mi

miiii

miiii

miii

xf

xf

xf

f

a

a

a

a

xxxx

xxxx

xxxx

xxxN

~

~

~

~

22

1

0

221

2432

132

2



Polynomial LS example

-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1-0.2

0

0.2

0.4

0.6

0.8

1

1.2

samples

quadratic 6th degree



Multidimensional LS fitting

● Polynomial in multiple dimensions:

ii fa

xyayxayaxa

xyayaxa

yaxaayxf

29

28

37

36

52

42

3

210),(~

● Number of coefficients ai for quadratic polynomial in Rn:

Curse of dimensionality!

)2)(1(2

1 nnm



Fractional factorial design

Response surface

● Generate datapoints through sampling:

– Generate design points through Design of Experiments

– Evaluate responses

● Fit analytical model

● Check accuracy

2n full factorial designx1

x2

x3



Latin Hypercube Sampling (LHS)

● Popular method: LHS

● Based on idea of Latin square:

● Properties:

– Space-filling

– Any number of design points

– Intended for box-like domains

– Matlab: lhsdesign0 0.2 0.4 0.6 0.8 1

0

0.2

0.4

0.6

0.8

1



(LS) Fit quality indicators● Accuracy? More / fewer terms?

● Examine the residuals

– Small

– Random! xi

Okay: >0.6

Okay: >>1

2

2

2

~1ff

Ri

i

1

~

2

2

mN

mffF

i

i

● Statistical quality indicators:

– R2 correlation measure:

– F-ratio (signal to noise):



Nonlinear LS

● Linear LS: intrinsically linear functions (linear in ai):

paTxaxaxaxf 22

11

00)(

paTx xaxaeaxf 210 log)(

● Nonlinear LS: more complicated functions of ai:

● More difficult to fit! (Nonlinear optimization problem)

● Matlab: lsqnonlin

1)(

221

0

xaxa

xaxf



LS pitfalls

● Scattered data:

● Wrong choice of

basis functions:

x

f

x

f



Kriging● Named after D.C. Krige, mining engineer, 1951

● Statistical approach: correlation between neighbouring points

– Interpolation by weighted sum:

– Weights depend on distance

– Certain spatial correlationfunction is assumed(usually Gaussian)

N

iiii yxxxy

1

),()(



Kriging properties

● Kriging interpolation is “most likely” in some sense (based on assumptions of the method)

● Interpolation: no smoothing / filtering

● Many variations exist!

● Advantage: no need to assume form of interpolation function

● Fitting process more elaborate than LS procedure



Kriging example

● Results depend strongly on statistical assumptions and

method used:

Dataset z(x,y) Kriging interpolation



Reduced order model

● Idea: describing system in reduced basis:

– Example: structural dynamics

fwKwMfKuuM~~~

● Select small number of “modes” to build basis

– Example: eigenmodes



Reduced order model (2)

● Reduced basis:

Bwu kωωωB 21

Nk

iiiw

1

ωu

Bwu

fBf

KBBK

MBBM

T

T

T

~

~

~

fKBwwMB

fKuuM

● Reduced system equations:N1 Nk k1

fBKBwBwMBB TTT

kN NkNN

N1kN



Reduced order models● Many approaches!

– Selection of type and number of basis vectors

– Dealing with nonlinearity / multiple disciplines

● Active research topic

● No interpolation / fitting, but approximate modeling



Aerodynamic model

Example:Aircraft model

Structural model

Mass model



Neural nets



(input)

output

Neural nets

To determine internal neuron parameters, neural nets must be trained on data.

x f(x)



Neural net features● Versatile, can capture complex behavior

● Filtering, smoothing

● Many variations possible

– Network

– Number of neurons, layers

– Transfer functions

● Many training steps might be required (nonlinear optimization)

● Matlab: see e.g. nndtoc



Genetic programming

● Building mathematical functions using

evolution-like approach

● Approach good fit by crossover and

mutation of expressions

^2

+

/

x2

x3

x1

2

32

1

x

x

x



Genetic programming

● LS fitting with population of analytic expressions

● Selection / evolution rules

● Features:

– Can capture very complex

behavior

– Danger of artifacts /

overfitting

– Quite expensive procedure



Simplified physical models

● Goal: capture trends from underlying physics through

simpler model:

– Lumped / Analytic / Coarse

● Parameters fitted to “high-fidelity” data

Simplified model

Correctionfunction

x f(x)

● Refinement: correction function, parameter functions ...



Model simplification summary

Many different approaches:

– Local: Taylor series (needs derivatives)

– Interpolation (exact fit):

(Polynomial) fitting

Kriging

– Fitting: LS

– Approximate modeling: reduced order / simplified models

– Other: genetic programming, neural nets, etc



Response surfaces in optimization● Popular approach for

computationally expen-sive problems:

1. DoE, generate samples (expensive) in part of domain

2. Build response surface (cheap)

3. Perform optimization on response surface (cheap)

4. Update domain of interest, and repeat

Expensivemodel

Optimizer

Cheapmodel

Optimizer

● Additional advantage: smoothens noisy responses

● Easy to combine with parallel computing



Example: Multi-point

Approximation Method

Trust region

Design domain

Response surface

Sub-optimal point

Optimum

(Expensive) simulation


wb1440 engineering optimization – concepts and applications engineering optimization concepts and...

Documents