Basics of Optimization

Meng-Sing LiouNASA Glenn Research Center

Spring Progress in Mathematican and Computational Studies on Science and Engineering ProblemsMay 3-5, 2014, National Taiwan University

1941: Aircraft Engine Research Laboratory of the National Advisory Committee for Aeronautics (NACA)1958: NASA Lewis Research Center begins making major contributions to manned space flight. 1999: Glenn Center, in honor of John H. Glenn, the first American to orbit Earth.

NASA Glenn Research Center, Lewis FieldCleveland, Ohio

Contents

• Introduction—why do optimization• What is optimization• Steps in an optimization task

– Formulation– Solution algorithms

• Aerospace design• Multidisciplinary analysis and design optimization

Applications of Optimization

Engineering, medicine, finance, air traffic routing, war, politics, …

What is Optimization

A process of searching for a set of decision variablesThat would minimize or maximize one or more

objective functions,subject to satisfying constraints

Decision variables: (x,y)Objective function: f(x,y)Constraint functions:Equality and inequality types

Properties of Optimization Problem

Mixed variables

Robust solutionMulti-modal

Reliable solution

Steps in an Optimization Task

Determine if need for optimization exists Formulate the Problem Single objective problem Multi-objective problem

Choose an optimization algorithm Criteria

Obtain solution(s) Local and global optimum Constrained and unconstrained optimum

Examine the result, reformulate and rerun if needed

Formulation of Optimization Problems

• Design variables– List any and every parameter related to the problem– Specify the type of each parameter (binary, discrete, continuous)– Choose a few of them as design variables– Use as few variables as possible, based on

• Knowledge on the problem, experience of the user• Sensitivity analysis

• Constraints– Inequality and equality types– Avoid equality constraints,

e.g. g(x)=3 2 < g(x) < 3.5– May be nonlinear

• Objective functions– Min or Max (convertible between them, duality)– Single or multiple objectives

A Typical Optimization Problem

• Decision (design) variables: x = (x1, x2 ,…, xn)• Constraints for feasible solutions

Min. f(x)s.t. gj(x) ≥ 0, j = 1,2,…,J

hk(x)=0, k = 1,2,…,Kxi

L ≤ xi ≤ xiU, i = 1,2,…,n

• Solutions are said to be optimal– Local and global optimum

Optimization Approaches

DeterministicBased on explicit gradient informationConverges to local optimumComputationally efficient (relatively)

StochasticMostly, biomimetic and driven by an inherent “search”

directionGenetic Algorithms (GAs)

• Based (loosely) on principles of natural evolution and survival of the fittest

Capable of finding global optimumSuitable for problems with multiple objective functions

Gradient (Descent) Method

Constrained Optimization

• Karush-Kuhn-Tucker (KKT) conditions for optimality– First-order necessary conditions– Convex search space, convex f:– KKT point is minimum

Himmelblau Problem

Genetic AlgorithmsHolland (1975) “Adaptation in Natural and Artificial Systems”

– The fittest survive (Charles Darwin)– Based on evolution of a population – “Computer programs that evolve in ways that resemble natural

selection can solve complex problems even their creators do not fully understand.”

Selection/pairing Crossover Mutation

For generation n,a population of chromosomes

chromosomes consists of genesfitness

constraints (environments, …) evolution of whole population

a set of states (solutions/designs)states are functions of design variables

objective functionconstraints (physical, …)iteration of whole set

15

NASA Rotor 67

Oyama, A., Liou, M.-S. and S. Obayashi, J. Propulsion & Power, Vol. 20, 612-619, 2004.

original optimized

16

NASA Rotor 67

0.8

0.82

0.84

0.86

0.88

0.9

0.92

0.94

30 31 32 33 34 35

rotor67optimized design

adia

batic

eff

icie

ncy

mass flow rate [kg/sec]

A 2% increase in efficiency!

0.007

0.0075

0.008

0.0085

0.009

0.0095

0.01

0 20 40 60 80 100

rotor67optimized designs

entro

py p

rodu

ctio

n

generation

Yielded milder flow separation, less losses

Real-world Engineering Project

• Motivation, ideas• Sketches (“back of envelope”)• Prototype• Analysis• Building• Testing

Wright Brothers’ Flight

Wilbur

First powered aircraft flown under pilot’s control, Dec. 17, 1903

40 ft span, 505 sq. ft airfoil surface750 pounds with pilot12 hp motor

Four flights850 ft max distance59 sec max duration

Orville

Design of a Complex Engineering System

Engineering system is typically Multidisciplinary Complex and nonlinear Unknown/uncertainty Hard to analyze Hard to build

Achieving a desirable performance requires Design and OptimizationMultiple objectives, constraints, uncertainties, Multilevel optimizationMultidisciplinary Decision making/formulation Expensive to compute

Modern Aerospace Design

• Computation-based design• Integration of high-fidelity modeling in multiple disciplines• Challenges

– Fidelity and applicability of modeling– Integration: efficiency, adaptability (models, computer systems, …), easiness to use– Verification and validation

• A new paradigm–Multidisciplinary Design Analysis and Optimization (MDAO), in which the synergistic effects of various interacting disciplines/phenomena are explored and exploited at every stage of the design process

• Challenges– Issues with optimization, such as choice of approaches, problem formulation, …– Configuration and its definition– Integration with CAD, mesh generation– Integration of different fidelity levels– Analysis of resulting designs– …

• Optimization

Complex System Development is Evolutionary

Boeing 707: 1958, M 0.82Airbus 340: 1993, M 0.86

• Design is often determined more by the problem formulation (parameterization, constraints) than by the optimization process.

• Design as parametric optimization can be problematic.

• To achieve more than evolutionary improvements requires advances in modeling, simulation, and multidisciplinary design.

• MDAO is a broadly applicable field, but has been pioneered in aerospace because of the maturity of modeling and

A typical MDAO problem

Bilevel Multi-obj. Opt.– Uncertainty– Coupling process– Computational cost– Other practicalities

Choose an MDO algorithm Solve for optimal solutions

Representative Analysis Architecture

• Flow structure and connectivity seems straightforward and logical

• But incredibly unwieldy—inefficient, prone to mistakes and hard to maintain

Tables taken from I. Kroo’s presentation given for NASA

Distributed Design • Decomposition of the design problem

– Design variables, constraints, disciplines, components, …

Collaborative optimization

Concurrent subspace optimization

Taken from I. Kroo, VKI lecture, 2004Numerical Propulsion System SimulationTaken from G. Follen

These variations are subject to satisfying the flow (Navier-Stokes) equations

Determine (adjoint variables) from the adjoint equations

=0

Adjoint Optimization Method

Form an adjoint system via the Lagrangian multiplier

Adjoint Optimization Method

Then the search gradient is obtained

And the step size for the design variables:Linear method

Nonlinear method is expensive as it requires Hessian

Concluding Remarks

• Optimization is common in practice, it is an essential step in engineering development, resulting in– Better design– Innovative solutions– Better understanding of the system

• Often theory is sound, but not easy to use in practice– Optimization algorithm is only a means to an end

• Choose an optimization algorithm better suited for the problem

• Ideas and formulation are the key to optimization

Any Questions?

Thank you for your attention!

No Free Lunch (NFL) Theorem

In the context of optimizationWolpert and McCardy (IEEE TEC, 1997)Algorithms A1 and A2For All possible problems FPerformances P1 and P2 using A1 and A2 for a fixed number of evaluations

P1 = P2

• NFL breaks down for a class of problems or algorithms• Find the best algorithm for a class of problems• Unimodal, multi-modal, quadratic etc.

Optimality Conditions for Unconstrained Minimum Points

• Single variable: df/dx = 0, d2f/dx2>0• General rule: Identify first non-zero derivative at

point x, say its order is n. If n is even, the sign of that derivative, + or –, determines minimum or maximum, respectively.

• Multiple variables: , Hessian matrix H(f) is positive definite for minimum– A matrix is + definite, if all eigenvalues are positive