solving optimization problems with matlab · perform exact computations using familiar matlab...

1© 2018 The MathWorks, Inc.

Solving Optimization Problems with MATLAB

2

Introduction

Least-squares minimization

Nonlinear optimization

Mixed-integer programming

Global optimization

Topics

3

Optimization Problems

Minimize Risk

Maximize Profits

Maximize Fuel Efficiency

4

Design Process

Initial

Design

Variables

System

Modify

Design

Variables

Optimal

DesignObjectives

met?

No

Yes

5

Manually (trial-and-error or iteratively)

Why use Optimization?

Initial

Guess

6

Automatically (using optimization techniques)

Initial

Guess

Why use Optimization?

7

Why use Optimization?

Finding better (optimal) designs and decisions

Faster design and decision evaluations

Automate routine decisions

Useful for trade-off analysis

Non-intuitive designs may be found

Antenna Design Using Genetic Algorithmhttp://ic.arc.nasa.gov/projects/esg/research/antenna.htm

8

Introduction

Least-squares minimization

Nonlinear optimization

Mixed-integer programming

Global optimization

Topics

9

Curve Fitting Demo

Given some data:

Fit a curve of the form:

teccty 21

)(

t = [0 .3 .8 1.1 1.6 2.3];

y = [.82 .72 .63 .60 .55 .50];

10

How to solve?

Ecc

cey

eccty

t

t

2

1

21

1

)(

As a linear system of equations:

2

2min yEc

Ecy

c

Can’t solve this exactly

(6 eqns, 2 unknowns)

An optimization problem!

2

1

3.2

6.1

1.1

8.0

3.0

0

1

1

1

1

1

1

50.0

55.0

60.0

63.0

72.0

82.0

c

c

e

e

e

e

e

e

11

Introduction

Least-squares minimization

Nonlinear optimization

Mixed-integer programming

Global optimization

Topics

12

Nonlinear Optimization

min𝑥

log 1 + 𝑥1 −4

3

2

+ 3 𝑥1 + 𝑥2 − 𝑥13 2

13

Nonlinear Optimization - Modeling Gantry Crane

Determine acceleration profile that

minimizes payload swing

s25s4

s20s1

s20s1

:sConstraint

2

1

21

•

•

•

•

f

p

p

ppf

t

t

t

ttt

14

Symbolic Math Toolbox

Perform exact computations using familiar

MATLAB syntax in MATLAB

– Integration

– Differentiation

– Equation solving

– Transformations

– Simplification

– Unit conversion

– Variable precision arithmetic

Results in typeset math in Live Editor

Integrates with MATLAB, Simulink, Simscape

15

Introduction

Least-squares minimization

Nonlinear optimization

Mixed-integer programming

Global optimization

Topics

16

Mixed-Integer Programming

Many things exist in discrete amounts:

– Shares of stock

– Number of cars a factory produces

– Number of cows on a farm

Often have binary decisions:

– On/off

– Buy/don’t buy

Mixed-integer linear programming:

– Solve optimization problem while enforcing that certain variables need to be integer

17

Continuous and integer variables

𝑥1 ∈ 0, 100 𝑥2 ∈ 1,2,3,4,5

Linear objective and constraints

min𝑥

−𝑥1 − 2𝑥2

ቊ𝑥1 + 4𝑥2 ≤ 20𝑥1 + 𝑥2 = 10

such that

Mixed-Integer Linear Programming

18

Optimize Gift Card Spending

Problem:

Given gift cards to different stores and a shopping list of desired purchases, decide how to

spend the gift cards to use as much of the gift card money as possible.

Constraints:

You cannot overspend the gift card.

You can purchase one of any item, and must purchase one of a specific item.

19

Traveling Salesman Problem

Problem

How to find the shortest path through a series of points?

Solution

Calculate distances between all combinations of points

Solve an optimization problem where variables correspond to trips between two points

1

11

0

1

1

0

0

00

20

Introduction

Least-squares minimization

Nonlinear optimization

Mixed-integer programming

Global optimization

Topics

21

Example Global Optimization Problems

Why does fmincon have a hard time finding the

function minimum?

0 5 10

-10

-5

0

5

10

x

Starting at 10

0 5 10

-10

-5

0

5

10

x

Starting at 8

0 5 10

-10

-5

0

5

10

x

Starting at 6

x s

in(x

) +

x c

os(2

x)

0 5 10

-10

-5

0

5

10

x

Starting at 3

0 5 10

-10

-5

0

5

10

x

Starting at 1

0 5 10

-10

-5

0

5

10

x

Starting at 0

x s

in(x

) +

x c

os(2

x)

22

Initial

Guess

Example Global Optimization Problems

Why didn’t fminunc find the maximum efficiency?

23

Example Global Optimization Problems

Why didn’t nonlinear regression find a good fit?

0 20 40 60 80 100 1200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

t

c

c=b1e-b

4t+b

2e-b

5t+b

3e-b

6t

24

Global Optimization

Goal:

Want to find the lowest/largest value of

the nonlinear function that has many local

minima/maxima

Problem:

Traditional solvers often return one of the

local minima (not the global)

Solution:

A solver that locates globally optimal

solutions

Global Minimum at [0 0]

Rastrigin’s Function

25

Global Optimization Solvers Covered Today

Multi Start and Global Search

Pattern Search

Genetic Algorithm

Surrogate Optimization

Particle Swarm

Simulated Annealing

26

MultiStart Demo – Nonlinear Regression

lsqcurvefit solution MultiStart solution

27

MULTISTART

28

What is MultiStart?

Run a local solver from each set

of start points

Option to filter starting points

based on feasibility

Supports parallel computing

29

MultiStart Demo – Peaks Function

30

GLOBAL SEARCH

31

What is GlobalSearch?

Multistart heuristic algorithm

Calls fmincon from multiple

start points to try and find a

global minimum

Filters/removes non-promising

start points

32

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview Schematic Problem

Peaks function

Three minima

Green, z = -0.065

Red, z= -3.05

Blue, z = -6.55

x

y

33

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 0Run from specified x0

x

y

34

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 1

3

6

0

00

4

0

-2

x

y

35

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 1

3

6

0

00

4

0

-2

x

y

36

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 1

x

y

37

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

x

y

38

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

x

y

39

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

x

y

40

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

x

y

41

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

6

Current penalty

threshold value : 4

x

y

42

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

x

y

43

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

Current penalty

threshold value : 4

-3

x

y

44

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2Expand basin of attraction if minimum already found

Current penalty threshold value : 2

-0.1

x

y Basins can overlap

45

GlobalSearch Demo – Peaks Function

46

PATTERN SEARCH

(DIRECT SEARCH)

47

What is Pattern Search?

An approach that uses a pattern

of search directions around the

existing points

Expands/contracts around the

current point when a solution is

not found

Does not rely on gradients: works

on smooth and nonsmooth

problems

48

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 1Run from specified x0

x

y

3

49

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 1Apply pattern vector, poll new points for improvement

x

y

3

Mesh size = 1

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

0_*_ xvectorpatternsizemeshPnew

0]0,1[*1 x1.6

0.4

4.6

2.8

First poll successful

Complete Poll (not default)

50

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 2

x

y

3

Mesh size = 2

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8

Complete Poll

51

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 3

x

y

3

Mesh size = 4

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8

52

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 4

x

y

3

Mesh size = 4*0.5 = 2

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8

53

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration NContinue expansion/contraction until convergence…

x

y

31.6

0.4

4.6

2.8

-4

0.3-2.8

-6.5

54

Pattern Search – Peaks Function

55

Pattern Search Climbs Mount Washington

56

GENETIC ALGORITHM

57

What is a Genetic Algorithm?

Uses concepts from evolutionary

biology

Start with an initial generation of

candidate solutions that are tested

against the objective function

Subsequent generations evolve

from the 1st through selection,

crossover and mutation

58

How Evolution Works – Binary Case

Selection

– Retain the best performing bit strings from one generation to the next. Favor these for

reproduction

– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]

– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]

Crossover

– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]

– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]

– child = [ 1 0 0 0 0 1 1 0 1 0 ]

Mutation

– parent = [ 1 0 1 0 0 1 1 0 0 0 ]

– child = [ 0 1 0 1 0 1 0 0 0 1 ]

59

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 1Evaluate initial population

x

y

60

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 1Select a few good solutions for reproduction

x

y

61

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 2Generate new population and evaluate

x

y

62

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 2

x

y

63

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 3

x

y

64

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 3

x

y

65

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration NContinue process until stopping criteria are met

x

y

Solution found

66

Genetic Algorithm – Peaks Function

67

Genetic Algorithm – Integer Constraints

Mixed Integer Optimization

s.t. some x are integers

Examples

Only certain sizes of components

available

Can only purchase whole shares of stock

)(min xfx

68

Application: Circuit Component Selection

6 components to size

Only certain sizes available

Objective:

– Match Voltage vs. Temperature

curve

Thermistor Circuit

300 Ω

330 Ω

360 Ω

…

180k Ω

200k Ω

220k Ω

Thermistors:

Resistance varies

nonlinearly with

temperature

?

𝑅𝑇𝐻 =𝑅𝑇𝐻,𝑁𝑜𝑚

𝑒𝛽𝑇−𝑇𝑁𝑜𝑚𝑇∗𝑇𝑁𝑜𝑚

69

SURROGATE OPTIMIZATION

70

What is Surrogate Optimization?

An approach that creates and

optimizes a surrogate of the

function

Searches randomly to explore

and adaptively to refine

Does not rely on gradients: works

on smooth and nonsmooth

problems

71

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Surrogate Optimization OverviewConstruction phase – evaluate at random points to construct surrogate

x

y

Random points

Incumbent

72

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Surrogate Optimization OverviewSearch phase – minimize merit function of surrogate and point spread

x

y

Random points

Adaptive points

Incumbent

73

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Surrogate Optimization OverviewReset – Repeat construction and search phases

x

y

Random points

Adaptive points

Incumbent

74

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Surrogate Optimization OverviewContinue process until stopping criteria are met

x

y

Random points

Adaptive points

Solution found

Incumbent

75

Surrogateopt Demo – Peaks Function

76

PARTICLE SWARM

77

What is Particle Swarm Optimization?

A collection of particles move

throughout the region

Particles have velocity and are

affected by the other particles in

the swarm

Does not rely on gradients: works

on smooth and nonsmooth

problems

78

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration 1Initialize particle locations and velocities, evaluate all locations

x

y

79

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration NUpdate velocities for each particle

x

yPrevious

Velocity

Best Location

for this Particle

Best Location for

Neighbor Particles

80

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration NUpdate velocities for each particle

x

y

New

Velocity

81

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration NMove particles based on new velocities

x

y

82

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration NContinue swarming until convergence

x

y

83

Particle Swarm – Peaks Function

84

SIMULATED ANNEALING

85

What is Simulated Annealing?

A probabilistic metaheuristic

approach based upon the physical

process of annealing in

metallurgy.

Controlled cooling of a metal

allows atoms to realign from a

random higher energy state to an

ordered crystalline (globally) lower

energy state

86

Simulated Annealing Overview – Iteration 1Run from specified x0

x

y

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

0.9

87

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1

3

x

y 0.9

Possible New Points:

Standard Normal N(0,1) * Temperature

Temperature = 1

88

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1

3

x

y 0.9

Temperature = 1

11.01

1/)9.03(

Taccepte

P

89

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1

3

x

y 0.9

Temperature = 1

0.3

90

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1

3

x

y 0.9

Temperature = 1

0.3

91

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 2

3

x

y 0.9

Temperature = 1

0.3

92

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 2

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

93

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N-1

3

x

y 0.9

Temperature = 0.1

0.3

-1.2

-3

94

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration NReannealing

3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

95

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration NReannealing

3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

27.01

1/))3(2(

Taccepte

P

96

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration NReannealing

3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

97

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N+1

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

98

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N+1

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

99

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration …

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

-6.5

100

Simulated Annealing – Peaks Function

101

Global Optimization Toolbox Solvers

GlobalSearch, MultiStart

– Well suited for smooth objective and constraints

– Relies on gradient calculations

– Return the location of local and global minima

ga, simulannealbnd, particleswarm

– Many function evaluations to sample the search space

– Work on both smooth and nonsmooth problems

patternsearch, surrogateopt

– Fewer function evaluations than ga, simulannealbnd, particlewarm

– Work on both smooth and nonsmooth problems

102

Optimization Toolbox Solvers

fmincon, fminbnd, fminunc, fgoalattain, fminimax

– Nonlinear constraints and objectives

– Gradient-based methods for smooth objectives and constraints

quadprog, linprog

– Linear constraints and quadratic or linear objective, respectively

intlinprog

– Linear constraints and objective and integer variables

lsqlin, lsqnonneg

– Constrained linear least squares

lsqnonlin, lsqcurvefit

– Nonlinear least squares

fsolve

– Nonlinear equations

103

Speeding-up with Parallel Computing

Global Optimization Toolbox– patternsearch, surrogateopt, ga,

gamultiobj, particleswarm: Points

evaluated in parallel at each iteration

– MultiStart: Start points evaluated in parallel

Optimization Toolbox

– fmincon, fminunc, fminimax,

fgoalattain, fsolve, lsqcurvefit,

lsqnonlin: Parallel evaluation of objective function

for finite differences

Parallel Computing can also be used in the

Objective Function– parfor

104

Speed up parallel applications

Take advantage of GPUs

Prototype code for your cluster

Parallel Computing Toolbox for the Desktop

Simulink, Blocksets,

and Other Toolboxes

Local

Desktop Computer

MATLAB

……

105

Scale Up to Clusters and Clouds

Cluster

Scheduler

Computer Cluster

…

…

…

…

…

…

… … …

Simulink, Blocksets,

and Other Toolboxes

Local

Desktop Computer

MATLAB

……

106

Learn More about Optimization with MATLAB

Recorded webinar: Optimization in

MATLAB: An Introduction to Quadratic

Programming

Optimization Toolbox Web demo:

Finding an Optimal Path using MATLAB

and Optimization Toolbox

MATLAB Digest: Improving

Optimization Performance with Parallel

Computing

MATLAB Digest: Using Symbolic

Gradients for Optimization

Recorded webinar: Linear and Mixed

Integer Linear Programming in MATLAB

Recorded webinar: Optimization in

MATLAB for Financial Applications

1 2 3 4 50

500

1000

1500

2000

2500

3000

Bonds

# P

urc

ha

se

d

Cash Flow Matching Example

107

Key Takeaways

Solve a wide variety of optimization problems in MATLAB

– Linear and Nonlinear

– Continuous and mixed-integer

– Smooth and Nonsmooth

Find better solutions to multiple minima and non-smooth problems using global

optimization

Use symbolic math for setting up problems and automatically calculating gradients

Using parallel computing to speed up optimization problems

108

MATLAB Answers

Blogs

Cody

File Exchange

and more…

Answers Blogs

ThingSpeak

Every month, over 2 million MATLAB & Simulink users visit MATLAB Central to get questions answered,

download code and improve programming skills.

MATLAB Central Community

MATLAB Answers: Q&A forum; most questions get

answered in only 60 minutes

File Exchange: Download code from a huge repository of

free code including tens of thousands of open source

community files

Cody: Sharpen programming skills while having fun

Blogs: Get the inside view from Engineers who build

and support MATLAB & Simulink

ThingSpeak: Explore IoT Data

And more for you to explore…

Learn ConnectContribute

109

Training: Optimization Techniques in MATLAB

After this 1-day course you will be able to:

Write objective function files and

pass extra parameters

Add different types of constraints

Select an appropriate solver and algorithm

Interpret the output from the solver and

diagnose the progress of an optimization

110© 2018 The MathWorks, Inc.