An Introduction to Optimization Theory

Outline

Introduction

Unconstrained optimization problem

Constrained optimization problem

Introduction

Mathematically speaking, optimization is the minimization of a objective function subject to constraints on its variables. Mathematically, we have

set) (Feasible choices possible all ofset theis

function objective is )( where

subject to

)(minarg

xf

x

xfx

Introduction

Introduction-Linear regression

Introduction-Battery charger

Unconstrained optimization problem

Definition for unconstrained optimization problem:

Unconstrained optimization problem

Gradient descent algorithm

Gradient descent algorithm

Gradient descent algorithm may be trapped into the local extreme instead of the global extreme

Gradient descent algorithm

Methodology for choosing suitable step size αk ---- Steepest descent algorithm

Gradient descent algorithm

Gradient descent algorithm

Steepest descent algorithm with quadratic cost function:

Gradient descent algorithm

bQxxfg kkk )()()( )(Update equation:

Newton method

Summary for Newton method

Newton method

Newton method

Procedure for Newton method

Quasi-Newton method

Quasi-Newton method

What properties of F(x(k))-1 should it mimic ?

1. Hk should be a symmetric matrix

2. Hk should with secant property

)1()(

)1()()( )(')(')(''

kk

kkk

xx

xfxfxf

Quasi-Newton method

Typical approaches for Quasi-Newton method

1. Rank-one formula

2. DFP algorithm

3. BFGS algorithm (L-BFGS , L indicates limited-memory)

Constrained optimization problem

Definition for constrained optimization problem

Problems with equality constraints ---- Lagrange multiplier

Problems with equality constraints ---- Lagrange multiplier

Problems with equality constraints ---- Lagrange multiplier

Problems with equality constraints ---- Lagrange multiplier

Suppose x* is a local minimizer

Karush-Kuhn-Tucker condition (KKT)

From now on, we will consider the following problem

Karush-Kuhn-Tucker condition (KKT)

Note that:

Image statistics & Image enhancement

Illustration for gradient descent with projection

operator Projection:

][][ :equation Projection

:equationdescent Gradient )()1()1(

)()1(

kkkkk

p

kkkk

dxxx

dxx

Constrained set Ω

Initial solution

Projection

Useful Matlab introductions for optimization

Useful instructions included in Matlab for optimization

1. fminunc: Solver for unconstrained optimization problems

2. fmincon: Solver for constrained optimization problems

3. linprog: Solver for linear programming problems

4. quadprog: Solver for quadratic programming problems