mathematical programming
DESCRIPTION
Mathematical Programming. ECE 665 Professor Maciej Ciesielski By DFG. Overview. Classical optimization deals with unconstrained optimization problems. For the minimization problem: Min f(x) subject to g(x) = 0 - PowerPoint PPT PresentationTRANSCRIPT
Computer Algorithms
Mathematical Programming
ECE 665Professor Maciej Ciesielski
By DFG
2Department of Electrical Engineering - UMASS
Overview Classical optimization deals with unconstrained
optimization problems.
For the minimization problem: Min f(x) subject to g(x) = 0
The solution is to move the constraints to the function and solve the unconstrained optimization problem.
• Penalty function method: replace f(x) byF(x) = f(x)+ M( g(x))2, M >> 0
• Lagrangian multipliers: replace f(x) byL(x,λ) = f(x) + Σ λi gi(x)At the optimum point, x*, L(x*, λ) = f(x*)
3Department of Electrical Engineering - UMASS
Outline
Mathematical programming paradigm Feasible solutions?
• Convexity• Concave• Concept example
Geometric solution Fundamental theorem of LP Simplex
• Example LP Modifications
4Department of Electrical Engineering - UMASS
Mathematical Programming Paradigm
A mathematical program is an optimization problem of the form:
• Maximize (or Minimize) f(x)
subject to:
• g(x) = 0• h(x) ≥ 0
where x = [x1,...xn] is a subset of Rn, the functions g and h are called constraints, and f is called the objective function.
MathematicalProgrammingparadigm
5Department of Electrical Engineering - UMASS
The Nature of Mathematical Programming
Of course there are some forms that deviate from this paradigm.
Important examples are:
Fuzzy mathematical program Goal program Multiple objectives Randomized program Stochastic program
Therefore it is typically a modeling issue to find an standard form to solve the mathematical programming.
6Department of Electrical Engineering - UMASS
When is it feasible?
A point x is feasible if it is in X and satisfies the constraints:
• g(x) = 0 and h(x) ≥ 0.
A point x* is optimal if it is feasible and if the value of the objective function is:
• For Maximizationnot less than that of any other feasible solution: f(x*) ≥ f(x) for all feasible x.
• For Minimizationnot greater than that of any other feasible solution: f(x*) ≤ f(x) for all feasible x.
Min or Max f(x)
Subject to: g(x) = 0h(x) ≥ 0
Mathematical ProgrammingParadigm
7Department of Electrical Engineering - UMASS
Convexity
To obtain global optimum, it is important that the constraint set
{x | g(x) = 0, h(x) ≥ 0}
and the function f(x) be convex.
If the function is convex over a convex constraint set, then local minimun is also a globa1 minimum
Min or Max f(x)
Subject to: g(x) = 0h(x) ≥ 0
Mathematical ProgrammingParadigm
8Department of Electrical Engineering - UMASS
Convexity
Set convexity (If any two points are in the set, so is their line segment)
Convex set non-convex set
Function convexity
Convex function non-convex function
???
9Department of Electrical Engineering - UMASS
Convexity
A function f:X–>R is said to be convex if:
Its epigraph is convex.
X is a convex set, and for x,y X and a[0, 1]:
f(ax + (1-a)y) ≤ af(x) + (1-a)f(y)
Epigraph: region on or above the graph
10Department of Electrical Engineering - UMASS
Convexity
If the function is convex over a convex constraint set, then local minimun is also a globa1 minimum
11Department of Electrical Engineering - UMASS
Concave
It is define for maximization, and it is the negative of convexity.
A function f is concave if its hypograph is convex
Hypograph: region below the graph
12Department of Electrical Engineering - UMASS
Concave
If the function is concave over a convex constraint set, then local maximun is also a globa1 maximum
13Department of Electrical Engineering - UMASS
Concept Example
Can we minimize based on the local minimum?
f(x) = xTQx
Subject to:
Ax ≥ bx ≥ 0
Where Q is an n x n symmetric matrix, A is a constraint matrix, and b is a constraint vector.
QuadraticProgramming
14Department of Electrical Engineering - UMASS
Concept Example If xTQx > 0 for every x 0.
The matrix Q is positive definite. In which case its eigenvalues are non-negative, and the quadratic form is a convex function. Thus f is convex.
As Q is a symmetric matrix, these eingenvalues will be reals.
Ax ≥ b defines a convex constraint set, as its solution is a Rk space. (A convex polyhedron)
f and the constraint set are convex, therefore, finding the local minimum also finds the global minimum.
15Department of Electrical Engineering - UMASS
Linear Programming (LP) The goal of Linear programming is to:
(Max)Minimize: CTxSubject to: Ax ≥ b
x ≥ 0
Where CT is a coefficient vector for f, A is a constraint matrix, and b es a constraint vector.
The constraint set: {x | Ax ≥ b} is a convex polyhedron, and f is linear so it is convex.
Therefore, LP has convexity, and the local min/max is the global min/max
16Department of Electrical Engineering - UMASS
LP Example (Problem Enunciation)
Optimization problem:
2 types of products are made by the factory 3 machines are needed to make each product
Constraints: Machines are available for a limited time during one production session:
time on A ≤ 50 hours time on B ≤ 35 hours time on C ≤ 80 hours
17Department of Electrical Engineering - UMASS
LP Example (Problem Enunciation)
To complete the product, each lot must be processed by all three machines for a certain number of hours Required manufacturing time
Machines A B C 1
Products 2Time Constraint: 50 35 80
10 5 55 5 15
18Department of Electrical Engineering - UMASS
LP Example (Problem Enunciation)
Profit:
Product 1 can be sold for $100 per lot Product 2 can be sold for $80 per lot
Optimization Problem: Design the production schedule which
maximizes the profit of both products manufactured during one production session.
19Department of Electrical Engineering - UMASS
LP Example (Problem Formulation)
Formulation x1 = number of lots of product 1
x2 = number of lots of product 2
max f (x) = 100x1 + 80x2
10x1 + 5x2 ≤ 505x1 + 5x2 ≤ 355x1 + 15x2 ≤ 80
20Department of Electrical Engineering - UMASS
LP Example (Geometric Solution)
Geometric solution: Family of curves, constant for x1, x2
F(x) = 100 x1 + 80 x2 = 800 (const)
(a) 10x1 + 5x2 50 (b) 5x1 + 5x2 35(c) 5x1 + 15x2 80
a bx1
x2
10
5
15105 c
P2
P13
4
P0
F(x) = 800
Sol:X1 = 3X2 = 4
Extremepoint
21Department of Electrical Engineering - UMASS
LP Example (Standard Linear Program Approach)
Convert the problem to a standard Linear Program (LP):
Max f(x) = 100x1 + 8Ox2 + 0s1 + 0s2 + 0s3
10x1 + 5x2 + s1 = 50 Subject to: 5x1 + 5x2 + s2 = 35 5x1 + 15x2+ s3 = 80
Basis
New Matrix A
Why theBasis??
22Department of Electrical Engineering - UMASS
Basis and Basic solution
Why the Basis?• Because it finds a basic solution
Given a system of equalities: Ax = b
where: x = n-vector, b = m-vector, A = mn matrix
select a set of m linearly independent columns such that (mm) matrix B is nonsingular, i.e. |B| 0.
Then one may uniquely solve the equation: Bxb= b
where: xb is an m-element subvector of x.
Max f(x) = 100x1 + 8Ox2 + 0s1 + 0s2 + 0s3
Subject to: 10x1 + 5x2 + s1
= 505x1 + 5x2 + s2
= 35 5x1 + 15x2 + s3
= 80
23Department of Electrical Engineering - UMASS
Basis and Basic solution
By putting x=(xb,0) we obtain a solution to
Ax = b
Such a solution (with n-m components of x not associated with columns of Bmm) to the resulting set of equations is said to be a basic solution with regard to the basis B.
The Xb variables, associated with columns of B, are called basic variables.
Basic Feasible Solution Basic Feasible Solution
If a feasible solution (i.e. the one which satisfies all contraints) is also basic, it is called a basic feasible solution.
Max f(x) = 100x1 + 8Ox2 + 0s1 + 0s2 + 0s3
Subject to: 10x1 + 5x2 + s1
= 505x1 + 5x2 + s2
= 35 5x1 + 15x2 + s3
= 80
B.F.S x1 = 0 s1 = 50 s3 = 80 x2 = 0 s2 = 35
basic variables
24Department of Electrical Engineering - UMASS
Fundamental Theorem of Linear Programming
Given an LP in standard form
min CTxsubject to Ax = bx 0
where A is mn matrix of rank m (i.e. m < n and the m rows are linear independent),
• If there is a feasible solution, there is a basic feasible solution
• If there is an optimal feasible solution, there is an optimal basic feasible solution
25Department of Electrical Engineering - UMASS
Fundamental Theorem of Linear Programming
This theorem shows that it is necessary only to consider basic feasible solutions when seeking an optimal solution to a linear program, as the optimal value is always achieved as such a solution
Ultimate goalUltimate goal::
• find a basic feasible solution with a base B composed of original variables only, and which is optimum.
26Department of Electrical Engineering - UMASS
LP Example (Basic Feasible solution)
ƒ (x) = 100 x1 + 80 x2 + 0 s1 + 0 s2 + 0 s3
Solution: x1 = 0, x2 = 0, s1 = 50, s2 = 35, s3 = 80, is a basic feasible solution: point P0(0,0) Extreme point
Function value: F(0, 0, 50, 35, 80) = 0
1
210 5 1 0 0 505 5 0 1 0 3515 15 0 0 1 802
3
xxsss
basis
27Department of Electrical Engineering - UMASS
LP Example (Basic Feasible solution)
The basic feasible solution in the Geometric approach is in the Extreme Point.
a bx1
x2
10
5
15105 c
P2
P13
4
P0
Extremepoint
28Department of Electrical Engineering - UMASS
Simplex Method
Proceed from one basic feasible solution (extreme point) to another, in such a way as to continually decrease / increase the value of the f (x) until a minimum / maximum is reached.
General comments:General comments:
It is easy find initial basic feasible solutions with slack variables.
Finding initial basic solution is part of the Simplex method (Lue 84).
29Department of Electrical Engineering - UMASS
Simplex Algorithm
1. Select the column, such that the new resulting basic feasible solution will yield a lower / greater value to f(x) than the previous one.
2. Select the pivot element in that column.
By an elementary evaluation determine:
• Which vector aj should enter the basis (so that f(x) is reduced), and
• Which vector should leave the basis.
30Department of Electrical Engineering - UMASS
Simplex Algorithm
Construct a Simplex Table, [A, b ] appended by a row at the top with cost coefficients and current cost.
j =j = 1 2 3 4 5 0
rrjj 100 80 0 0 0 0yy1j1j 10 5 1 0 0 50
yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80
yi1 yi2 yi3 yi4 yi5 yi0
Point P0 (0,0)F(0,0,50,35,80) = 0
31Department of Electrical Engineering - UMASS
Simplex Algorithm
Step 1. Select column q with rq > 0. (xq enters the basis) q : rq = max ri
Step 2. Select row p, which defines pivot element p in column q Which minimizes ratio:
p : = min for positive yiqi
0i
iq
yy
0i
iq
yy
0p
pq
yy
32Department of Electrical Engineering - UMASS
Simplex Algorithm
Step 3. Pivot on element ypq and update the table (update all rows, including the top one)
• In row p divide ypj by ypq , j = 0, 1,…, n
• Subtract from each row i p
ypj , j = 0, 1,…, n 0p
pq
yy
33Department of Electrical Engineering - UMASS
Simplex Algorithm Example
Pivot on y11
j =j = 1 2 3 4 5 0
rrjj 100 80 0 0 0 0yy1j1j 10 5 1 0 0 50yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80
yi1 yi2 yi3 yi4 yi5 yi0 Point P0 (0,0)F(0,0,50,35,80) = 0
Step1: max rq
10
11
20
21
30
31
50 51035 75
80 165
yyyyyy
Step2: min of
Therfore: ypq = 10
34Department of Electrical Engineering - UMASS
Simplex Algorithm Example
Update table
j =j = 1 2 3 4 5 0
rrjj 100 80 0 0 0 0yy1j1j 10 5 1 0 0 50yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80
yi1 yi2 yi3 yi4 yi5 yi0Point P0 (0,0)F(0,0,50,35,80) = 0
1iq iq
ij ij pj ij pjpq
y yy y y y y
y
Step 3.B: in row i ≠ p
1 1
11 10pj j j
pq
y y yy y
Step 3.A: in row p
j =j = 1 2 3 4 5 0
rrjj 100 80 0 0 0 0yy1j1j 1 0.5 0.1 0 0 5yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80
yi1 yi2 yi3 yi4 yi5 yi0
j =j = 1 2 3 4 5 0
rrjj 0 30 -10 0 0 -500yy1j1j 1 0.5 0.1 0 0 5yy2j2j 0 2.5 -0.5 1 0 10yy3j3j 0 12.5 -0.5 0 1 55
yi1 yi2 yi3 yi4 yi5 yi0 Point P1 (5,0)F(5,0,0,10,55) = 500
35Department of Electrical Engineering - UMASS
Simplex Algorithm Example
Pivot on y22
j =j = 1 2 3 4 5 0
rrjj 0 30 -10 0 0 -500yy1j1j 1 0.5 0.1 0 0 5yy2j2j 0 2.5 -0.5 1 0 10yy3j3j 0 12.5 -0.5 0 1 55
yi1 yi2 yi3 yi4 yi5 yi0Point P1 (5,0)F(5,0,0,10,55) = 500
Step1: max rq
10
12
20
22
30
32
5 100.510 42.555 4.4
12.5
yyyyyy
Step2: min of
Therfore: ypq = 4
36Department of Electrical Engineering - UMASS
j =j = 1 2 3 4 5 0
rrjj 100 80 0 0 0 0yy1j1j 1 0.5 0.1 0 0 5yy2j2j 5 5 0 1 0 35yy3j3j 5 15 0 0 1 80
yi1 yi2 yi3 yi4 yi5 yi0
Simplex Algorithm Example
Update table
Point P1 (5,0)F(5,0,0,10,55) = 500
1iq iq
ij ij pj ij pjpq
y yy y y y y
y
Step 3.B: in row i ≠ p
2 2
22 2.5pj j j
pq
y y yy y
Step 3.A: in row p
Point P2 (3,4)F(3,4,0,0,5) = 620
j =j = 1 2 3 4 5 0
rrjj 0 30 -10 0 0 -500yy1j1j 1 0.5 0.1 0 0 5yy2j2j 0 1 -0.2 0.4 0 4yy3j3j 0 12.5 -0.5 0 1 55
yi1 yi2 yi3 yi4 yi5 yi0
j =j = 1 2 3 4 5 0
rrjj 0 0 -4 -12 0 -620yy1j1j 1 0 1 0 0 3yy2j2j 0 1 -0.2 0.4 0 4yy3j3j 0 0 2 -5 1 5
yi1 yi2 yi3 yi4 yi5 yi0
37Department of Electrical Engineering - UMASS
Simplex Algorithm Example
Pivot on
j =j = 1 2 3 4 5 0
rrjj 0 0 -4 -12 0 -620yy1j1j 1 0 1 0 0 3yy2j2j 0 1 -0.2 0.4 0 4yy3j3j 0 0 2 -5 1 5
yi1 yi2 yi3 yi4 yi5 yi0Point P2 (3,4)F(3,4,0,0,5) = 620
Step1: all rq < 0Terminate algorithm
Sol:X1 = 3X2 = 4
38Department of Electrical Engineering - UMASS
Linear Programming Modifications
1. Maximization
2. Unconstrained xmax c t x = min (-c t x)
let x = x+ - x - ; x+, x - 0)
min (c t x+ - c t x -)
[A, -A] = b
x+
x -
x+, x - 0
s. to
39Department of Electrical Engineering - UMASS
Linear Programming Modifications
3. Constraint set: Ax b
Slack variables, xs : Ax + xs = b
[A, I ] = bxxxxss
x , xs 0min [cT , 0]
xxs
Columns corresponding to xs form a basic solution
40Department of Electrical Engineering - UMASS
Linear Programming Modifications
4. Constraint set: Ax ≥ b
Surplus variables, xp : Ax - xp = b
[A, -I ] = bxxxxpp
x , xp 0min [cT , 0]
xxp
41Department of Electrical Engineering - UMASS
Linear Programming Modifications5. Artificial variables, xa : Ax - xp + xa = b
(cannot star with negative basis of xp )
[A, -I , I ] = b
x , xp , xa 0
min [cT , 0 , k I ]
xxp
xa
xxp
xaS. to
Columns corresponding to xa form a basic solution