15-853 Page1
15-853:Algorithms in the Real World
Linear and Integer Programming I– Introduction– Geometric Interpretation– Simplex Method
15-853 Page2
Linear and Integer Programming
Linear or Integer programming minimize z cTx cost or objective function
subject to Ax b equalities x 0 inequalities
c n, b m, A n x m
Linear programming: x n (polynomial time)Integer programming: x n (NP-complete)Very general framework, especially IP
15-853 Page3
Related Optimization Problems
Unconstrained optimizationmin{f(x) : x n}
Constrained optimizationmin{f(x) : ci(x) 0, i I, cj(x) 0, j E}
Quadratic programmingmin{1/2xTQx + cTx : ai
Tx bi, i I, aiTx bj,
j E}Convex programming
min{f(x) : fi(x) bi, i I, f, fi are convex}
Mixed Integer Programmingmin{cTx : Ax b, x 0, xi , i I, xr
, r R}
15-853 Page4
How important is optimization?
• 50+ packages available• 1300+ papers just on interior-point methods
• 100+ books in the library• 10+ courses at most Universities• 100s of companies• All major airlines, delivery companies, trucking companies, manufacturers, … make serious use of optimization.
15-853 Page5
Linear+Integer Programming Outline
Linear Programming– General formulation and geometric interpretation
– Simplex method– Ellipsoid method– Interior point methods
Integer Programming– Various reductions of NP hard problems– Linear programming approximations– Branch-and-bound + cutting-plane techniques– Case study from Delta Airlines
15-853 Page6
Applications of Linear Programming
1. A substep in most integer and mixed-integer linear programming (MIP) methods
2. Used to approximate various NP-Hard problems
3. Selecting a mix: oil mixtures, portfolio selection
4. Distribution: how much of a commodity should be distributed to different locations.
5. Allocation: how much of a resource should be allocated to different tasks
6. Network Flows
15-853 Page7
Linear Programming for Max-Flow
Create two variables per edge:
Create one equality per vertex:x1 + x2 + x3’ x1’ + x2’ + x3
and two inequalities per edge:x1 3, x1’ 3
add edge x0 from out to in
maximize x0
in out
1
2
3
7
6
2
3
5
7
x1x3
x2
x1 x1’
15-853 Page8
In Practice
In the “real world” most problems involve at least some integral constraints.
• Many resources are integral• Can be used to model yes/no decisions (0-1 variables)
Therefore “1. A subset in integer or MIP programming” is the most common use in practice
15-853 Page9
Algorithms for Linear Programming
• Simplex (Dantzig 1947)• Ellipsoid (Kachian 1979) first algorithm known to be polynomial time
• Interior Point first practical polynomial-time algorithms– Projective method (Karmakar 1984)– Affine Method (Dikin 1967)– Log-Barrirer Methods (Frisch 1977, Fiacco 1968, Gill et.al. 1986)
Many of the interior point methods can be applied to nonlinear programs. Not known to be poly. time
15-853 Page10
State of the art
1 million+ variables10 million+ nonzerosNo clear winner between Simplex and Interior Point– Depends on the problem– Interior point methods are subsuming more and more cases
– All major packages supply bothThe truth: the sparse matrix routines, make or break both methods.
The best packages are highly sophisticated.
15-853 Page11
Comparisons
problemSimplex (primal)
Simplex (dual)
Barrier + crossover
binpacking
29.5 62.8 560.6
distribution
18,568.0 won’t run too big
forestry 1,354.2 1,911.4 2,348.0
maintenace
57,916.3 89,890.9 3,240.8
crew 7,182.6 16,172.2 1,264.2
airfleet 71,292.5 108,015.0 37,627.3
energy 3,091.1 1,943.8 858.0
4color 45,870.2 won’t run too big
15-853 Page12
Formulations
There are many ways to formulate linear programs:– objective (or cost) functionmaximize cTx, orminimize cTx, orfind any feasible solution
– (in)equalitiesAx b, orAx b, orAx b, or any combination
– nonnegative variablesx 0, or not
Fortunately it is pretty easy to convert among forms
15-853 Page13
Formulations
The two most common formulations:
minimize cTx
subject to Ax b
x 0
minimize cTx
subject to Ax b
x 0
slack variables
e.g.
7x1 + 5x2 7x1, x2 0
7x1 + 5x2 - y1 7
x1, x2, y1 0
y1
More on slack variables later.
1 2
15-853 Page14
Geometric View
A polytope in n-dimensional spaceEach inequality corresponds to a half-space.
The “feasible set” is the intersection of the half-spaces.
This corresponds to a polytopeThe optimal solution is at a corner.
Simplex moves around on the surface of the polytope
Interior-Point methods move within the polytope
15-853 Page15
Geometric View
An intersection of 5 halfspaces
minimize:
z -2x1 - 3x2
subject to:
x1 – 2x2
4
2x1 + x2
18
x2 10
x1, x2 0
x2 10
x1 – 2x2 4
x1
x2
FeasibleSet Corners
ObjectiveFunction-2x1 – 3x2
2x1 + x2 18
15-853 Page16
Notes about higher dimensions
For n dimensions and no degeneracyEach corner (extreme point) consists of:
– n intersecting n-1 dimensional hyperplanese.g. 3, 2d planes in 3d
– n intersecting edgesEach edge corresponds to moving off of one hyperplane (still constrained by n-1 of them)
# Corners can be exponential in n (e.g. a hypercube)
Simplex will move from corner to corner along the edges
15-853 Page17
Optimality and Reduced Cost
The Optimal solution must include a corner. The Reduced cost for a hyperplane at a corner is the cost of moving one unit away from the plane along its corresponding edge.
1 eiz
pi
ri = z ei
For minimization, if all reduced cost are non-negative, then we are at an optimal solution.
Finding the most negative reduced cost is a heuristic for choosing an edge to leave on
15-853 Page18
x1
x2
z -2x1 – 3x2
Reduced cost example
In the example the reduced cost of leaving the plane x1 is(-2,-3) (2,1) -7since moving one unit off of x1 will move us (2,1) units along the edge. We take the dot product of this and the cost function.
(0,1)(2,1)
15-853 Page19
Simplex Algorithm
1. Find a corner of the feasible region2. Repeat
A. For each of the n hyperplanes intersecting at the corner, calculate its reduced cost
B. If they are all non-negative, then doneC. Else, pick the most negative reduced cost
This is called the entering planeD. Move along corresponding edge (i.e. leave
that hyperplane) until we reach the next corner (i.e. reach another hyperplane)The new plane is called the departing plane
15-853 Page20
x1
x2
z -2x1 – 3x2
Start
Example
Entering
Departing
Step 1
Step 2
15-853 Page21
Simplifying
Problem:– The Ax b constraints not symmetric with the x 0 constraints.We would like more symmetry.
Idea:– Make all inequalities of the form x 0.
Use “slack variables” to do this.Convert into form:
minimize cTx
subject to Ax b
x 0
15-853 Page22
Standard and Slack Form
x1
x2
x4
x3
x5
x1
x2
x2 10
x1 – 2x2 4
2x1 + x2 18
Standard Form Slack Form
minimize cTx
subject to Ax b
x 0
minimize cTx’
subject to A’x’ b
x’ 0
slack variables
|A| = m x n i.e. m equations, n variables
|A’| = m x (m+n) i.e. m equations, m+n variables
2x1 + x2 + x4 18
15-853 Page23
Example, again
minimize:
z -2x1 - 3x2
subject to:
x1 – 2x2 + x3
4
2x1 + x2 + x4
18
x2 + x5 10
x1, x2, x3, x4, x5 0
x1
x2
x1
x2
x4
x3
x5
The equality constraints impose a 2d plane embedded in 5d space, looking at the plane gives the figure above
15-853 Page24
Using MatricesIf before adding the slack variables A has size m x n then after it has size m x (n + m)
m can be larger or smaller than n
m
n m
A =
1 0 0 …0 1 0 …0 0 1 ……
Assuming rows are independent, the solution space of Ax = b is a n dimensional subspace.
slack vrs.
15-853 Page25
Gauss-Jordan Elimination
000010
liA
A
liA
AAA
B
lk
lj
lk
ikljij
ij
Gauss-Jordan elimination
j
il
k
k
lijA
15-853 Page26
Simplex Algorithm, again
1. Find a corner of the feasible region2. Repeat
A. For each of the n hyperplanes intersecting at the corner, calculate its reduced cost
B. If they are all non-negative, then doneC. Else, pick the most negative reduced cost
This is called the entering planeD. Move along corresponding line (i.e. leave
that hyperplane) until we reach the next corner (i.e. reach another hyperplane)The new plane is called the departing plane
15-853 Page27
Simplex Algorithm (Tableau Method)
r
b’
-z
FI
0
Basic Vars.
Free Variables reduced costs
n
current cost
m
This form is called a Basic Solution• the n “free” variables are set to 0• the m “basic” variables are set to b’A valid solution to Ax = b if reached using Gaussian
EliminationRepresents n intersecting hyperplanesIf feasible (i.e. b’ 0), then the solution is called
a Basic Feasible Solution and is a corner of the feasible set
15-853 Page28
Corner
1 0 0 1 -2 4
0 1 0 2 1 18
0 0 1 0 1 10
0 0 0 -2 -3 0
x3 x4 x5 x1 x2
x1
x2
x4
x3
x5
free variablesbasic variables
15-853 Page29
Corner
1 0 0 -.5 -1 -2
0 1 0 2.5 1 20
0 0 1 .5 1 12
0 0 0 -3.5 -3 -6
x2 x4 x5 x1 x3
x1
x2
x4
x3
x5
15-853 Page30
Corner
1 0 0 1 -2 4
0 1 0 -2 5 10
0 0 1 0 1 10
0 0 0 2 -7 8
x1 x4 x5 x3 x2
x1
x2
x4
x3
x5
15-853 Page31
Corner
1 0 0 .2 .4 8
0 1 0-.4
.2 2
0 0 1 .4-.2
8
0 0 0-.8
1.4
22
x1 x2 x5 x3 x4
x1
x2
x4
x3
x5
15-853 Page32
Corner
1 0 0 .5-2.5
-5
0 1 0 .5 .5 9
0 0 1 0 1 10
0 0 0 1 -2 18
x3 x1 x5 x4 x2
x1
x2
x4
x3
x5
Note that in general there are n+m choose m corners
15-853 Page33
Simplex Method Again
Once you have found a basic feasible solution (a corner), we can move from corner to corner by swapping columns and eliminating.
ALGORITHM1. Find a basic feasible solution2. Repeat
A. If r (reduced cost ) 0 , DONEB. Else, pick column with most negative rC. Pick row with least positive b’/(selected
column)D. Swap columnsE. Use Gauss-Jordan elimination to restore
form
15-853 Page34
Tableau Method
A. If r are all non-negative then done
r
b’
z
FI
0
BasicVariables
Free Variablesvalues are 0 reduced costs
if all 0 then done
n
current cost
15-853 Page35
Tableau Method
B. Else, pick the most negative reduced costThis is called the entering plane or variable
r
b’
z
FI
0
n
min{ri}entering variable
15-853 Page36
Tableau Method
C. Move along corresponding line (i.e. leave that hyperplane) until we reach the next corner (i.e. reach another hyperplane)The new plane is called the departing plane
r
b’
z
FI
0
1
departing variable
min positive bj’/uj
u
15-853 Page37
Tableau Method
D. Swap columns
E. Gauss-Jordan elimination
r
b’
z
x
swap
ri+1 zi+1
Fi+1I
0
bi+1’
x
xx
x
No longer inproper form
Back toproper form
15-853 Page38
Examplex1 x2 x3 x4 x5
1 -2 1 0 0 4x1 – 2x2 + x3 =
4
2 1 0 1 0 182x1 + x2 + x4 =
18
0 1 0 0 1 10 x2 + x5 = 10
-2 -3 0 0 0 0 z = -2x1 – 3x2
1 0 0 1 -2 4
0 1 0 2 1 18
0 0 1 0 1 10
0 0 0 -2 -3 0
x3 x4 x5 x1 x2
x1
x2
x4
x3
x5
x1 = x2 = 0 (start)
Find corner
15-853 Page39
Example
1 0 0 1 -2 4
0 1 0 2 1 18
0 0 1 0 1 10
0 0 0 -2 -3 0
x3 x4 x5 x1 x2
1 0 0 1 -2 4 -2
0 1 0 2 1 18 18
0 0 1 0 1 10 10
0 0 0 -2 -3 0
x3 x4 x5 x1 x2 bj/vjminpositive
x1
x2
x4
x3
x5
10
18
-2
15-853 Page40
Example
1 0 -2 1 0 4
0 1 1 2 0 18
0 0 1 0 1 10
0 0 -3 -2 0 0
x3 x4 x2 x1 x5
1 0 0 1 2 24
0 1 0 2 -1 8
0 0 1 0 1 10
0 0 0 -2 3 30
x3 x4 x2 x1 x5
Gauss-JordanElimination
swap
x1
x2
x4
x3
x5
10
18
-2
15-853 Page41
Example
1 0 0 1 2 24
0 1 0 2 -1 8
0 0 1 0 1 10
0 0 0 -2 3 30
x3 x4 x2 x1 x5
x1
x2
x4
x3
x5
10
18
-21 0 0 1 2 24 24
0 1 0 2 -1 8 4
0 0 1 0 1 10 ∞0 0 0 -2 3 30
x3 x4 x2 x1 x5 bj/vj
minpositive
15-853 Page42
Exampleswap
x1
x2
x4
x3
x5
10
18
-2
1 1 0 0 2 24
0 2 0 1 -1 8
0 0 1 0 1 10
0 -2 0 0 3 30
x3 x1 x2 x4 x5
1 0 0 -.5 2.5 20
0 1 0 .5 -.5 4
0 0 1 0 1 10
0 0 0 1 2 38
x3 x1 x2 x4 x5
Gauss-JordanElimination
15-853 Page43
Simplex Concluding remarks
For dense matrices, takes O(n(n+m)) time per iteration
Can take an exponential number of iterations.
In practice, sparse methods are used for the iterations.
15-853 Page44
Duality
Primal (P): maximize z cTx subject to Ax b x 0 (n equations, m variables)
Dual (D): minimize z yTb subject to ATy c y 0 (m equations, n variables)
Duality Theorem: if x is feasible for P and y is feasible for D, then cx yb and at optimality cx yb.
15-853 Page45
Duality (cont.)
feasible solutions for Dual (maximization)
feasible solutions for Primal (minimization)
Optimal solution for both
Quite similar to duality of Maximum Flow andMinimum Cut.
Useful in many situations.
15-853 Page46
Duality Example
Solution to both is 38 (x1 4, x2 10), (y1 0, y2 1, y3
2)
Primal:maximize:
z 2x1 + 3x2
subject to:
x1 – 2x2
4
2x1 + x2
18
x2 10
x1, x2 0
Dual:minimize:
z 4y1 + 18y2 + 10y3
subject to:
y1 + 2y2 2
-2y1 + Y2 + Y3
3
y1, y2, y3 0
x1
x2
C1
C2
C3
C7C5C6
C4
4/7 C1 + 5/7 C4
maximize:
2x1 + 3x3
s.t.: x1 + 4x2
44
C1
x1 + 2x2
30
C2
x1 + x2 17
C3
2x1 + x2
18
C4
x1 - 2x2
4 C5
-x1 0 C6
-x2 0 C7
2x1 + 3x2