Linear ProgrammingThe Simplex Algorithm: Part II
Chapter 5
University of ChicagoBooth School of Business
Kipp Martin
October 17, 2017
1
Outline
List of Files
Key Concepts
Revised Simplex
Revised Simplex with LU Decomposition
Other IssuesFinding a Starting BasisUnbounded ProblemsConvergenceThe Future
2
List of Files
I revisedsimplex.m
I revisedsimplexlu.m
I lptest1.m
I lptest2.m
I lptest3.m
I artificial.m
I unbounded.m
3
Key Concepts
I nonzero elements are the enemy
I density is our enemy
I zeros are our friend
I sparsity is our friend
I there is a big difference between toy textbook problems andproblems that need to get solved in practice
I deja vu – nonzero elements are the enemy – we will encountera number of other enemies throughout the quarter
4
Key Concepts
I revised Simplex
I revised Simplex with LU decomposition
I finding an initial basis
I unbounded linear programs
I degeneracy
5
Revised Simplex
In basicsimplex.m we partitioned the A matrix into basic andnonbasic columns
ABxB + ANxN = b
And then multiplied by the basis inverse
A−1B ABxB + A−1B ANxN = A−1B b
xB + A−1B ANxN = A−1B b
Here is the problem – even if AB and AN are sparse, A−1B AN isoften dense. Multiplying AN by A−1B often results in a very densematrix.
6
Revised Simplex
The non-basic columns of a covering problem before pivoting.
7
Revised SimplexThe non-basic columns after pivoting – note the nonzero fill in.
8
Revised Simplex
Bottom Line: No one ever, ever works with matrix A−1B AN inpractice. It is not practical. So we really have two problems ordragons to slay.
I we need the updated columns of AN (that is A−1B AN) in orderto do the minimum ratio test,
I we need A−1B to find the updated columns, the updatedright-hand side, and reduced costs.
9
Revised Simplex
In terms of basicsimplex.m we do the following:
T = inv(AB)*A;
bBAR = inv(AB)*b;
which is necessary for the minimum ratio test
if bBAR(k) / T( k, N(negRedCostIdx)) < minRatioVal
We also calculate
w = c’ - (cB’*inv(AB))*A;
which is necessary to determine the pivot column
if wN( k) < -.00001
so in general, T is VERY dense.
10
Revised Simplex
So let’s slay these two dragons one at a time. First, we eliminatethe need to explicitly keep T around. We store two things:
I The original A matrix
I The basis inverse (for now)
11
Revised SimplexStoring A: most real problems are actually quite sparse – that isthe ratio of nonzero elements in A to the number of elements in Ais quite small.
The vast majority of elements are zero. Consider for example atransportation problem.
A = [1 1 1 1 1 0 0 0 0 0;
0 0 0 0 0 1 1 1 1 1;
1 0 0 0 0 1 0 0 0 0;
0 1 0 0 0 0 1 0 0 0 ;
0 0 1 0 0 0 0 1 0 0;
0 0 0 1 0 0 0 0 1 0;
0 0 0 0 1 0 0 0 0 1]
Zeros are very good friends. We never need to use them inarithmetic operations such as plus or times so we don’t bother tokeep them around.
12
Revised Simplex
Sparse Matrix Storage: it is necessary to understand how amatrix is stored in practice.
Remember that real problems are sparse! Never store or operateon the zero elements.
Store the A matrix (constraint matrix) with three arrays (vectors):
I values – an array that holds the nonzero elements in the Amatrix, the size is equal to the number of nonzero elements inthe matrix
I indexes – an array that holds the row (column) indexes of thenonzero elements in the A matrix, the size is equal to thenumber of nonzero elements in the matrix
I starts – a pointer array that points the start of each column(row), the size is equal to the number of columns (rows) plusone in the matrix;
13
Revised Simplex
Column Major Storage:
Consider the following matrix.
A =
0 1.5 0 7.7 0
2.0 6.7 1.1 7.7 07.5 0 1.1 10 1
0 0 1.1 12 03.1 0 0 1.9 0
Then
values = 2.0, 7.5, 3.1|1.5, 6.7|1.1, 1.1, 1.1|7.7, 7.7, 10, 12, 1.9|1indexes = 1, 2, 4|0, 1|1, 2, 3|0, 1, 2, 3, 4|2starts = 0, 3, 5, 8, 13, 14
Note: we use 0-based counting for the array indexes
14
Revised Simplex
Key Idea: In revised simplex, we only store A and only update thecolumn we pivot on – none of the other columns! Seerevisedsimplex.m
aBAR = inv( AB)*AN(:,negRedCostIdx);
minRatioVal = inf;
for k = 1:numRows
if aBAR( k) > 0
if bBAR(k) / aBAR(k) < minRatioVal
minRatioVal = bBAR(k) /aBAR( k);
%record the index
minRatioIdx = k;
end
end
end
15
The Revised Simplex Algorithm
Here is a formal mathematical statement of the Revised SimplexAlgorithm.
Step 1: (Initialization) Initialize with a basis B corresponding toa basic feasible solution and for the corresponding basis matrixcalculate initial values of the right-hand-side vector
xBi= A−1B b
Step 2: (Pivot Column Selection) Calculate the reduced costs
wN = c>N − c>B A−1B AN
If wN ≥ 0, stop, the current solution is optimal; else, select q ∈ Nsuch that wq < 0.
16
The Revised Simplex Algorithm
How do we do the multiplication c>B A−1B AN in practice?
Matrix multiplication is associative.
(c>B A−1B )AN = c>B (A−1B AN)
Which do we prefer?
Why?
17
The Revised Simplex AlgorithmStep 3: (Minimum Ratio Test and Pivot Row Selection)Perform the minimum ratio test on column q that was selected inStep 2. First find the updated column q.
aq = A−1B aq
i∗ ← argmin {xBi/aiq | aiq > 0, i = 1, . . .m}
Step 4: (Update the right-hand-side)
xBi← xBi
− (xBi∗/ai∗q)aq
xBi∗ ← xBi∗/ai∗q
18
The Revised Simplex Algorithm
Step 5: (Update the Basic and Non-Basic Index Sets)
B ← (B\{Bi∗}) ∪ {q} N ← (N\{q}) ∪ {Bi∗}
Go to Step 2.
19
The Revised Simplex Algorithm
In Step 2 we calculate the reduced costs by
wN = c>N − (c>B A−1B )AN
not
wN = c>N − c>B (A−1B AN)
20
Revised Simplex
The MATLAB program, revisedsimplex.m, implements therevised Simplex algorithm as just described. (We assume a minproblem in standard form.)
Inputs:
I A – the constraint matrix
I b – the right-hand-side vector
I c – the objective function vector
I B – an index set of the basic variables (feasible)
Outputs:
I X – the optimal solution to the linear program
I objVal – the optimal objective function value
I B – the optimal basis
21
Revised SimplexA problem in standard form:
A = [.7 1 1 0 0 0
.5 5/6 0 1 0 0
1 2/3 0 0 1 0
1/10 1/4 0 0 0 1]
c = [-10; -9; 0; 0; 0; 0 ]
b = [630; 600; 708; 135]
B = [3 4 5 6]
Make the call
[X, objVal, B] = revisedsimplex( A, b, c, B)
22
Revised Simplex with LU Decomposition
Okay, so with revised simplex we avoid working with the updatedtableau,
xB + A−1B ANxN = A−1B b
Time to slay the second dragon – we don’t want to explicitly haveto calculate A−1B because it is too dense.
There are two places in the iterative process of revised simplex thatwe need A−1B .
I In Step 2 we need to calculate c>B A−1B
I In Step 3 we need to calculate aq = A−1B aq
23
Revised Simplex with LU Decomposition
We avoid calculating A−1B by using LU decomposition.
Reduced Cost Calculation: We need to calculate
u> = c>B A−1B
u>AB = c>B
(u>AB)> = (c>B )>
A>Bu = cB
In other words solve the system
A>Bu = cB
24
Revised Simplex with LU Decomposition
Solve the system A>Bu = cB with LU decomposition.
We know there exists a lower triangular matrix L, with diagonalelements of 1, and upper triangular matrix U, and permutationmatrix P such that
PA>B = LU
We solve
LUu = PcB
Let y = Uu. Do a forward solve on Ly = PcB to obtain y and thendo a backward solve on Uu = y .
25
Revised Simplex with LU Decomposition
Assume that the original A matrix is
A = [.7 1 1 0 0 0
.5 5/6 0 1 0 0
1 2/3 0 0 1 0
1/10 1/4 0 0 0 1]
A basis corresponding to a basic feasible solution is
B = [3, 4, 5, 2]
c> =[−10 −9 0 0 0 0
]Set up a system of equations to find the reduced costs.
26
Revised Simplex with LU Decomposition
>> [L, U, P] = mylu( AB’)
L =
1.0000 0 0 0
0 1.0000 0 0
0 0 1.0000 0
1.0000 0.8333 0.6667 1.0000
U =
1.0000 0 0 0
0 1.0000 0 0
0 0 1.0000 0
0 0 0 0.2500
P = I 27
Revised Simplex with LU Decomposition
We are solving
LUu = PcB
We set y = Uu and a forward solve on Ly = PcB
y1 = 0
y2 = 0
y3 = 0
y1 + (5/6)y2 + (2/3)y3 + y4 = −9
28
Revised Simplex with LU Decomposition
Do a backward solve on Uu = y
u1 = y1
u2 = y2
u3 = y3
(1/4)u1 = y4
29
Revised Simplex with LU Decomposition
This is implemented in revisedsimplexlu.m. In this code we do
% reduced cost calculations
u = lusolve(AB’, cB);
wN = cN’ - u’*AN;
Pretty easy!
30
Revised Simplex with LU Decomposition
We also need, aq = A−1B aq. That is, we need to solve
PABaq = Paq
LUaq = Paq
Let y = Uaq.
Do a forward solve on Ly = Paq to obtain y and then do abackward solve on Uaq = y .
In the revisedsimplexlu.m code this is implemented as
aBAR = lusolve(AB, AN(:,negRedCostIdx) );
31
Revised Simplex with LU DecompositionCalculate the updated column for variable 1 for use in theminimum ratio test.
>> [L, U, P] = mylu( AB)
L =
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
U =
1.0000 0 0 1.0000
0 1.0000 0 0.8333
0 0 1.0000 0.6667
0 0 0 0.2500
P = I32
Revised Simplex with LU Decomposition
We solve LUaq = Paq.
Do a forward solve on Ly = a1.
y1 = .7
y2 = .5
y3 = 1
y4 = .1
33
Revised Simplex with LU Decomposition
Do a backward solve on Uu = y
u1 + u4 = y1
u2 + (5/6)u4 = y2
u3 + (2/3)u4 = y3
u4 = y4
34
Revised Simplex
The MATLAB program, revisedsimplexlu.m, implements therevised Simplex algorithm as just described with LUdecomposition. (We assume a min problem in standard form.)
Inputs:
I A – the constraint matrix
I b – the right-hand-side vector
I c – the objective function vector
I B – an index set of the basic variables (feasible)
Outputs:
I X – the optimal solution to the linear program
I objVal – the optimal objective function value
I B – the optimal basis
35
Revised SimplexA problem in standard form:
A = [.7 1 1 0 0 0
.5 5/6 0 1 0 0
1 2/3 0 0 1 0
1/10 1/4 0 0 0 1]
c = [-10; -9; 0; 0; 0; 0 ]
b = [630; 600; 708; 135]
B = [3 4 5 6]
Make the call
[X, objVal, B] = revisedsimplexlu( A, b, c, B)
36
Revised Simplex with LU Decomposition
I actually have hidden a lot of detail from you.
In the MATLAB code revisedsimplexlu.m, at every iteration,we do a brand new LU decomposition of the basis matrix.
However, from iteration-to-iteration the basis matrix changes byonly one column. Hence an LU decomposition from scratch is ahuge waste!!!!
An incredible amount of research has gone into doing sparse LUupdates. We will not get into this.
If you understand what is in these slides you have a pretty goodfeeling for what is happening with real-world codes.
37
Finding a Starting BasisThe MATLAB simplex routines require as input, B, which indexesa basic feasible solution. For some problems it is easy to find astarting basic feasible solution. The linear program
max 10x1 + 9x2
.7x1 + x2 ≤ 630
.5x1 + (5/6)x2 ≤ 600
x1 + (2/3)x2 ≤ 708
.1x1 + .25x2 ≤ 135
x1, x2 ≥ 0
in standard form is
min −10x1 −9x2.7x1 +x2 +x3 = 630.5x1 +(5/6)x2 +x4 = 600x1 +(2/3)x2 +x5 = 708
.1x1 +.25x2 +x6 = 135x1, x2, x3, x4, x5, x6 ≥ 0
38
Finding a Starting Basis
Given the standard form
min −10x1 −9x2.7x1 +x2 +x3 = 630.5x1 +(5/6)x2 +x4 = 600x1 +(2/3)x2 +x5 = 708
.1x1 +.25x2 +x6 = 135x1, x2, x3, x4, x5, x6 ≥ 0
a trivial basic feasible solution is indexed by
B = [3 4 5 6]
Indeed, if the original linear program has all constraints of the formAx ≤ b and b ≥ 0, then we convert to standard form by adding anonnegative slack variable to each row and putting the slackvariable in the basis. The nonnegativity of b guarantees that this isa basic feasible solution.
39
Finding a Starting BasisFor problems with more general structure, finding a basic feasiblesolution by inspection is not so easy. Fortunately, there asystematic procedure for finding a starting basic feasible solution,but it requires solving a linear program. (See Section 6.5 of thetext).First convert the original linear program to standard form.
min c>x
s.t. Ax = b
x ≥ 0
Then convert the standard form to a Phase I problem.
min 0x + e>a
s.t. Ax + Ia = b
x , a ≥ 0
40
Finding a Starting BasisConverting to Phase I:
I Append to the constraint matrix A an m ×m identity matrixwhere m is the number of constraints. Thus, the newconstraint matrix is [A I ].
I Add m new variables to the problem. These are calledartificial variables. The coefficient of the new variablescorrespond to the identity matrix added.
I Give the original variables a cost of zero in the objectivefunction.
I Give each artificial variable a cost of 1.0 in the objectivefunction.
Result: If the simplex algorithm is applied to the Phase I problemfor a feasible and bounded linear program, then it will find a basicfeasible solution with no artificial variables and have optimal valueequal to zero.
41
Finding a Starting Basis
Phase I Example: Consider the following linear program instandard form:
MIN 2 X1 + 11 X2 + 7 X3 + 7 X4 + 20 X5 +
2 X6 + 5 X7 + 5 X8
SUBJECT TO
X1 + X2 = 1
X3 + X4 = 1
X5 + X6 = 1
X7 + X8 = 1
3 X1 + 6 X3 + 5 X5 + 7 X7 + X9 = 13
2 X2 + 4 X4 + 10 X6 + 4 X8 + X10 = 10
Convert to a Phase I linear program.
42
Finding a Starting Basis
Phase I Example: The Phase I version of this problem is:
MIN A1 + A2 + A3 + A4 + A5 + A6
SUBJECT TO
X1 + X2 + A1 = 1
X3 + X4 + A2 = 1
X5 + X6 + A3 = 1
X7 + X8 + A4= 1
3 X1 + 6 X3 + 5 X5 + 7 X7 + X9 + A5= 13
2 X2 + 4 X4 + 10 X6 + 4 X8 + X10 + A6 = 10
43
Finding a Starting BasisPhase I Example: The problem in MATLAB, artificial.m is
A = [1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0;
0 0 1 1 0 0 0 0 0 0 0 1 0 0 0 0;
0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 ;
0 0 0 0 0 0 1 1 0 0 0 0 0 1 0 0 ;
3 0 6 0 5 0 7 0 1 0 0 0 0 0 1 0;
0 2 0 4 0 10 0 4 0 1 0 0 0 0 0 1]
b = [1; 1; 1; 1; 13; 10];
c = [0; 0; 0; 0; 0; 0; 0; 0;
0 ; 0; 1; 1; 1; 1; 1; 1];
B = [11 12 13 14 15 16];
[X, objVal, B] = revisedsimplexlu( A, b, c, B)
44
Finding a Starting Basis
Phase I Example: The MATLAB solution is
objVal =
0
B =
1 3 6 5 7 4
45
Unbounded Problems
Recall that a linear program in standard form can be rewritten as
z0 = c>B A−1B b + w>NxN
xB = A−1B b − A−1B ANxN
xB , xN ≥ 0
What if there is a variable k ∈ N such that?
I wk < 0
I ak = A−1B ak ≤ 0
46
Unbounded Problems
We are unbounded!!! We currently are at the point x1 = [xB xN ]with objective function value c>B A−1B b. Move to the new point:
x2 = x1 + λr
for positive λ where component k of the r vector is 1, and theother nonzero components are given by ak .
This is tough. Let’s illustrate.
47
Unbounded Problems
Consider the following linear program in standard form.
min −2x1 −3x2x2 −x3 = 1
x1 +x2 −x4 = 2−.5x1 +x2 +x5 = 8−x1 +x2 +x6 = 6x1, x2, x3, x4, x5, x6 ≥ 0
The associated MATLAB file is unbounded.m.
48
Unbounded ProblemsThe MATLAB file is unbounded.m. defines the following:
% the constraint matrix
A = [0 1 -1 0 0 0
1 1 0 -1 0 0
-.5 1 0 0 1 0
-1 1 0 0 0 1]
% the objective function
c = [-2; -3; 0; 0; 0; 0 ]
% the right-hand-side
b = [1; 2; 8; 6]
%Initial Basis
B = [1 2 5 6]
49
Unbounded Problems
The associated MATLAB file is unbounded.m.
[tableau, B] = simplexpivot( A, b, c, B)
-0.00 -0.00 1.00 2.00 -0.00 -0.00 -5.00
----------------------------------------------------------------
1.00 0.00 1.00 -1.00 0.00 0.00 1.00
0.00 1.00 -1.00 0.00 0.00 0.00 1.00
0.00 0.00 1.50 -0.50 1.00 0.00 7.50
0.00 0.00 2.00 -1.00 0.00 1.00 6.00
Variable x4 has a reduced cost of -2. What is the effect of bringingthis variable into the basis?
50
Unbounded ProblemsIf variable x4 pivots in at any positive value, no other variablepivots out. The current solution is
x0 =
1100
7.56.0
If we increase x4 by λ the new solution is
x = x0 + λr =
1100
7.56.0
+ λ
1001.5
1.0
51
Unbounded ProblemsThe vector
r =
1001.5
1.0
is called an extreme ray or extreme direction of the linearprogram. I can take any positive multiple of this, add it to afeasible solution, and still be feasible.
Since c>r = −2,
c>(x0 + λr) = c>x0 + λc>r = −5− 2λ
52
Unbounded ProblemsExperiment a bit with unbounded.m to get a feeling for what isgoing on.
x0 = [1; 1; 0; 0; 7.5; 6.0]
r = [1; 0; 0; 1; .5; 1]
lambda = 100
%new solution
x = x0 + lambda*r
%new objective function value
objVal = c’*x
%should be feasible, i.e.
%this should evaluate to 0
A*x - b
53
Unbounded Problems
Here is the geometry again.
2 4 6 8x1
2
4
6
8
10
12x2
Feasible Region
Iteration 1
Iteration 2
Iteration 3
Iteration 4
54
Unbounded Problems
Every point (working in (x1, x2) space) in the feasible region can bewritten as:
x = λ1
[11
]+ λ2
[02
]+ λ3
[06
]+ λ4
[4
10
]+ λ5
[10
]+ λ6
[21
]
λ1 + λ2 + λ3 + λ4 = 1
λ1, λ2, λ3, λ4, λ5, λ6 ≥ 0
Observations:
I The feasible region is unbounded.
I How did I find the extreme ray corresponding to variable λ6?
55
Unbounded Problems
In practice, what does an unbounded problem tell you?
56
Convergence
Why does the Simplex algorithm terminate?
I Every basic feasible solution corresponds to an extreme point.
I There are a finite number of extreme points.
I Assuming no degenerate pivots the objective function valuestrictly decreases at every iteration. Hence we cannot repeatextreme points. Therefore we terminate with a finite numberof iterations.
Remember – if there is an optimal solution to the problem, there isan optimal extreme point solution. A linear program can fail tohave an optimal solution if and only if it is either infeasible orunbounded.
57
Convergence
Observations:
1. The change in the objective function value for a pivot oncolumn k in row i is
wk
(biaik
)2. Since wk < 0, aik > 0, and bi ≥ 0, this change can only be
zero if bi = 0.
A pivot on row i when bi = 0 is called a degenerate pivot. It istheoretically possible for the Simplex algorithm to cycle in thepresence of degeneracy. Cycling is not a problem in practice.
However, degeneracy is a real problem in that it can really slowthings down. Commercial codes have pivot selection rules that willprevent cycling and reduce the number of degenerate pivots.
58
The Future – Pricing
Here is where we are headed. Right now we explicitly price everycolumn when we make the calculationStep 2: (Pivot Column Selection) Calculate the reduced costs
wN = c>N − (c>B A−1B )AN
For many important applications this is not practical:
I there are so many columns in AN it would require too muchmemory and time to price
I it is not even practical to enumerate all of the columns in AN
We will do the pricing operation by solving another optimizationproblem or by implementing an algorithm.
59
The Future – Interior Point
The Simplex algorithm moves from extreme point to extreme pointon the boundary of the feasible region.
At the end of the quarter we will study interior point algorithmsthat move through the interior of the feasible region.
We will apply interior point algorithms to both linear andnon-linear programs.
Even for linear programs, interior point algorithms are based onideas from nonlinear programming so we need to cover nonlinearprogramming in order to understand how interior point algorithmswork.
60