Solving LP and MIP Models with Piecewise Linear Objective Functions

Zonghao Gu Gurobi Optimization Inc. Columbus, July 23, 2014

Overview }  Introduction }  Piecewise linear (PWL) function ◦  Convex and convex relaxation

}  Modeling ◦  Variables for pieces ◦  SOS2, binary formulations for non-convexity ◦  Direct handing

}  Convex PWL objective ◦  How to extend primal and dual simplex

}  Non-convex PWL objective ◦  How to extend branch-and-bound algorithm

}  Possible future work

2

Introduction

3

Definition A  linear  program  with  separable  PWL  objec4ve  func4on  is  an  op4miza4on  problem  of  the  form  

4

njlinearpiecewiseareWhere

nj

mitoSubject

Minimize

xcuxl

bxa

xc

jj

jjj

ij

n

jij

j

n

jj

,,1,)(

,,1,

,,1,

)(

1

1

…

…

…

=

=≤≤

==∑

∑

=

=

Types

}  Convex cj(xj) ◦  Treated as LP

}  Non-convex cj(xj) ◦  Treated as MIP

5

Motivations }  Demands ◦  Models with true piecewise linear structures ◦  Approximation of nonlinear functions ◦  A lot of different applications ◦  Customer models and requests

}  Traditional approaches ◦  One variable for each piece ◦  SOS2 or binary variables for non-convex function

}  New approach ◦  Can we handle it directly to improve performance?

6

Previous Work quick incomprehensive survey

}  Convex case ◦  Fourer and Marsten, Solving Piecewise-Linear

Programs: Experiments with a Simplex Approach, 1992 §  Extend primal simplex to handle variables with piecewise

objective function directly §  No piece variables §  Use XMP subroutine library

}  Non-convex case ◦  SOS2 formulation, Beale and Tomlin, 1970 ◦  Branch-and-cut without binary variables, Keha, de

Farias and Nemhauser, 2006 ◦  Any work without adding piece variables?

7

Piecewise Linear Function

8

Piecewise Linear Function }  Definition c(x) = akx +bk, pk ≤ x ≤ pk+1, k = 1,…,t-1 where ak, bk, pk are constants for k = 1,…,t and p1 < p2 < … < pt

}  Convex c((x+y)/2) ≤ (c(x) + c(y))/2

9

Convex PWL Function }  Continuous }  Slopes are non-decreasing

10

Non-convex PWL Function }  There can be jumps at breaking points

11

Convex Relaxation }  Convex relaxation r(x) is convex and r(x) ≤ c(x) for all x }  Strongest convex relaxation ◦  A convex relaxation ◦  For all x, there exist x1 and x2 such that r(x) = α c(x1) + (1-α) c(x2) with x = αx1+(1- α) x2, 0 ≤ α ≤ 1 ◦  Hereafter, relaxation always means strongest one

}  Relaxation of a PWL function ◦  is a convex PWL function ◦  May contain fewer pieces

12

Convex Relaxation

13

Finding Convex Relaxation }  Algorithm ◦  Step 1: initialize a set of ordered points S = {(p1, c(p1)), (p2, c(p2))} ◦  Step 2: Loop k from 3 to t §  Find max j, such that the slope between (pk, c(pk)) and point j

in S is larger than the slope between points j and j-1 in S §  If no such j, set j = 1 §  Remove the points after j from S and add (pk, c(pk)) to S

}  Results ◦  S defines PWL convex relaxation ◦  Complexity O(t) §  Linear, because each breaking point can be only removed

from S once §  Quite similar to Graham scan algorithm for convex hull of a

finite points, but no sorting needed 14

Modeling

15

Direct PWL Formulation

16

njlinearpiecewiseareWhere

nj

mitoSubject

Minimize

xcuxl

bxa

xc

jj

jjj

ij

n

jij

j

n

jj

,,1,)(

,,1,

,,1,

)(

1

1

…

…

…

=

=≤≤

==∑

∑

=

=

Commonly Used Approachλ Formulation

}  One variable for each piece of PWL function ◦  Suppose variable x have a PWL function c(x)

defined by t points, (p1, c1), (p2, c2), …, (pt, ct) ◦  We introduce variables λ1,λ2,…, λt for the points,

such that

17

)4(,...,1,0

)3(1

)2()(

)1(

1

1

1

tjfor

cxc

px

j

t

jj

j

t

jj

j

t

jj

=≥

=∑

∑=

∑=

=

=

=

λ

λ

λ

λ

Convex PWL Functions }  Translation ◦  Substitute c(x) in the direct formulation by using

equations (2) ◦  Add equations (1) and (3), and inequalities (4)

}  Pure LP ◦  Size can be much bigger, if PWL functions have a lot

of pieces ◦  Direct handling of PWL by extending simplex may

have a big advantage

18

Non-Convex PWL Functions }  MIP formulations (SOS2 and binary) ◦  Substitute c(x) in the direct formulation by using equations (2) ◦  Add equations (1) and (3), and inequalities (4) ◦  Add either SOS2 constraints or binary variables

§  SOS2 formulation: add SOS2 constraints on λ variables §  Binary formulation: add binaries y1, y2,…,yt-1 with following constraints λ1 ≤ y1 λj ≤ yj-1 + yj, for j = 2, …, t-1 λt ≤ yt-1 Σ yj = 1

}  Relaxations ◦  Direct formulation: replace c(x) with r(x) ◦  Relaxations of the three formulations have the same objective

value ◦  Model size

§  Binary: biggest; SOS2: smaller; Direct: smallest (could be much smaller)

19

Simplex For convex PWL Formulation

20

Primal Simplex for LP }  Important aspects ◦  Crash basis and phase I ◦  Pricing to find enter variable ◦  Ratio test to find leaving variable ◦  Linear algebra to compute and update basis

factorization and to solve equations (ftran, btran)

21

Primal Simplex for PWL LP }  Important aspects ◦  Crash basis and phase I §  Pretty much the same ◦  Pricing to find enter variable

§  Need to consider both directions for a nonbasic variable at a breaking point of PWL function

◦  Ratio test to find leaving variable §  Different §  Longer step ◦  Linear algebra to compute and update basis

factorization and to solve equations (ftran, btran) §  Pretty much the same

22

Ratio Test of PWL Primal Simplex

}  Example, consider the dictionary for ratio test x1 = 2.5 + 0.5 x3 + ∑ aj1 xj x2 = 10 + x3 + ∑ aj2 xj

x1: basic, PWL with points (0, 0), (1, 1), (2, 3), (3, 6), (4, 10) x2 : basic, no PWL, lb = 0, ub = inf x3 : nonbasic at 0, entering with reduced cost -1.7, PWL with points (0, 0), (2, 1), (4, 4), (6, 10) }  Ratio test with a shorter step

x1 : (2.5-2)/0.5 = 1 x2: (10-0)/1 = 10 ◦  x3 enters the basis with step length 1, x1 leaves. It can be

mapped to the λ formulation ◦  No need to consider objective or reduced costs

23

Ratio Test of PWL Primal Simplex

}  Ratio test with a longer step x1 : (2.5-2)/0.5 = 1, (2.5 – 1)/0.5 =3, (2.5-0)/0.5 = 5 x2: (10-0)/1 = 10 (> 5, eliminated) x3 : 2, 4, 6 (> 5 eliminated) Possible steps 1, 2, 3, 4, 5 with corresponding unit

objective changes -1.7, -1.2 (-1.7+0.5), -0.2(-1.2+1), 0.3(-0.2+0.5), 1.8(0.3+1.5) ◦  x3 enters the basis with step length 3, x1 leaves to 1. This is

equivalent to 3 iterations for the λ formulation ◦  We can use the median algorithm to do the ratio test. Its

complexity is O(m log(t)), which is usually much cheaper than solving (ftran, btran)

24

Dual Simplex for LP }  Important aspects ◦  Crash basis and phase I §  Bounded variables don’t matter much, cheaper ◦  Pricing to find leaving variable ◦  Ratio test to find entering variable §  Basically solve an LP with a single constraint on

variables with possible lower and upper bounds ◦  Linear algebra to compute and update basis

factorization and to solve equations (ftran, btran)

25

Dual Simplex for PWL LP }  Important aspects ◦  Crash basis and phase I §  Different, not important, just do something simple ◦  Pricing to find leaving variable §  Pretty much the same ◦  Ratio test to find leaving variable §  Solve a PWL LP with a single constraint on variables with

possible lower and upper bounds §  Median algorithm ◦  Linear algebra to compute and update basis factorization

and to solve equations (ftran, btran) §  Pretty much the same ◦  Values for basic variables may change from one piece to

another for each iteration §  It needs to address

26

Preliminary Computational Test }  Model set ◦  100 easy models from our LP set, many from netlib ◦  Replace objective of every variable with a PWL function with

10 pieces }  Method ◦  Primal simplex ◦  No presolve

§  To avoid removing pieces }  Comparison between direct PWL and λ formulations ◦  All 100 models:

§  5.91X fewer iterations, 3.28X faster (too fast to be reliable) ◦  21 models with > 1s runtime

§  5.89X fewer iterations, 7.13X faster ◦  Still some issues in the code

§  Possible further fewer iterations and better runtime performance

27

Advantages of PWL Simplex }  Faster iteration ◦  Because of smaller model size §  Especially for dual simplex and for primal devex or

steepest edge pricing

}  Fewer iterations ◦  One iteration often equals several iterations on the λ formulation

28

Branch-and-bound For non-convex PWL

Formulation

29

MIP Bound-and-Bound Solver }  Important aspects ◦  Presolve §  Stronger primal reductions like bound strengthening §  Harder for dual reductions ◦  Solve relaxation ◦  Select variable to branch ◦  Cutting planes ◦  Heuristics §  Any relaxation solution is MIP feasible

30

Solving Relaxation }  Root relaxation ◦  Replace c(x) with convex relaxation r(x) and solve the

relaxation, the objective value is the same as that for the λ formulation before adding cuts

}  Node relaxations ◦  Use changed bounds to update r(x) ◦  It is primal feasible for one branch ◦  Warm start with primal or dual depending on the

situation

}  Comment ◦  The advantages of PWL simplex carry over

31

Selecting Variable }  Variable Candidates ◦  Every variable x with r(x) < c(x) is a candidate ◦  Basic (not at a breaking point) vs nonbasic (at a breaking

point)

}  Selecting ◦  Pick x with max c(x) – r(x)? ◦  Extend current pseudo, strong and reliability branching?

}  Where to split ◦  Not good at non-breaking point ◦  Easy for nonbasic variable ◦  Two choices for basic variable, left or right breaking point

32

Choosing Split (1) }  Pick right breaking point to x = a ◦  c(x) – r(x) doesn’t change at x = a

33

Choosing Split (2) }  Pick left breaking point to x = a ◦  c(x) – r(x) becomes zero at x = a

34

Cutting Planes }  Primal feasible regions ◦  Same for the relaxation and the original models ◦  Impossible to have primal cuts

}  Cutting planes with dual arguments ◦  Possible and how?

35

Possible Future Work

36

Separable PWL Coefficients }  Much more useful for approximating

nonlinear programs }  The λ formulation ◦  Model size may be too big

}  Extend simplex ◦  To handle the convex case directly to have faster

iterations and fewer iterations ◦  Harder

}  Extend branch-and-bound algorithm ◦  To handle the non-convex case directly

37

Non-separable PWL Function }  Separating ◦  We can separate Q by factorization and

introduction of new variables

}  Mixed-integer models for nonseparable piecewise linear optimization: Unifying framework and extensions, Vielma, Ahmed and Nemhauser, 2009

38

Computational Tests }  Translation the λ models back }  Approximate convex QP and non-convex QP }  Ask you and our customers to try and to send us

models }  Ask you for ideas

39

Thank You