QUADRATIC AND PARAMETRIC QUADRATIC
OPTIMIZATION
AN INTERIOR POINT APPROACH TO QUADRATIC AND
PARAMETRIC QUADRATIC OPTIMIZATION
By
Oleksandr Romanko, B.Sc., M.A.
A Thesis
Submitted to the School of Graduate Studies
in Partial Fulfilment of the Requirements
for the Degree
Master of Science
McMaster University
c© Copyright by Oleksandr Romanko, August 2004
MASTER OF SCIENCE (2004) McMaster University(Computing and Software) Hamilton, Ontario
TITLE: An Interior Point Approach to Quadratic andParametric Quadratic Optimization
AUTHOR: Oleksandr Romanko, B.Sc., M.A.
SUPERVISOR: Dr. Tamas Terlaky
NUMBER OF PAGES: x, 123
ii
Abstract
In this thesis sensitivity analysis for quadratic optimization problems is stud-
ied. In sensitivity analysis, which is often referred to as parametric optimization
or parametric programming, a perturbation parameter is introduced into the op-
timization problem, which means that the coefficients in the objective function
of the problem and in the right-hand-side of the constraints are perturbed. First,
we describe quadratic programming problems and their parametric versions. Sec-
ond, the theory for finding solutions of the parametric problems is developed. We
also present an algorithm for solving such problems.
In the implementation part, the implementation of the quadratic optimiza-
tion solver is made. For that purpose, we extend the linear interior point package
McIPM to solve quadratic problems. The quadratic solver is tested on the prob-
lems from the Maros and Meszaros test set.
Finally, we implement the algorithm for parametric quadratic optimization.
It utilizes the quadratic solver to solve auxiliary problems. We present numerical
results produced by our parametric optimization package.
iii
Acknowledgments
The thesis was written under the guidance and with the help of my super-
visor, Prof. Tamas Terlaky. His valuable advices and extended knowledge of the
area helped me to do my best while working on the thesis. I am very grateful to
Mr. Alireza Ghaffari Hadigheh and Ms. Xiaohang Zhu for their contribution to
my thesis. My special thanks are to the members of the examination committee:
Dr. Ryszard Janicki (Chair), Dr. Antoine Deza, Dr. Jiming Peng and Dr. Tamas
Terlaky.
It would not be possible to complete this thesis without support and help of
all members of the Advanced Optimization Laboratory and the Department of
Computing and Software. I would like to acknowledge McMaster University for
the Ashbaugh Graduate Scholarship. I am thankful to the Canadian Operational
Research Society (CORS) for the 2004 Student Paper Competition Prize that
encouraged me in my work on the thesis.
Finally, I appreciate the support of my friends and parents and thankful to
them for their patience and understanding.
iv
Contents
List of Figures ix
List of Tables x
1 Introduction 1
1.1 Linear and Quadratic Optimization Problems . . . . . . . . . . . 1
1.2 Parametric Optimization . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Origins of Parametric Optimization: Portfolio Example . . . . . . 4
1.4 Parametric Optimization: DSL Example . . . . . . . . . . . . . . 7
1.5 Outline of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . 9
2 Interior Point Methods for Quadratic Optimization Problems 11
2.1 Quadratic Optimization Problems . . . . . . . . . . . . . . . . . . 11
2.2 Primal-Dual IPMs for QO . . . . . . . . . . . . . . . . . . . . . . 13
2.2.1 The Central Path . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.2 Computing the Newton Step . . . . . . . . . . . . . . . . . 14
2.2.3 Step Length . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.4 A Prototype IPM Algorithm . . . . . . . . . . . . . . . . . 17
2.3 Homogeneous Embedding Model . . . . . . . . . . . . . . . . . . . 18
v
2.3.1 Description of the Homogeneous Embedding Model . . . . 19
2.3.2 Finding Optimal Solution . . . . . . . . . . . . . . . . . . 20
2.4 Computational Practice . . . . . . . . . . . . . . . . . . . . . . . 22
2.4.1 Solving Homogeneous Embedding Model with Upper Bound
Constrains . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.4.2 Step Length . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.3 Recovering Optimal Solution and Detecting Infeasibility . 29
2.5 Solving the Newton System of Equations . . . . . . . . . . . . . . 30
2.5.1 Solving the Augmented System: Shermann-Morrison For-
mula . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.5.2 Augmented System vs. Normal Equations . . . . . . . . . 31
3 Implementation of Interior Point Methods for Quadratic Opti-
mization Problems 33
3.1 General Interface of the McIPM Package . . . . . . . . . . . . . . 34
3.1.1 Reading QO Data Files . . . . . . . . . . . . . . . . . . . . 35
3.1.2 Preprocessing and Postprocessing in Quadratic Optimization 38
3.2 Structure of the McIPM Package . . . . . . . . . . . . . . . . . . 41
3.3 Solving the Newton System . . . . . . . . . . . . . . . . . . . . . 43
3.3.1 Sparse Linear Algebra Package McSML . . . . . . . . . . . 43
3.3.2 Sparse Linear Algebra Package LDL . . . . . . . . . . . . 46
3.4 Computational Algorithm for QO . . . . . . . . . . . . . . . . . . 47
3.4.1 Predictor-Corrector Strategy . . . . . . . . . . . . . . . . . 47
3.4.2 Self-Regular Functions and Search Directions . . . . . . . . 48
3.4.3 Stopping Criteria . . . . . . . . . . . . . . . . . . . . . . . 50
vi
3.5 Numerical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4 Parametric Quadratic Optimization 59
4.1 Origins of Quadratic and Linear Parametric Optimization and the
Existing Literature . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.2 Properties of Convex QO Problems . . . . . . . . . . . . . . . . . 62
4.3 The Optimal Value Function in Simultaneous Perturbation Sensi-
tivity Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.4 Properties of the Optimal Value Function . . . . . . . . . . . . . . 65
4.4.1 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . 66
4.4.2 Derivatives, Invariancy Intervals, and Transition Points . . 74
4.4.3 Computational Algorithm . . . . . . . . . . . . . . . . . . 83
4.5 Simultaneous Perturbation in Linear Optimization . . . . . . . . . 85
5 Implementation of Parametric Quadratic Optimization 89
5.1 Illustrative Example . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.2 Implementing the Parametric Algorithm for QO . . . . . . . . . . 92
5.2.1 Implementation Algorithm . . . . . . . . . . . . . . . . . . 92
5.2.2 Determining Optimal Partitions and Support Sets . . . . . 95
5.3 Structure of the McParam Package . . . . . . . . . . . . . . . . . 97
5.3.1 McParam Arguments and Output . . . . . . . . . . . . . . 97
5.3.2 McParam Flow Chart . . . . . . . . . . . . . . . . . . . . 100
5.4 Computational Results . . . . . . . . . . . . . . . . . . . . . . . . 100
5.5 Analysis of Results . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6 Conclusions and Further Work 109
vii
A McIPM and McParam Options 113
B Maros and Meszaros Test Set 115
Bibliography 119
viii
List of Figures
1.1 Mean-Variance Efficient Portfolio Frontier . . . . . . . . . . . . . 6
3.1 General Structure of the McIPM Package . . . . . . . . . . . . . . 41
3.2 General Structure of the McIPM Quadratic Solver . . . . . . . . . 42
4.1 The Invariancy Intervals and Transition Points . . . . . . . . . . . 71
4.2 The Optimal Value Function on Invariancy Interval . . . . . . . . 73
4.3 Neighboring Invariancy Intervals . . . . . . . . . . . . . . . . . . . 81
5.1 The Optimal Value Function for Illustrative Problem . . . . . . . 91
5.2 Optimal Partition Determination Counterexample . . . . . . . . . 96
5.3 Flow Chart of the McParam Package . . . . . . . . . . . . . . . . 101
5.4 Optimal Value Function for Perturbed lotschd QO Problem . . . 103
5.5 Optimal Value Function for Perturbed lotschd LO Problem . . . 103
5.6 Optimal Value Function for Perturbed qsc205 QO Problem . . . 104
5.7 Optimal Value Function for Perturbed qsc205 LO Problem . . . . 105
5.8 Optimal Value Function for Perturbed aug3dc QO Problem . . . 106
ix
List of Tables
3.1 McIPM Performance on QO Test Set (I) . . . . . . . . . . . . . . 55
3.2 McIPM Performance on QO Test Set (II) . . . . . . . . . . . . . . 56
3.3 McIPM Performance on QO Test Set (III) . . . . . . . . . . . . . 57
3.4 McIPM Performance on Difficult QO Problems . . . . . . . . . . 58
5.1 Transition Points, Invariancy Intervals, and Optimal Partitions
for the Illustrative Problem . . . . . . . . . . . . . . . . . . . . . 90
5.2 McParam Output for Perturbed lotschd Problem (QO Formula-
tion) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
5.3 McParam Output for Perturbed lotschd Problem (LO Formulation)102
5.4 McParam Output for Perturbed qsc205 Problem (QO Formulation)104
5.5 McParam Output for Perturbed qsc205 Problem (LO Formulation)105
5.6 McParam Output for Perturbed aug3dc Problem (QO Formulation)106
B.1 The Maros and Meszaros QO Test Set (I) . . . . . . . . . . . . . 116
B.2 The Maros and Meszaros QO Test Set (II) . . . . . . . . . . . . . 117
B.3 The Maros and Meszaros QO Test Set (III) . . . . . . . . . . . . 118
x
Chapter 1
Introduction
Optimization is a technique used for searching extremum of a function. This
term generally refers to mathematical problems where the goal is to minimize
(maximize) an objective function subject to some constraints. Depending on the
nature and the form of the objective function and the constraints, continuous
optimization problems are classified to linear, quadratic, conic and general non-
linear. Correspondingly, we distinguish the research areas of linear optimization,
quadratic optimization, etc.
1.1 Linear and Quadratic Optimization
Problems
Linear optimization, where the objective function and the constraints are linear,
is the most extensively studied branch in the optimization field. History of linear
optimization was started in the 40’s of the 20th century and has gained wide
attention in the scientific community after the development of Simplex Method
by George Dantzig in the 50’s. The Simplex algorithm and its extensions were
extensively studied since then and did not have practical competitors until the
discovery of Interior Point Methods (IPMs) in the middle 80’s. The milestone
1
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
work of Karmarkar in 1984 started the era of IPMs, the major rivals of Simplex
Methods, that usually outperform Simplex algorithms when solving large-scale
problems.
The main conceptual feature that differentiates IPMs from the Simplex
Method is how they search for an optimal solution. In Simplex Methods the
optimal solution is searched by moving from one vertex of the feasible region
to another until an optimal (minimum or maximum) solution is found, while in
IPMs the problem is solved by following a path inside the feasible region of the
problem that leads to optimality. IPMs allow solving large sparse optimization
problems efficiently in polynomial time. Please consult [24] for the comprehensive
description of Simplex methods and [26] for the theoretical and algorithmic back-
ground on IPMs for linear optimization. We will explore many nice properties
of IPMs that allow us to derive strong theoretical results.
In this thesis we will study quadratic optimization problems. Quadratic
Optimization (QO) problems, where the objective function is convex quadratic
while the constrains remain linear, are widely used by scientists for more than
a half-century. This class of optimization problems gained much attention in
the middle of the 50’s. A series of developments followed with the appearance
of complementarity theory and the formulation of the linear complementarity
problem.
The class of quadratic optimization problems gained its importance among
business community after Markowitz [17] used it for conducting mean-variance
analysis of investment portfolios. In the following sections we would discuss the
Markowitz model [17] in more details. Other well-known applications of QO
are coming from engineering, see e.g., [31] and other recent papers on optimal
multi-user spectrum management for Digital Subscriber Lines (DSL). Quadratic
2
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
optimization problems appear naturally when we want to minimize a variation
(or variance) of some quantity. In general, QO is a natural extension of linear
optimization (LO) and most of the solution methods developed for LO were
extended to QO as well.
1.2 Parametric Optimization
In many practical applications we are interested not only in the value of the
optimal solution of an optimization problem, but also in its sensitivity. In other
words, it is necessary to know how sensitive the solution is to data perturbations.
Knowing sensitivity of the solution allows to adjust the constraints (such as
resource constraints or budget constraints) or the coefficients of the objective
function (such as individual preferences or parameters of a production process)
to meet the modelling objectives in a better way and to get a ”better” solution
to the problem.
Let us look at a couple of definitions of sensitivity analysis in order to
understand its meaning better. The first definition from the Financial Glos-
sary [9] grasps the general picture and actually applies to any project or model:
”Investigation of how projected performance varies along with changes in the
key assumptions on which the quantitative estimate of future economic or finan-
cial performance are based” (in other words how the performance of the model
varies with changes in the values of important variables). The second definition
borrowed from the Mathematical Programming Glossary [11] is mathematically
more precise: ”The concern with how the solution changes if some changes are
made in either the data or in some of the solution values.”
If we ”project” both definitions to the optimization problem, we can see
that sensitivity analysis is a technique to determine how the optimal objective
3
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
function value varies with the change in one or more coefficients of the objective
function or right-hand side values of the problem constraints. When changes
occur in many coefficients this type of analysis is also referred to as sensitivity
analysis in many cases, but it is usually known as parametric analysis. According
to [11] ”parametric analysis is concerned with larger changes in parameter values
that affect the data in the mathematical program, such as a cost coefficient or
resource limit.”
Generally speaking, in parametric analysis a parameter λ is introduced into
the original optimization problem transforming it to the parametric one:
min f(x) φ(λ) = min f(x, λ)s.t. gi(x) ≤ 0, ∀ i =⇒ s.t. gi(x, λ) ≤ 0, ∀ i.
The goal of this thesis is to use the Interior Point Methods framework for
developing a methodology that allows finding an optimal solution vector x∗(λ)
and the optimal value function φ(λ) without discretization of the parameter space
λ and without solving the (quadratic) optimization problem at each discretiza-
tion point. This type of sensitivity analysis is often referred to as parametric
programming or parametric optimization. For the reader not to be confused,
we will use the terms sensitivity analysis, parametric analysis and parametric
optimization interchangeably throughout the thesis.
1.3 Origins of Parametric Optimization:
Portfolio Example
Mean-variance portfolio models, which are based on investor’s utility maximiza-
tion, can be formulated as optimization problems and, more precisely, as para-
metric quadratic optimization problems. Consider a universe of n assets. In such
problems the historical data known is c = (c1, ..., cn)T and Q = [σij], where ci is
4
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
the expected return for asset i and σij is the covariance of returns for assets i and
j. So, c is the vector of expected returns and Q is the n× n variance-covariance
matrix of asset returns (Q is a symmetric positive semidefinite matrix which fol-
lows from the properties of variance-covariance matrices). Let x = (x1, ..., xn)T
denote the vector of asset holdings. In this case the expected return of the
portfolio x is cT x and its variance is σ2 = xT Qx.
Markowitz [17] defined a portfolio to be efficient if for some fixed level of
expected return no other portfolio gives smaller variance (risk). Equivalently,
an efficient portfolio can be defined as the one for which at some fixed level of
variance (risk) no other portfolio gives larger expected return. The determina-
tion of the efficient portfolio frontier in the Markowitz mean-variance model is
equivalent to solving the following parametric QO problem due to Farrar [8]
min −λcT x + 12xT Qx
s.t. Ax = bx ≥ 0.
(1.3.1)
Here, λ > 0 is an investor’s risk aversion parameter. The linear constraints
Ax = b can represent budget constraints, bounds on asset holdings, etc. Non-
negativity constraints x ≥ 0 are short-sale constraints (non-negative asset hold-
ings). If λ is allowed to vary, (1.3.1) becomes a parametric optimization problem.
Furthermore, in this case solutions of the optimization problem for different val-
ues of λ trace the so-called efficient frontier in the mean-variance space. When
λ is large, indicating high tolerance to risk, the solution of (1.3.1) is a portfolio
with the highest expected return. When λ becomes smaller, the solution of the
optimization problems will emphasize the minimization of the portfolio variance
and put little weight on the maximization of the expected portfolio return.
If we plot the solutions of a particular instance of problem (1.3.1) for dif-
ferent values of λ in the expected return – standard deviation coordinates, they
5
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
3.5 4 4.5 5 5.5 6 6.5 7 7.5 84
5
6
7
8
9
10
11
12
Standard Deviation (percent per year)
Exp
ecte
d P
ortfo
lio R
etur
n (p
erce
nt p
er y
ear)
Efficient Portfolio FrontierCorner PortfoliosIndividual Stocks
λ=0
λ=∞
Figure 1.1: Mean-Variance Efficient Portfolio Frontier
trace the mean-variance efficient frontier (Figure 1.1). The mean-variance effi-
cient frontier is known to be the graphical depiction of the Markowitz efficient
set of portfolios and represents the boundary of the set of feasible portfolios that
have the maximum return for a given level of risk. Portfolios above the frontier
cannot be achieved. It was noticed that there exist some corner portfolios on the
frontier, and in between this corner portfolios the frontier is piecewise quadratic.
Figure 1.1 shows the efficient frontier in the mean-standard deviation space in
order to be consistent with the existing literature. Note that, the efficient frontier
is a piecewise quadratic function in the mean-variance space.
From the observations it seems likely that we do not need to find a solution
6
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
of the parametric problem for every value of λ, but instead it is necessary to
determine the corner portfolios only and ”restore” the efficient frontier between
them by calculating the quadratic function.
1.4 Parametric Optimization: DSL Example
One of the recent examples of QO problems is a model of optimal multi-user
spectrum management for Digital Subscriber Lines (DSL). Considering the be-
havior of this model under perturbations, we get a parametric quadratic problem.
Moreover, the DSL model can have simultaneous perturbation of the coefficients
in the objective function and in the right-hand side of the constraints.
Let us consider a situation when M users are connected to one service
provider via telephone line (DSL), where M cables are bundled together into the
single one. The total bandwidth of the channel is divided into N subcarriers
(frequency tones) that are shared by all users. Each user i tries to allocate his
total transmission power P imax to subcarriers to maximize his data transfer rate
N∑
k=1
pik = P i
max.
The bundling causes interference between the user lines at each subcarrier k =
1, . . . , N , that is represented by the matrix Ak of cross-talk coefficients. In addi-
tion, there is a background noise σk at frequency tone k.
Current DSL systems use fixed power levels. In contrast, allocating each
users’ total transmission power among the subcarriers ”intelligently” may result
in higher overall achievable data rates. In noncooperative environment user i
allocates his total power P imax selfishly across the frequency tones to maximize
his own rate. The DSL power allocation problem can be modelled as a mul-
tiuser noncooperative game. Nash equilibrium points of the noncooperative rate
7
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
maximization game correspond to optimal solutions of the following quadratic
minimization problem:
minN
∑
k=1
σkeT pk +
1
2
N∑
k=1
pTk Akpk
s.t.N
∑
k=1
pik = P i
max, i = 1, . . . ,M
pk ≥ 0, k = 1, . . . , N,
where pk = (p1k, . . . , p
Mk )T .
Engineers always look at the behavior of such models under different con-
ditions. One of the parameters that influences the model is noisiness of the
environment where the telephone cable is laid. The noisiness depends on the
material the cable is made of, on the type of insulation used, etc. This noisiness,
in turn, determines the background noise to the line σk. Users of telephone lines
residing in noisy environments may get their total transmission power Pmax in-
creased (i.e., get a more expensive modem) to improve the signal to noise ratio
of the line. Such setup results in the parametric model:
minN
∑
k=1
(σk + λ△σk)eT pk +
1
2
N∑
k=1
pTk Akpk
s.t.N
∑
k=1
pik = P i
max + λ△P imax, i = 1, . . . ,M
pk ≥ 0, k = 1, . . . , N
(1.4.1)
Parametric QO problem (1.4.1) represents a model with the noisiness pa-
rameter λ. The same parameter λ appears in the objective function and in the
right-hand side of the constraints. The parametric model allows to look at the
equilibria when the background noise and the total transmission power changes
as λ varies. This formulation, for instance, can help answering such questions
as: what happens if the background noise to the line increases two times faster
than the total transmission power available to users.
8
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
1.5 Outline of the Thesis
The thesis describes the theoretical background of both quadratic optimization
and parametric quadratic optimization as well as implementations of solution
techniques for them into software packages. This predetermines the following
organization of the thesis.
In the current Chapter 1, we outline the history and the use of quadratic
optimization techniques. In addition, we introduce the concept of parametric
quadratic optimization and provide an example of portfolio problem which is
formulated as parametric quadratic optimization problem. We also consider an
engineering model that is formulated as simultaneous perturbation parametric
optimization problem. Finally, the outline of the thesis is provided.
Chapter 2 contains the background of using interior point methods for
quadratic optimization. We use the homogenous embedding model and self-
regular proximity functions. Chapter 2 also contains all theoretical results nec-
essary for the implementation.
Chapter 3 is devoted to the implementation of the interior point method
algorithm outlined in Chapter 2. We describe the algorithm itself, problem
specification formats, preprocessing and postprocessing, as well as the core of the
methodology – sparse linear system solvers. Finally, we provide computational
results to benchmark our software with existing quadratic solvers.
In Chapter 4 we make a link from the quadratic optimization to its para-
metric counterpart. We provide the necessary background, prove some proper-
ties of such problems and suggest an algorithm for solving parametric quadratic
optimization problems when the perturbation occurs simultaneously at the right-
hand side of the constraints and in the objective function. In addition, we spe-
cialize our results to parametric linear optimization.
9
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Chapter 5 describes the implementation of the parametric programming
algorithm in the MATLAB environment and provide an illustrative example of
solving parametric problem. Chapter 5 also presents our computational results.
Finally, Chapter 6 contains concluding remarks and suggestions for future
work.
10
Chapter 2
Interior Point Methods for
Quadratic Optimization
Problems
In this chapter we extend our introductory knowledge about quadratic optimiza-
tion (QO) problems, describe their properties and solution techniques. As we
already know, QO problem consists of minimizing a convex quadratic objective
function subject to linear constraints. In addition to showing problem formu-
lations, we review the duality theory for QO problems. Finally, Interior Point
Methods (IPMs) for solving these problems are described. Results presented in
this chapter are mainly based on [2], [35] and [36].
2.1 Quadratic Optimization Problems
A primal convex QO problem is defined as:
min cT x + 12xT Qx
(QP ) s.t. Ax = bx ≥ 0,
(2.1.1)
where Q ∈ IRn×n is a symmetric positive semidefinite matrix, A ∈ IRm×n,
rank(A) = m, c ∈ IRn, b ∈ IRm are fixed data and x ∈ IRn is an unknown
11
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
vector.
The Wolfe Dual of (QP ) is given by
max bT y − 12uT Qu
(QD) s.t. AT y + z − Qu = cz ≥ 0,
(2.1.2)
where z, u ∈ IRn and y ∈ IRm are unknown vectors. Note that when Q = 0 we
get a Linear Optimization (LO) problem.
The feasible regions of (QP ) and (QD) are denoted by
QP = {x : Ax = b, x ≥ 0},
QD = {(u, y, z) : AT y + z − Qu = c, z, u ≥ 0},
and their associated optimal solution sets are QP∗ and QD∗, respectively. It is
known that for any optimal solution of (QP ) and (QD) we have Qx = Qu and
xT z = 0, see e.g., Dorn [7]. It is also known from [7] that there are optimal
solutions with x = u. Since we are only interested in the solutions where x = u,
therefore, u will be replaced by x in the dual problem. The duality gap cT x +
xT Qx − bT y = xT z being zero is equivalent to xizi = 0 for all i ∈ {1, 2, . . . , n}.This property of the nonnegative variables x and z is called the complementarity
property.
Solving primal problem (QP ) or dual problem (QD) is equivalent to solving
the following system, which represents the Karush-Kuhn-Tucker (KKT) optimal-
ity conditions [33]:
Ax − b = 0, x ≥ 0,AT y + z − Qx − c = 0, z ≥ 0,
xT z = 0,(2.1.3)
where the first line is the primal feasibility, the second line is the dual feasibility,
and the last line is the complementarity condition. The complementarity condi-
12
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
tion can be rewritten as xz = 0, where xz denotes the componentwise product
of the vectors x and z. System (2.1.3) is referred to as the optimality conditions.
Let X = diag(x1, . . . , xn) and Z = diag(z1, . . . , zn) be the diagonal matrices
with vectors x and z forming the diagonals, respectively. For LO the Goldman-
Tucker Theorem states that there exists a strictly complementary optimal solution
(x, y, z) if both the primal and dual problems are feasible. For LO, the feasible
primal-dual pair (x, y, z) is strictly complementary if xizi = 0 and xi + zi > 0
for all i = 1, . . . , n. Equivalently, strict complementarity can be characterized by
xz = 0 and rank(X) + rank(Z) = n.
Unlike in LO, where strictly complementary optimal solution always exists,
for QO the existence of such solution is not ensured. Instead, a maximally
complementary solution can be found. A pair of optimal solutions (x, y, z) for
the QO problem is maximally complementary if it maximizes rank(X)+rank(Z)
over all optimal solution pairs. As we see in Chapter 4, this leads to tri-partition
of the optimal solution set.
2.2 Primal-Dual IPMs for QO
Primal-dual IPMs are iterative algorithms that aim to find a solution satis-
fying the optimality conditions (2.1.3). IPMs generate a sequence of iterates
(xk, yk, zk), k = 0, 1, 2, . . . that satisfy the strict positivity (interior point) condi-
tion xk > 0 and zk > 0, but feasibility (for infeasible IPMs) and optimality are
reached as k goes to infinity. In this thesis we are concerned about feasible IPM
methods which produce a sequence of iterates where the following interior point
condition (IPC) holds for every iterate (x, y, z)
Ax = b, x > 0,AT y + z − Qx = c, z > 0.
(2.2.1)
13
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
2.2.1 The Central Path
We perturb the complementarity condition in the optimality conditions (2.1.3)
asAx = b, x > 0,
AT y + z − Qx = c, z > 0,Xz = µe,
(2.2.2)
where µ > 0 and e = (1, . . . , 1)T . It is obvious that the last nonlinear equation
in (2.2.2) becomes the complementarity condition for µ = 0.
A desired property of system (2.2.2) is the uniqueness of its solution for
each µ > 0. The following theorem [12] shows such conditions.
Theorem 2.2.1 System (2.2.2) has a unique solution for each µ > 0 if and only
if rank(A) = m and the IPC holds for some point.
When µ is running through all positive numbers, the set of unique solutions
(x(µ), y(µ), z(µ)) of (2.2.2) define the so-called primal-dual central path. The sets
{x(µ) |µ > 0} and {(y(µ), z(µ)) |µ > 0} are called primal central path and dual
central path respectively.
One iteration of primal-dual IPMs consists of taking a Newton step applied
to the central path equations (2.2.2) for a given µ. The central path stays in the
interior of the feasible region and the algorithm approximately follows it towards
optimality. For µ → 0 the set of points (x(µ), y(µ), z(µ)) gives us a maximally
complementary optimal solution of (QP) and (QD).
2.2.2 Computing the Newton Step
Newton’s method is used to solve the system (2.2.2) iteratively. At each step
we need to compute the direction (△x,△y,△z). A new point in the computed
14
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
direction (x + △x, y + △y, z + △z) should satisfy
A(x + △x) = b,AT (y + △y) + (z + △z) − Q(x + △x) = c,
(x + △x)(z + △z) = µe.
For a given strictly feasible primal-dual pair (x, y, z) we can write this system
with the variables (△x,△y,△z) as
A△x = 0,AT△y + △z − Q△x = 0,
x△z + z△x + △x△z = µe − xz.(2.2.3)
System (2.2.3) is non-linear. Consequently, the Newton step is obtained by
dropping the non-linear term that gives the linearized Newton system
A△x = 0,AT△y + △z − Q△x = 0,
x△z + z△x = µe − xz.(2.2.4)
The linear system (2.2.4) is referred to as the primal-dual Newton system. It has
2n + m equations and 2n + m unknowns. The system has a unique solution if
rank(A) = m.
In matrix form the Newton system (2.2.4) can be written as
A 0 0−Q AT IZ 0 X
△x△y△z
=
00
µe − Xz
. (2.2.5)
Solving system (2.2.5) for △z gives
△z = X−1(µe − Xz − Z△x),
and substituting △z into system (2.2.5) we get(
−Q − D AT
A 0
)(
△x△y
)
=
(
µX−1e − z0
)
, (2.2.6)
where D = X−1Z is a diagonal matrix with Dii = zi
xifor i = 1, . . . , n. System
(2.2.6) is called the augmented system.
15
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
From the first equation of (2.2.6) we can express △x as
△x = (Q + D)−1(AT△y − µX−1e + z).
Then the augmented system reduces to:
A(Q + D)−1AT△y = A(Q + D)−1(µX−1e − z), (2.2.7)
that is often called the normal equation form.
2.2.3 Step Length
In this section we describe how to determine the next iteration point. After solv-
ing the Newton system (2.2.5) we are getting the search direction (∆x, ∆y, ∆z).
This Newton search direction is computed assuming that the step length α is
equal to one. But taking such step can lead to loosing strict feasibility of the
solution as (x+∆x, y+∆y, z+∆z) might be infeasible. Our goal is to keep strict
feasibility, therefore we want to find such an α that the next iteration point is
strictly feasible, i.e.,
(xk+1, yk+1, zk+1) = (xk, yk, zk) + α(∆xk, ∆yk, ∆zk),
with xk+1 > 0 and zk+1 > 0. This can be done in two steps:
• find the maximum possible step size αmax such that
αmax = arg maxα>0
{(
xz
)
+ α
(
△x△z
)
≥ 0
}
,
• as strict feasibility is not warranted by the previous step, we need to use a
damping factor ρ ∈ (0, 1) to choose such α that
α = min{ραmax, 1}.
16
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Finally, we can compute αmax in the following way:
αmaxP = min
∆xj<0{−xj
∆xj
, j = 1, . . . , n},
αmaxD = min
∆zj<0{−zj
∆zj
, j = 1, . . . , n},αmax = min{αmax
P , αmaxD }.
2.2.4 A Prototype IPM Algorithm
Here we present a prototype Primal-Dual Path-Following IPM algorithm for QO
problems in standard form. We suppose that the IPC is satisfied. As we see in
Section 2.3, by using the homogeneous embedding technique we can always con-
struct a QO problem in a way that the IPC holds. After that, we apply Newton’s
method to the central path equations (2.2.2) to get the search direction. The step
length is determined as described in Section 2.2.3 that ensures that the iterate
remains in the interior of the feasible set. IPMs stop when the complementarity
gap is reduced below some predetermined tolerance level or when infeasibility is
detected.
One issue that we have not discussed so far is a strategy for reducing the
parameter µ. We want to follow the central path approximately (as Newton
system is solved disregarding the nonlinear terms) and so we define a proximity
function Ψ(xz, µ) to measure the distance of the current point from the central
path. A proximity parameter δ > 0 defines the bound for the proximity function
which takes the value of zero if the point is on the central path and approaches
infinity if the point approaches the boundary of the nonnegative orthant. There
are many proximity measures defined in the literature, we will describe the fam-
ily of so-called self-regular proximity functions in Section 3.4.2.
17
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
A Prototype IPM Algorithm
input:
a proximity parameter δ > 0;
an accuracy parameter ǫ > 0;
an update parameter 0 < θ < 1;
µ0 = 1, k = 0;
(x0, y0, z0) satisfying x0 > 0, z0 > 0 and Ψ(x0z0, µ0) ≤ δ;
begin
while (xk)T zk ≥ ǫ do
begin
µk = (1 − θ)( (xk)T zk
n);
while Ψ(xkzk, µk) ≥ δ do
begin
solve the Newton system (2.2.5) to find (△xk,△yk,△zk);
determine the step size α;
xk = xk + α△xk, yk = yk + α△yk, zk = zk + α△zk;
end
xk+1 = xk, yk+1 = yk, zk+1 = zk, k = k + 1;
end
end
2.3 Homogeneous Embedding Model
In this section we present a homogeneous algorithm to solve the QO problem.
The homogeneous embedding model is one of the ways to formulate the system
of linear equations associated with the QO problem. Such model for the LO
has been developed by Ye, Todd and Mizuno [32]. Later on it was extended by
18
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Anderson and Ye [2] to monotone complementarity problems (MCP). The QO
problem is a special case of MCP and so, the algorithm applies for QO as well.
2.3.1 Description of the Homogeneous Embedding Model
From the Weak Duality Theorem it follows that system (2.1.3) is equivalent to
Ax − b = 0, x ≥ 0,AT y + z − Qx − c = 0, s ≥ 0,bT y − cT x − xT Qx ≥ 0.
(2.3.1)
A way to solve system (2.3.1) is to introduce a slack variable κ for the last
inequality in the system and a homogenization variable τ . The following system
is a homogeneous reformulation of the QO problem:
Ax − bτ = 0, x ≥ 0, τ ≥ 0,AT y + z − Qx − cτ = 0, z ≥ 0, y free,
bT y − cT x − xT Qx
τ− κ = 0, κ ≥ 0.
(2.3.2)
System (2.3.2) has some attractive properties. First, with τ = 1 and κ = 0
the solution of the system (2.3.2) gives an optimal solution of the QO problem.
Second, it has zero as its trivial solution and can be considered as an MCP
with zero right-hand side vector. Third, considered as an MCP, system (2.3.2)
does not satisfy the IPC. The modification of problem (2.3.2) to a homogeneous
problem having an interior point is (Anderson and Ye [2])
min xT z + τκs.t. Ax − bτ − rP ν = 0,
−AT y + Qx + cτ − rDν − z = 0,
bT y − cT x − xT Qx
τ− rGν − κ = 0,
rTP y + rT
Dx + rGτ = −β,y, ν free, x ≥ 0, τ ≥ 0, z ≥ 0, κ ≥ 0,
(2.3.3)
where ν is an artificial variable added in order to satisfy the IPC, the coefficients
rP , rD and rG represent the infeasibility of the primal and dual initial interior
19
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
points and the duality gap, respectively. These coefficients for a given initial
point (x0 > 0, y0, z0 > 0, τ 0 > 0, κ0 > 0, ν0 > 0) are defined as follows:
rP = (Ax0 − bτ 0)/ν0,rD = (−AT y0 + Qx0 + cτ 0 − z0)/ν0,
rG = (bT y0 − cT x0 − x0T
Qx0
τ0 − κ0)/ν0,β = −rT
P y0 − rTDx0 − rGτ 0.
The homogeneous embedding model (2.3.3) has several advantages. First,
the algorithm based on it does not need to use a ”big-M” penalty parameter [2].
Second, by utilizing the homogeneous embedding model we can avoid using the
two-phase method, where we need to find a feasible interior initial point to start
with, that even might not exist for many problems. It is not difficult to note
that for properly chosen rP , rD and rG the point (x0 = e, y0 = 0, z0 = e, τ 0 = 1,
κ0 = 1, ν0 = 1) is feasible for the embedding model. Third, the homogeneous
embedding model generates a solution sequence converging towards an optimal
solution of the original problem, or it produces an infeasibility certificate for
either (QP ), or (QD), or for both. We can apply the IPM algorithm outlined
in Section 2.2.4 for solving the homogeneous embedding model. The size of the
Newton system for this model is not significantly larger than the size of the
original system. Finally, a small update IPM for solving the homogeneous model
has the iteration bound O(√
n log nǫ).
2.3.2 Finding Optimal Solution
Let us consider a strictly complementary solution (y∗, x∗, τ ∗, ν∗ = 0, z∗, κ∗) of the
homogeneous problem (2.3.3). Our goal is to recover the information about solu-
tions of the original primal (QP ) and dual (QD) QO problems. We distinguish
three cases:
1. If τ ∗ > 0 and κ∗ = 0, then (x∗
τ∗, z∗
τ∗, y∗
τ∗) is a strictly complementary optimal
20
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
solution for (QP ) and (QD).
2. If τ ∗ = 0 and κ∗ > 0, the solution of the embedding model provides a
Farkas certificate for the infeasibility of the dual and/or primal problems
and then:
• if cT x∗ < 0, then the dual problem (QD) is infeasible;
• if −bT y∗ < 0, then the primal problem (QP ) is infeasible;
• if cT x∗ < 0 and −bT y∗ < 0, then both primal (QP ) and dual (QD)
problems are infeasible.
3. If τ ∗ = 0 and κ∗ = 0 then neither a finite solution nor a certificate proving
infeasibility exists. This cannot happen for convex QO problems, because
if no optimal solutions exist, then an infeasibility certificate always exist
[33].
Let us prove the conclusion of case 2. For τ ∗ = 0, κ∗ > 0 and ν∗ = 0 we
haveAx∗ = 0,
AT y∗ − Qx∗ + z∗ = 0.
In addition, the third constraint of (2.3.3) imply that
bT y∗ − cT x∗ − x∗T Qx∗
τ ∗− κ∗ = 0,
and as κ∗ > 0 and x∗T Qx∗
τ∗≥ 0, thus
bT y∗ − cT x∗ > 0,
i.e.,
cT x∗ − bT y∗ < 0.
21
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
The last inequality means that at least one of the components of the left-hand
side, namely cT x∗ or −bT y∗, is strictly less than zero. Let us consider these three
cases separately:
• Consider the case when cT x∗ < 0. Let us assume to the contrary that
a feasible solution (x∗, y, z) for the dual problem such that z ≥ 0 and
AT y − Qx∗ + z = c exists. Then
0 > cT x∗
= (AT y − Qx∗ + z)T x∗
= yT (Ax∗) − x∗T Qx∗ + zT x∗
= zT x∗
≥ 0,
that is a contradiction. Consequently, the dual problem is infeasible.
• Consider the case when −bT y∗ < 0. Let us assume to the contrary that
a feasible solution x for the primal problem such that x ≥ 0 and Ax = b
exists. Then0 > −bT y∗
= (−Ax)T y∗
= xT (−AT y∗)= xT z∗
≥ 0,
that is a contradiction. Consequently, the primal problem is infeasible.
• If both cT x∗ < 0 and −bT y∗ < 0, then by the same reasoning we get both
the primal and dual problems to be infeasible.
2.4 Computational Practice
The majority of QO problems are not given in the standard form (2.1.1). Instead,
problem formulations may include inequality constraints, free variables as well as
lower and upper bounds on the variables. In this section we derive the augmented
22
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
system of the type (2.2.6) and the normal equations of the type (2.2.7) for such
formulations using the homogeneous embedding model. The first step in this
process is the preprocessing stage when problem transformations are applied to
bring a general form problem to the standard form (2.1.1), while upper bounds
for the variables are allowed and treated separately. These transformations are
referred to as preprocessing.
If a QO problem has inequality constraints Ax ≤ b and non-negativity
constraints x ≥ 0, non-negative slack variables x are added to the constraints to
transform the problem to the standard form
(A I)
(
xx
)
= b,
(
xx
)
≥ 0,
where I is the identity matrix of appropriate dimension.
If a problem contains variables xj not restricted to be non-negative (free
variables), then these variables are split to two non-negative variables xj = x+j −
x−j , x+
j ≥ 0, x−j ≥ 0. So, after taking care of inequality constraints and free
variables we get a larger problem conform with the standard form (2.1.1).
Problems including variables that have upper and lower bounds require
more attention. If a variable has a lower bound xi ≥ li, then we shift the
lower bound to zero by substituting the variable xi by xi − li ≥ 0. Appropriate
changes should be made in the vectors c and b: c is substituted by c + Q l and b
by b−A l. If the variable had an upper bound xi ≤ ui, then this upper bound is
shifted as well ui = ui − li. After such shift we have a problem of the same size
with nonnegative variables and possible upper bounds on some variables. The
appropriate back transformation of variables should be made after solving the
QO problems at the postprocessing stage.
For the variables having upper bounds xi ≤ ui, extra slack variables are
added to transform the inequality constraints to equality constraints. This ob-
23
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
viously leads to increase in the number of variables and constraints. So, the last
two sets of constraints of the primal QO problem
min cT x + 12xT Qx
s.t. Ax = b,0 ≤ xi ≤ ui, i ∈ I,0 ≤ xj, j ∈ J ,
(2.4.1)
where A ∈ IRm×n, c, x ∈ IRn, b ∈ IRm, and I and J are disjoint partition of the
index set {1, . . . , n}, can be rewritten as
Fx + s = u, x ≥ 0, s ≥ 0, F ∈ IRmf×n,
where mf = |I| and s ∈ IRmf is the slack vector. Matrix F consists of the
unit vectors associated with the index set I as its rows. Consequently, the QO
problem becomesmin cT x + 1
2xT Qx
s.t. Ax = bFx + s = ux ≥ 0, s ≥ 0.
(2.4.2)
Its dual ismin bT y − uT w − 1
2xT Qx
s.t. AT y − F T w + z − Qx = c,w ≥ 0, z ≥ 0.
(2.4.3)
where y ∈ IRm, w ∈ IRmf and z ∈ IRn. The complementarity gap is
gap = cT x + 12xT Qx − (bT y − uT w − 1
2xT Qx)
= (AT y − F T w + z − Qx)T x − yT (Ax) + wT (Fx + s) + xT Qx= xT z + sT w.
Finally, the optimality conditions for (2.4.2) and (2.4.3) can be written as
Ax = b,Fx + s = u,
AT y − F T w + z − Qx = c,Xz = 0,Sw = 0.
(2.4.4)
24
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
2.4.1 Solving Homogeneous Embedding Model
with Upper Bound Constrains
Addition of the primal equality constraint Fx+s = u to the problem formulation
results in the following homogeneous embedding model that is a straightforward
generalization of (2.3.3):
min xT z + sT w + τκs.t. Ax − bτ − rP1ν = 0,
− Fx + uτ − rP2ν − s = 0,−AT y + F T w + Qx + cτ − rDν − z = 0,
bT y − uT w − cT x − xT Qx
τ− rGν − κ = 0,
rTP1y + rT
P2w + rTDx + rGτ = −β,
y, ν free, w ≥ 0, x ≥ 0, τ ≥ 0, z ≥ 0, s ≥ 0, κ ≥ 0,
(2.4.5)
whererP1 = (Ax0 − bτ 0)/ν0,rP2 = (−Fx0 + uτ 0 − s0)/ν0,rD = (−AT y0 + F T w0 + Qx0 + cτ 0 − z0)/ν0,
rG = (bT y0 − uT w0 − cT x0 − x0T
Qx0
τ0 − κ0)/ν0,β = −rT
P1y0 − rT
P2w0 − rT
Dx0 − rGτ 0.
The objective function of problem (2.4.5) can be also expressed as follows.
Multiplying the first, second, third, forth and fifth equality constraints of (2.4.5)
by yT , wT , xT , τ and ν correspondingly, and summing them up, we get
xT z + sT w + τκ = νβ.
Designing the IPM algorithm for the homogeneous embedding model we
can follow the same reasoning as in Section 2.2.1 and define the central path for
problem (2.4.5). The central path is the set of solutions of the following system
25
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
for all µ > 0:
Ax − bτ − rP1ν = 0,− Fx + uτ − rP2ν − s = 0,
−AT y + F T w + Qx + cτ − rDν − z = 0,
bT y − uT w − cT x − xT Qx
τ− rGν − κ = 0,
rTP1y + rT
P2w + rTDx + rGτ = −β,
Xz = µe,Sw = µe,τκ = µ.
(2.4.6)
The Newton system for (2.4.6) is
0 0 A −b −rP1 0 0 00 0 −F u −rP2 −I 0 0
−AT F T Q c −rD 0 −I 0
bT −uT −cT − 2xT Q
τ
xT Qx
τ2 −rG 0 0 −1rTP1 rT
P2 rTD rT
G 0 0 0 00 S 0 0 0 W 0 00 0 Z 0 0 0 X 00 0 0 κ 0 0 0 τ
△y△w△x△τ△ν△s△z△κ
=
00000
rsw
rxz
rτκ
, (2.4.7)
wherersw = µe − Sw,rxz = µe − Xz,rτκ = µe − τκ.
From the last three lines of (2.4.7) we get
△s = W−1(rsw − S△w),△z = X−1(rxz − Z△x),△κ = τ−1(rτκ − κ△τ).
(2.4.8)
We also have
△ν =rTswe + rT
xze + rτκ
β.
Consequently, system (2.4.7) reduces to
0 0 A −b0 W−1S −F u
−AT F T Q + X−1Z c
bT −uT −cT − 2xT Q
τ
xT Qx
τ2 + κτ
△y△w△x△τ
=
r′
r′swr′xz
r′τκ
, (2.4.9)
26
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
wherer′ = rP1△ν,
r′sw = rP2△ν + W−1rsw,r′xz = rD△ν + X−1rxz,r′τκ = rG△ν + τ−1rτκ.
From the second equation of (2.4.9) we get
△w = WS−1r′sw + WS−1F△x − WS−1u△τ, (2.4.10)
that allows us to reduce the system to
Q+X−1Z+F T WS−1F −AT c − F T WS−1uA 0 −b
−cT − 2xT Q
τ−uT WS−1F bT xT Qx
τ2 + κτ+uT WS−1u
△y△x△τ
=
r′′xz
r′′
r′′τκ
, (2.4.11)
whereD−1 = Q + X−1Z + F T WS−1F,
r′′ = r′,r′′xz = r′xz − F T WS−1r′sw,r′′τκ = r′τκ + uT WS−1r′sw.
Equation (2.4.11) is the augmented system. From the second equation of (2.4.11),
we have
△x = Dr′′xz + DAT△y − D(c − F T WS−1u)△τ.
Consequently, system (2.4.11) reduces to(
ADAT a1
−(a2)T a3
)(
△y△τ
)
=
(
r′′′
r′′′τκ
)
, (2.4.12)
where
a1 = −b − AD(c − F T WS−1u),
−(a2)T = bT − (cT + 2xT Q
τ+ uT WS−1F )DAT ,
a3 = xT Qx
τ2 + κτ
+ uT WS−1u + (cT + 2xT Q
τ+ uT WS−1F )D(c − F T WS−1u),
r′′′ = r′′ − ADr′′xz,
r′′′τκ = r′′τκ + (cT + 2xT Q
τ+ uT WS−1F )Dr′′xz.
From the last equation of (2.4.12), we get
△τ =1
a3(r′′′τκ + (a2)T△y),
27
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
and then system (2.4.12) reduces even further to
[ADAT + aaT ]△y = ξ, (2.4.13)
where a = a1
a3 , a = a2 and ξ = r′′′ − r′′′τκ
a3 a1.
At this point we have two choices: to solve the normal equations (2.4.13)
or to solve the augmented system (2.4.11). First, let us reduce the augmented
system (2.4.11) by rewriting it in the following form:
−Q − X−1Z − F T WS−1F AT −a′
A 0 −b−(a′′)T bT a′′′
△y△x△τ
=
−r′′xz
r′′
r′′τκ
, (2.4.14)
wherea′ = c − F T WS−1u,
a′′ = c + 2Qx
τ+ F T WS−1u,
a′′′ = xT Qx
τ2 + κτ
+ uT WS−1u.
From the last equation of (2.4.14), we get:
△τ =1
a′′′
[
r′′τκ −(
−a′′
b
)T (
△y△x
)
]
, (2.4.15)
and then system (2.4.14) reduces to:
[(
−Q−X−1Z−F T WS−1F AT
A 0
)
+ aaT
](
△y△x
)
=
(
−r′′xz + a′ r′′
τκ
a′′′
r′′ + b r′′τκ
a′′′
)
, (2.4.16)
where
a =
(
a′
b
)
,
aT = 1a′′′
(
−a′′
b
)T
.
Now, we have got two linear systems: the normal equations and the aug-
mented system. We can solve either of them to find the solution of the Newton
system for the homogeneous model and get the search direction for IPMs QO
algorithms. Efficient solution of those systems is the subject of Section 2.5.
28
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
2.4.2 Step Length
We can follow the same reasoning as in Section 2.2.3 to derive analogous results
about the step length for the homogeneous embedding model. After solving the
augmented system (2.4.16), we obtain ∆y and ∆x. Doing back substitutions to
the equations (2.4.15), (2.4.10) and (2.4.8) we get ∆τ , ∆w, ∆z, ∆s and ∆κ. The
maximum acceptable step length α for getting strictly interior point in the next
iteration is determined as follows:
αmaxx = min
∆xj<0{−xj
∆xj
, j = 1, . . . , n},
αmaxz = min
∆zj<0{−zj
∆zj
, j = 1, . . . , n},
αmaxw = min
∆wj<0{−wj
∆wj
, j = 1, . . . , n},
αmaxs = min
∆sj<0{−sj
∆sj
, j = 1, . . . , n},
αmaxτκ = min
∆τ<0, ∆κ<0{−τ
∆τ,−κ
∆κ, j = 1, . . . , n},
αmax = min{αmaxx , αmax
z , αmaxw , αmax
s , αmaxτκ , },
α = min{ραmax, 1}.
(2.4.17)
The damping parameter ρ is usually close to 0.99, in our implementation it has
been chosen equal to 0.995.
2.4.3 Recovering Optimal Solution and Detecting
Infeasibility
This subsection generalizes the results of Section 2.3.2 for the problems with
upper bounds. After getting a strictly complementary solution (y∗, w∗, x∗, τ ∗,
ν∗ = 0, z∗, s∗, κ∗) of the homogenous embedding model (2.4.5) we can either
recover the optimal solution of the original primal (2.4.2) and dual (2.4.3) QO
problems or get an infeasibility certificate. We present the generalized results
below:
1. If τ ∗ > 0 and κ∗ = 0, then (x∗
τ∗, z∗
τ∗, y∗
τ∗, w∗
τ∗, s∗
τ∗) is a strictly complementary
optimal solution for (2.4.2) and (2.4.3).
29
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
2. If τ ∗ = 0 and κ∗ > 0, the solution of the embedding model provides a
primal and/or dual infeasibility certificate for problems (2.4.2) and (2.4.3):
• if cT x∗ < 0, then the dual problem is infeasible;
• if −bT y∗ + uT w∗ < 0, then the primal problem is infeasible;
• if cT x∗ < 0 and −bT y∗ + uT w∗ < 0, then both primal and dual prob-
lems are infeasible.
2.5 Solving the Newton System of Equations
2.5.1 Solving the Augmented System:
Shermann-Morrison Formula
The Sherman-Morrison formula [33] is a formula that allows to compute the
inverse of a modified matrix P + UV T for a given matrix P and matrices U and
V . Here, P is n×n nonsingular matrix and the dimensions of U and V are n×k,
then the Sherman-Morrison formula is given as
(P + UV T )−1 = P−1 − P−1U(1 + V T P−1U)−1V T P−1. (2.5.1)
We can use the Sherman-Morrison formula (2.5.1) to solve the augmented
system (2.4.16) or the normal equations (2.4.13) efficiently. First, we show how
to use the Sherman-Morrison formula for the augmented system. The reasons for
choosing the augmented system over the normal equations approach will become
clear in the next subsection. Let us denote the right-hand side of the system
(2.4.16) by ξ. Furthermore, let U = a, V = a and
P =
(
−Q − X−1Z − F T WS−1F AT
A 0
)
.
Consequently, the solution of the system (2.4.16) is given by(
△y△x
)
= P−1ξ − P−1a(1 + aT P−1a)−1aT P−1ξ. (2.5.2)
30
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
It is known to be inefficient and numerically difficult to compute inverse matrices,
so instead of inverting P to compute P−1ξ and P−1a, we solve the following two
systems of linear equations:P v = ξ,P v = a.
(2.5.3)
Combining the solutions of the two systems above with formula (2.5.2), we get
(
△y△x
)
= v − vaT v
1 + aT v.
To solve the systems of equations (2.5.3) involving the coefficient matrix
P , we need to compute the LDLT factorization of P , as it is explained in the
next section.
2.5.2 Augmented System vs. Normal Equations
Solving the system of linear equations (augmented system or normal equations)
is the most time-consuming part of IPMs. As computational practice shows, the
number of iterations in IPMs is nearly independent of the problem size. So, the
speed of computing one iteration is extremely important and this speed is mainly
related to the costs of solving the linear systems (2.5.3).
As we know from the previous section, we have two choices: to solve the
normal equations system (2.4.13) or to solve the augmented system (2.4.16). In
the first case the effort is related to the costs of the Cholesky factorization of the
symmetric positive definite matrix ADAT [21]. In the second case the symmetric
indefinite matrix(
−Q − X−1Z − F T WS−1F AT
A 0
)
(2.5.4)
needs to be factored by using LDLT factorization [21].
We can use Cholesky decomposition in the normal equations approach as
matrix ADAT is positive definite. We can first perform the Cholesky decompo-
sition LLT = D−1 = −Q−X−1Z −F T WS−1F, to get the system A(LLT )−1AT ,
31
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
which is factored by the second Cholesky decomposition LLT = A(LLT )−1AT .
In practice the permutation and symbolic factorization of the matrices D and
A(LLT )−1AT is performed independently that allows to reduce the fill-in dur-
ing Cholesky decompositions. The attractive property of the normal equations
approach is that it is possible to use Cholesky decomposition, which is usually
faster than the symmetric one, for solving the linear system. Still, the price we
pay is the need to perform two decompositions.
The second approach includes performing the symmetric LDLT factoriza-
tion of the matrix given by (2.5.4). There are again two possibilities. First, we
can use Bunch-Parlett strategy to perform the LDLT factorization, where L is
a lower triangular matrix and D is a non-singular block diagonal matrix with
2× 2 diagonal blocks. Another approach to performing the symmetric factoriza-
tion follows from the observation that matrix (2.5.4) is quasi-definite. For such
matrices there exists an LDLT factorization that D is a diagonal matrix, see
e.g., Vanderbei [29]. In our implementation, we use a factorization based on the
Vanderbei’s technique. You can find more details about that in Chapter 3.
It was shown by Meszaros [21] that factoring the augmented system allows
more freedom in the pivot sequence and is advantageous in the majority of the
cases. Because of that we are using the augmented system approach for our
implementation.
In the linear optimization case usage of the augmented system has no sig-
nificant advantages over the normal equations approach. As the matrix D in the
LO case is diagonal with positive elements, both the augmented system and the
normal equations require factoring only one symmetric nonsingular (quasidefi-
nite or definite, respectively) system. The only difference is that the augmented
system approach, even in the linear case handle dense columns automatically,
while with normal equations dense columns need to be separated in advance.
32
Chapter 3
Implementation of Interior Point
Methods for Quadratic
Optimization Problems
This chapter is devoted to implementation of interior point methods for solving
QO problems. Our implementation is based on and extends implementation of
the McIPM software package for linear optimization by X. Zhu [35, 36]. As QO
is a generalization of LO, many implementation features are very similar. The
crucial difference is in the system of Newton equations which is solved at each
IPM iteration. As we have seen in the previous chapter, it is more beneficial to
solve the augmented system instead of normal equations in QO for most cases.
The complication is that the augmented system is indefinite and, consequently,
we cannot use Cholesky factorization. Instead, we employ LDLT factorization
of the system using a sparse numerical linear algebra package. We describe the
complete sequence of steps for solving a QO problem and provide the algorithm
pointing out the differences of the QO case from the linear one.
We start Section 3.1 by describing the general interface of the McIPM pack-
age, which is now capable to solve both linear and quadratic problems. Next,
we describe the input formats for the solver. The standard input format for
33
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
quadratic problems is the so-called QPS or extended MPS format. We modified
the MPS file reader developed for linear problems by X. Huang [15] and added
to it the ability to read QPS files. We devote a section to basic preprocessing
and postprocessing techniques for QO and emphasize their differences from the
LO case. Section 3.2 describes the general structure of the McIPM package.
Section 3.3 is a brief description of sparse linear system solvers and contains
the information how the Newton system is solved by using those packages. Sec-
tion 3.4 describes the implemented algorithm including the predictor-corrector
strategy, self-regular search directions and stopping criteria. Finally, in Sec-
tion 3.5 we present numerical results and comparison of our quadratic solver
with the existing solvers such as BPMPD, MOSEK and CPLEX.
3.1 General Interface of the McIPM Package
The McIPM package, developed by X. Zhu [35], is a Matlab based software for
solving LO problems by using IPMs. We augmented it by adding the ability to
solve QO problems as well as extended its interface for user convenience. The
package is implemented in Matlab with C subroutines for reading input files and
solving the sparse linear systems. The input format for McIPM has the following
form:min cT x + 1
2xT Qx
s.t. Ax = b,l ≤ x ≤ u,
(3.1.1)
According to Matlab conventions a user can call the package from Matlab
by the following command
[x,y,z,objp,exitflag] = mcipm(A,b,c,Q,l,u,BIG,NAME,options)
where all input arguments except for A, b and c are optional. If matrix Q is
34
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
not provided, the problem is treated as linear. If no lower bounds l and/or
upper bounds u are provided, then the lower bound is set to zero for all variables
and the upper bound is set to +∞. Optional parameter BIG represents a large
number that is used as the upper bound for the variables without upper bounds,
the default value is BIG=1e32. NAME is optional problem name. The description
of all options is given in Appendix A. The output of the package is a primal
optimal solution vector x, a dual optimal solution (y,z), the optimal function
value objp and the indicator exitflag that gives us the solution status of the
problem (optimal, infeasible, etc., the reader may consult Appendix A for the
list of possible solution statuses).
Alternatively, the package may be called with only one input parameter
and options as follows
[x,y,z,objp,exitflag] = mcipm(pname, options)
Here it is assumed that the problem inputs are contained in the input file pname.
The McIPM package accepts input files in two formats:
• Matlab MAT file;
• MPS/QPS file.
McIPM automatically searches for the pname file in the matdir and mpsdir
directories inside the McIPM directory or in a user-specified directory.
3.1.1 Reading QO Data Files
The standard input format for QO problems is QPS. It stands for quadratic
programming system and is an extension of the MPS format of LO problems.
The MPS file reader is already implemented in C programming language by
X. Huang [15]. For the purpose of this thesis, we extend the MPS reader to
35
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
the QPS reader and add several useful features to it. We start by presenting an
example of the QPS file and briefly describing the MPS file format and continue
with pointing out the extension of the format to QPS.
The sample QPS file below represents the following QO problem:
min 4 + 1.5x1 + 2x2 + 12(8x2
1 + 2x1x2 + 2x2x1 + 10x22)
s.t. 2x1 + x2 ≥ 2−x1 + 2x2 ≤ 60 ≤ x1 ≤ 20, x2 ≥ 0.
(3.1.2)
A Sample QPS File
NAME QP example
ROWS
N obj
G r1
L r2
COLUMNS
c1 r1 2.0 r2 -1.0
c1 obj 1.5
c2 r1 1.0 r2 2.0
c2 obj 2.0
RHS
rhs1 obj -4.0
rhs1 r1 2.0 r2 6.0
BOUNDS
UP bnd1 c1 20.0
QUADOBJ
c1 c1 8.0
c1 c2 2.0
c2 c2 10.0
ENDATA
36
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
The MPS file is a text file with the following sections: NAME, ROWS,
COLUMNS, RHS, RANGES, BOUNDS, ENDATA. The input records inside
each section are divided into fields with fixed positions and have the following
format:
Field 1 Field 2 Field 3 Field 4 Field 5 Field 6Columns 2-3 5-12 15-22 25-36 40-47 50-61Contents Indicator Name Name Value Name Value
Indicator is the record type (e.g., indicator UB in the BOUNDS section means
”upper bound” on the variable). Name in fields 2, 3 and 5 represents row and
column names with the corresponding values in fields 4 and 6. The reader may
consult [15] and [23] for more comprehensive description of the records and fields.
The NAME section specifies the problem name and the ENDATA section
ends the file. Brief descriptions of the all other sections are provided below:
• NAME
• ROWS: the ROWS section lists the row names of matrix A, one per line, with
the name preceded by its type. Type L means ≤, E means =, a G means
≥, and an N means not constrained (e.g., the objective row);
• COLUMNS: the COLUMNS section lists nonzero elements of the matrix A,
preceded by the column name and row name in which it appears. Column
names correspond to variables x;
• RHS: the RHS section lists nonzero right-hand side elements of the matrix
A and the objective function preceded by the name of the row in which it
appears. One must also give a column name to the right-hand side vector;
• RANGES: an optional RANGES section allows one to supply the ranges of
rows – upper and lower limits on constraints. One must give a column
name for the range vector;
• BOUNDS: an optional BOUNDS section allows one to supply upper and lower
bounds on variables. One must give a column name for the bounds vector.
• ENDATA
37
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
In the QPS files, the new section QUADOBJ is added after the BOUNDS
section. The QUADOBJ section specifies the symmetric matrix Q, and, conse-
quently, only its lower triangular part is given. The elements of the matrix Q
appear in columnwise order, the same way as elements of the matrix A. The
0.5 multiplier of the quadratic part is implied and so the double of the diagonal
elements must be entered. The relative order of columns in QUADOBJ must be
the same as in the COLUMNS section.
Our QPS reader is able to read the QUADOBJ section of files. In addition,
it returns the constant in the objective function (equals to −4 in the example
above), which is given as an RHS value. Furthermore, the QPS reader converts
the problem to the standard form by introducing slack variables, splitting free
variables to two nonnegative variables and shifting the lower bounds to zero.
An alternative way to input the QO problem is to read it from a Matlab
MAT file. The MAT file should contain all the problem data and resembles
McIPM command line arguments: A,b,c,Q,l,u,BIG,NAME. Problem name NAME
and BIG constant are optional as well as Q, l and u. The constant in the objective
function CONSTOBJ can be optionally included. The problem in the MAT file as
well as in McIPM command line needs to be specified in the form (3.1.1). Data
from a QPS file is transformed to the form (3.1.1) by the MPS/QPS file reader.
3.1.2 Preprocessing and Postprocessing in Quadratic
Optimization
In this subsection we describe basic preprocessing and postprocessing techniques
for QO problems as well as their differences from the LO case (see [14] for com-
plete description of basic and advanced QO preprocessing techniques and [15]
for implementation of LO preprocessing). As LO preprocessing techniques deal
38
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
primarily with the problem constraints, most of them can be easily extended to
QO with the only difference that appropriate changes in the matrix Q should be
made. Postprocessing allows to get the solution vectors in terms of the original
problem. We proceed with the description of the basic preprocessing techniques
used in our implementation.
• Comparing the lower and the upper bounds of variables: If li > ui for some
variable xi, infeasibility is reported.
• Deleting fixed variables: Fixed variable li ≤ xi ≤ ui is a variable for which
li = ui. These variables are deleted from the problem formulation. Conse-
quently, vectors c and b are adjusted to
c = c + Q·jxj,
b = b − A·jxj,
where j is an index set of the fixed variables. Afterwards, columns j are
deleted from the matrix A, rows and columns j are deleted from Q, and
the j-th elements are deleted from the vectors c, b, l and u.
• Deleting zero rows: Rows of A containing exclusively zero entries are
deleted. When for such row the corresponding element of vector b is non-
zero, problem infeasibility is reported.
• Deleting zero columns: If in a LO problem a zero column appears in the i-
th position of the matrix A, the value of the corresponding variable xi is set
to li or ui depending on the sign of ci. This technique cannot be applied
for QO problems in general. It can be applied only if the i-th columns
(rows) of the matrix Q are zero rows or contain diagonal elements only. In
39
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
addition, if for some zero column i, ui = +∞ and ci < 0, the LO problem
is unbounded. Similarly, the QO problem is unbounded if in addition i-th
column (and row) of Q is zero.
• Solve singleton rows: Variables corresponding to singleton rows in A are
solved. Infeasibility can be detected if more than one singleton exist in
some column of A. Appropriate changes in A, Q, c, b, l and u are made to
accommodate for solved variables.
• Shift nonzero lower bounds: Non-zero lower bounds are shifted to zero by
transformation xi = xi − li. Vectors b, c and u are adjusted to b = b − Al,
c = c + Ql, and u = u − l.
In addition to preprocessing techniques described above, we scale the prob-
lem for better numerical properties. Geometric scaling [22] of the coefficients of
the constraint matrix A in order to reduce their spread improves the numeri-
cal properties of the problem to be solved in 95% of cases. Scaling is repeated
twice on the matrix A – two vectors handle the resulting row and column scaling
factors. The purpose of the scaling is to replace the original matrix A by the
matrix A = RAC, where R and C are row and column diagonal scaling matrices
respectively, in hope for better numerical properties of the matrix A.
Scaling is performed for badly scaled matrices only, which means that the
measure of the spread – ratio of the maximum to the minimum of the absolute
values of matrix elements – is large (greater than 104 in our implementation).
Scaling factors for the columns of A are the reciprocals of the square root of the
max norm (maximum of the absolute values) of each column. After the column
scaling for the constraint matrix has been applied, we apply row scaling similarly.
40
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
McIPM
LO QO SOCO
McSML
solvers/command/ solvers/ipm/
LDL
Ng & Peyton Fortran linear
solver
Sparse Linear Solvers
solvers/lib/
mpsdir/
matdir/
Input Data
InterfaceRead problem,
preprocess, postprocess
Qua
drat
ic op
t.(a
ugm
ente
d sy
stem
)
Line
ar o
pt.
(nor
mal
eq u
atio
ns)
IPM solver subroutines
MPS/QPS files
MAT files
mcipm arguments
Figure 3.1: General Structure of the McIPM Package
Postprocessing, just as in the case of LO, for QO consists of making back-
ward changes in order to recover the original vector x. The operations are per-
formed backwards from the ones described above. After that the original value
for the objective function is computed.
3.2 Structure of the McIPM Package
The McIPM package is a software package developed by a team of people at
Advanced Optimization Laboratory of McMaster University. Currently it allows
solving linear, quadratic and second-order conic optimization problems. Our
contribution to it is extending the code to QO problems.
The general structure of the McIPM package is depicted in Figure 3.1.
The general structure of the quadratic solver is shown in Figure 3.2.
41
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Figure 3.2: General Structure of the McIPM Quadratic Solver
42
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
3.3 Solving the Newton System
In this section, we describe how the linear systems given in (2.5.3) are solved.
Solving such systems is the most time consuming process at each iteration. To
solve a linear system efficiently we use sparse linear algebra packages. Princi-
pally, any sparse linear algebra package can be hooked into McIPM. The speed
and solution precision are crucial. It is essential that such sparse linear alge-
bra packages are implemented in an efficient way. That is why all sparse linear
algebra packages described below are implemented in C language.
The requirements to sparse linear algebra packages from the IPM point of
view are the following:
• efficient ordering for reducing fill-in during factorization. Ordering should
be performed once and with taking sparsity and structure of the augmented
system into account;
• symbolic factorization of a matrix, that is done once at the beginning;
• numeric matrix factorization is done once at each iteration and it uses
previously completed symbolic factorization and ordering;
• solving the system of equations with the given factorization (backsolve).
3.3.1 Sparse Linear Algebra Package McSML
The McSML (McMaster Sparse Matrix Library for IPMs) package (current ver-
sion is 1.2) is a sparse numerical linear algebra package developed by T. Seol.
McSML can solve normal equations systems (NE) and augmented systems (AS)
43
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
frequently arising in IPMs of the following general forms
(NE) AEAT v = rhs,
(AS)
(
−E AT
A 0
)
v = rhs.
Every McSML function is coded in C language and wrapped to a MEX-file for
interfacing Matlab. All the functions can be classified into two groups:
1. Normal equations system solver:
• sf mmd ne() – minimum degree ordering for normal equations system;
• sf scc ne() – symbolic factorization and building supernode struc-
tures;
• nf aat() – normal equations builder;
• nf scc ne() – numerical Cholesky factorization;
• nf sub ne() – solving equations for provided numerical factorization
and refinement;
• nf error ne() – checking solution error.
2. Augmented system solver:
• nf mmd as() – minimum degree ordering for augmented system re-
garding dense columns in A;
• nf scc as() – symbolic factorization and building supernode struc-
tures;
• nf aug() – augmented system builder;
• nf scc as() – numeric LDLT factorization with a dynamic regular-
ization;
• nf sub as() – backsolve and refinement;
• nf error as() – checking solution error.
44
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
The McSML package has a number of features specific to solving augmented
systems. It uses minimum (external) degree ordering with multiple elimination
in order to reduce fill-in. To make the (AS) matrix quasi-definite and apply Van-
derbei’s technique [29] the dynamic regularization is implemented. A symmetric
matrix is quasi-definite if it has the form:(
−E AT
A K
)
,
where E and K are positive definite matrices and A has full row rank. As the
matrix (2.5.4) is not quasi-definite, the following regularization is applied(
−Q − X−1Z − F T WS−1F AT
A 0
)
+
(
−RpI 00 RdI
)
, (3.3.1)
which makes both parts more positive definite. The regularization parameters
Rp and Rd are kept small (10−8 − 10−9) to get the reliable solution for non-
regularized matrix. Dynamic regularization applies only to very small diagonal
elements. In addition to transforming the matrix to quasi-definite form, regular-
ization improves stability of the linear algebra.
Iterative refinement of the solution is automatically performed by McSML
when the solution error is greater than 10−9. A preconditioned Conjugate Gra-
dient method is applied to refine the solution. McSML assumes that every input
matrix is sparse and every input vector is full.
Below we show the sequence of McSML function calls for solving an aug-
mented system.
info_out=1; % output level
AT=A’; % McSML requires matrix AT
(P,Pinv)=sf_mmd_as(A,E,info_out); line 3
(L,LNODE)=sf_scc_as(A,AT,E,P,info_out); line 4
D=nf_aug(A,AT,E,P,L,LNODE,0,0,info_out); line 5
45
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
R=nf_scc_as(L,LNODE,D,P,Rp,Rd,info_out); line 6
x=nf_sub_as(A,AT,E,L,LNODE,D,P,rhs,R,info_out); line 7
Ordering is performed in line 3, the permutation vector P and its inverse
P inv are returned. Symbolic factorization is shown in line 4. Ordering and
symbolic factorization are performed only once at the beginning of an IPM algo-
rithm. Numeric factorization, which is done at every IPM iteration, is performed
in lines 5-6, here Rp and Rd are the regularization parameters. Finally, line 7
shows how to get the solution x of the linear system with the right-hand side
vector rhs. This step is performed at least twice at each iteration.
3.3.2 Sparse Linear Algebra Package LDL
The LDL software package written by T. Davis [6] is a set of concise routines
for performing factorization of symmetric matrices. It is applicable to positive
definite and indefinite sparse matrices. The package is written in C and has
Matlab interface which has been modified for our needs.
In contrast to the McSML package, which was designed to be used in IPMs,
LDL does not provide all subroutines used for solving linear systems. That is why
we use the external approximate minimum degree ordering package (AMD) to
reduce the fill-in while solving a linear system. The sequence of steps for solving
augmented systems, i.e., ordering, symbolic factorization, numeric factorization
and backsolve is the same as for the McSML package. The input matrix should
be made quasi-definite prior to factorization by adding the regularization terms
as in (3.3.1).
The sequence of commands for solving the system with LDL is:
Permut = amd([-E A’; A speye(m,m)]); line 1
[Lp,Lnz,Parent,Pinv] = ldlsymb([-E A’; A speye(m,m)], Permut); line 2
[L,D] = ldlnum([-E-regeps*speye(m) A’; A regeps*speye(m))],...
Permut, Lp, Parent, Lnz, Pinv); line 3
46
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
x = ldlsolve(L, D, Permut, rhs); line 4
3.4 Computational Algorithm for QO
3.4.1 Predictor-Corrector Strategy
Mehrotra’s predictor-corrector approach is known to be the most efficient way to
implement IPMs as it allows to reduce the total number of iterations by perform-
ing a second-order correction to the pure Newton direction. This actually means
that a number of steps are made within each iteration. Making several steps
within one iteration is much cheaper as we are solving the system of equations
with the same left-hand side and just changing the right-hand side vector. So,
the same factorization is used and a number of backsolves is performed.
The first step of the approach is to make a predictor or so-called affine-
scaling step. The predictor direction is found by solving the Newton system
(2.4.7) for µ = 0. As a result, the direction (∆xa, ∆ya, ∆za, ∆wa, ∆sa, ∆τa, ∆κa,
∆νa) is produced. The complementarity gap is reduced by the factor (α − 1) at
the predictor step, where α is the maximum acceptable step length from (2.4.17).
The second step is computing the corrector direction for the chosen cen-
tering parameter µ suggested by Mehrotra:
µ =
(
ga
g
)2ga
n + mf + 1,
where g is the actual complementarity gap and ga is the predicted complemen-
tarity gap
ga = (x + αa∆xa)T (z + αa∆za) + (s + αa∆sa)
T (w + αa∆wa)
+(τ + αa∆τa)T (κ + αa∆κa) = (1 − αa)(x
T z + sT w + τκ).
In the corrector step instead of neglecting the second-order terms ∆x∆z, ∆w∆s
and ∆τ∆κ in the Newton system, they are replaced by their approximate values
∆xa∆za, ∆wa∆sa and ∆τa∆κa computed at the predictor step. Consequently,
47
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
the left-hand side of the Newton system (2.4.7) solved at the corrector step is
unchanged and the coefficients on the right-hand side become
rsw = µe − Sw − ∆Sa∆wa,rxz = µe − Xz − ∆Xa∆za,rτκ = µe − τκ − ∆τa∆κa.
As a result, we solve two linear systems instead of one at each IPM iteration.
However, the matrix is factored only once that leads to much lower computational
costs.
3.4.2 Self-Regular Functions and Search Directions
Significant efficiency and complexity improvement in our implementation of IPM
comes from using self-regular proximity functions and search directions. We have
already seen a proximity function Ψ(xz, µ) in the prototype IPM algorithm in
Chapter 2. To make a quick tour into a new method for finding search direc-
tions in IPM based on self-regular proximity functions introduced in [25], let
us consider a function ψ(t) ∈ C2 : (0,∞) → IR+. Observing that the last
set of equations defining the central path (2.2.2) can be written equivalently as
ψ(xz) = ψ(µe) or ψ(xzµ
) = ψ(e) and applying Newton’s method to such trans-
formed central path equations, we get
A△x = 0,AT△y + △z − Q△x = 0,
xµψ′(xz
µ)△z + z
µψ′(xz
µ)△x = ψ(e) − ψ(xz
µ).
(3.4.1)
Introducing a new variable v =√
xzµ
and the notations dx = v△x
xand dz = v△z
z
we get
µv(dx + ds) = x△z + z△x
and the linear system (3.4.1) can be written as
Adx = 0,AT△y + dz − Qdx = 0,
dx + dz = dv,(3.4.2)
48
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
where A = A diag(xv), Q = Q diag(x
v) and
dv =ψ(e) − ψ(v2)
vψ′(v2).
If ψ(t) = t, we have dv = v−1 − v that corresponds to the standard primal-
dual IPM. We can observe that Newton steps increase the components vi < 1 as
dvi≥ 0 and decrease the components vi ≥ 1 as dvi
≤ 0 with iterations.
The idea behind the new search direction is that we can increase more the
small components of v and decrease more the large components of v by making
dv = v−q − vp instead of using the standard direction dv = v−1 − v. The class
of self-regular functions ψ(t), that are strongly convex functions, have global
minimum zero at the point t = 1 (i.e., ψ(1) = ψ′(1) = 0) and satisfy some other
convexity conditions is described in [25]. The following subclass of self-regular
functions gives the value dv = v−q − vp:
Γp,q(t) =tp+1 − 1
p + 1+
t1−q − 1
q − 1, p ≥ 1, q > 1. (3.4.3)
Note that for ψ(t) = Γp,q(t) the derivative is
ψ′(t) = tp − t−q.
In IRn we define a self-regular proximity function as
Ψ(v) =n
∑
i=1
ψ(vi).
Taking the gradient of Ψ(v) we get
∇Ψ(v) = ∇(
n∑
i=1
ψ(vi)
)
= ψ′(v) = (ψ′(v1), ψ′(v2), . . . , ψ
′(vn))T .
For the self-regular function Ψ(v) =∑n
i=1 Γp,q(vi), we also have
∇Ψ(v) = ∇(
n∑
i=1
Γp,q(vi)
)
= −(v−q − vp).
49
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Consequently, we can rewrite system (3.4.1) as
Adx = 0,AT△y + dz − Qdx = 0,
dx + dz = −∇Ψ(v),(3.4.4)
and the corresponding Newton system becomes
A△x = 0,AT△y + △z − Q△x = 0,
x△z + z△x = −µv∇Ψ(v).(3.4.5)
The use of such self-regular search direction allows improving the iteration com-
plexity of large-update IPMs to O(√
n log n log nǫ) [25].
Putting self-regular search directions into the framework of the homoge-
neous embedding model, we can compute the corrector step by solving the New-
ton system (2.4.7) with the following coefficients at the right-hand side
rsw = −µvsw∇Ψ(vsw) − ∆Sa∆wa,rxz = −µvxz∇Ψ(vxz) − ∆Xa∆za,rτκ = −µvτκ∇Ψ(vτκ) − ∆τa∆κa,
where
vsw =
√
sw
µ, vxz =
√
xz
µ, vτκ =
√
τκ
µ.
In the implementation we use the self-regular proximity function Γp,q(v)
with p = 1 and initial value of q = 1. The value of q is increased dynamically to
3 if at some iteration the iterate gets too close to the boundary of the feasible
region. The reader may consult [35, 36] or [25] for more details about self-regular
proximity functions.
3.4.3 Stopping Criteria
In this section we describe the stopping criteria of the algorithm as well as crite-
ria for reporting the solution status (optimal, infeasible, numerical difficulties).
In the description below we use the modifications of the criteria developed by
X. Zhu [35].
50
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
The following measures are defined as the parameters of McIPM:
tol emb – duality gap tolerance for the embedding problem;
tol ori – duality gap tolerance for the original problem;
tol kt – accuracy parameter for τ/κ; in the solution, one of the values, either τ
or κ should be zero and the other one should be positive.
tol end – the tolerance for the parameter µ.
Define the relative errors for the k-th iteration of the algorithm
fea p = ||Axk−τkb||+||Fxk+sk−uτk||τk+||xk||
,
fea d = ||AT y+zk+Qxk−cτk−F T wk||τk+||zk||
,
gap ori =||cT xk−bT yk+uT wk+xkT
Qxk
τk ||
τk+||bT yk−uT wk− 1
2
xkTQxk
τk ||,
and
error = max{fea p, fea p, gap ori}. (3.4.6)
The algorithm terminates if:
• errork < tol ori and κk < τ k tol kt. The solution
(x∗, z∗, y∗, w∗, s∗) =
(
xk
τ k,zk
τ k,yk
τ k,wk
τ k,sk
τ k
)
is feasible and firm optimal in this case;
• µk < tol end. The solution
(x∗, z∗, y∗, w∗, s∗) =
(
xk
τ k,zk
τ k,yk
τ k,wk
τ k,sk
τ k
)
is feasible and is reported to be optimal up to a certain precision;
• µk < tol emb and τ k < κk tol kt the problem is infeasible (see Sec-
tion 2.3.2 for the description of infeasibility detection);
• µk < tol emb, τ k > κk, but τ k > κk tol kt the problem is non-firm
infeasible and we move to the next iteration k + 1. If the solution is still
51
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
non-firm infeasible after a couple of iterations as for numerical reasons no
more precise solution can be reached, infeasibility is reported up to a certain
precision.
3.5 Numerical Results
In this section our computational results for the McIPM-QO solver are presented.
All computations are performed on an IBM RS/6000 workstation with four pro-
cessors and 8Gb RAM. The operating system is IBM AIX 4.3.
The test problems used for our numerical testing are coming from the Maros
and Meszaros QP test set [18] which consist of 138 problems. The sizes of
problems vary from a couple of variables to around 300, 000 variables. You can
find the description of the problems in the testset, including original problem
dimensions and problem dimensions after McIPM preprocessing, in Appendix B.
The McIPM parameter settings used for testing are the following:
% PARAMETERS - Algorithmic parameters for MCIPM
p = 1;
q = 1;
regeps = 1.e-9; % regularization
MAXITER = 150; % Maximum number of iteration
tol emb = 5.0e-17; % an accuracy parameter for mu
tol ori = 5.0e-09; % an accuracy parameter for gap ori
tol end = 1.e-15; % an accuracy parameter for nu
tol bad = 1.e-15;
tol kt = 1.e-3; % an accuracy parameter for tau/ka
damp = 0.995; % a variable damping parameter
bad step = 0.4;
worst step= 0.1;
MAXQ = 3;
The testing results are shown in Tables 3.1, 3.2 and 3.3. As there are no
52
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
standard confirmed solutions for the test set, we have solved all the problems with
CPLEX and MOSEK and also used the BPMPD solution reported in [22] for
comparison. In columns ’Diff.’ of Tables 3.1–3.3 we report the difference of the
optimal solutions found by McIPM with the solutions found by CPLEX, MOSEK
and BPMPD (ObjCPLEX − ObjMcIPM , ObjMOSEK − ObjMcIPM and
ObjBPMPD − ObjMcIPM). Furthermore, column ’Sign. digits’ shows
the number of significant digits in the McIPM solutions as compared with the
solutions found by CPLEX, MOSEK and BPMPD. We also report the number of
iterations ’Iter’, optimal function value ’Objective’ and CPU time. Column
’Resid.’ is McIPM error computed according to (3.4.6).
All the test were done using the LDL package for solving the augmented
system. As we see from the results, McIPM produces reliable solutions with
six or more significant digits for the majority of the problems. The problematic
problems are depicted in bold font. The imprecise solution on this set of problems
is due to precision of solving the Newton system with the default regularization
parameter 10−9. The two sparse linear packages – LDL and McSML – are not
always able to produce precise solutions due to numerical difficulties, that result
in less precise solutions produced by McIPM. We plan to use iterative refinement
of solution of the liner system produced by LDL to improve the current results.
It is worth mentioning that using different regularization, solving the linear
system without ordering and regularization with the LDL package (it works in
most cases as zero elements does not appear in the diagonal in the upper left
block E of the matrix) or employing McSML as sparse linear system solver
allow to get the solution precision of seven or more significant digits for all
the problems in the test set except for the largest problem boyd2 which has
four significant digits in the solution. Table 3.4 shows the solutions to bold-
font problematic problems in Tables 3.1–3.3 computed using McSML or LDL
without ordering and regularization of the augmented system. In general, almost
all packages experience troubles on the number of problems shown in Table 3.4.
One can notice that McIPM has lower number of iterations than CPLEX or
53
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
MOSEK on the majority of difficult problems. Solution precision on this set of
difficult problems is not very good especially for commercial solvers, furthermore,
MOSEK and BPMPD produce incorrect solutions for one problem each.
Comparison of the number of iterations used by CPLEX, MOSEK and
McIPM does not show a clear trend, but in the majority of the cases McIPM
beats the commercial packages confirming that the usage of self-regular proximity
functions and search directions can reduce the number of IPM iterations. In
the column ’#iter q>1’ we show the number of iterations when self-regular
parameter q was dynamically increased from 1 to 3. The solution time of McIPM
in most cases is more than the one of CPLEX and MOSEK. This is partially
explained by the fact that CPLEX and MOSEK are pure C packages while
McIPM is partially coded in Matlab. The second reason for the difference in
solution time is that all packages – CPLEX, MOSEK and BPMPD are utilizing
more advanced preprocessing techniques than McIPM. For example, BPMPD
reduces problem size for the problem qcapri from 271 × 373 to 148 × 214 while
McIPM only to 267 × 353. Similarly, MOSEK reduces the size of the largest
problem in the test set boyd2 from 186531 × 279785 to 119721 × 150580 while
McIPM leaves it unchanged.
We do not show all the results obtained when the McSML package is em-
ployed for solving the augmented systems as it generally exhibit slightly poorer
performance than LDL. Still, McIPM with the McSML sparse linear solver is
able to solve more than 90% of the test set problems.
In general, we may conclude that McIPM is competitive even with the
commercial packages. One of the sources for improving competitiveness is the
possibility of using more advanced sparse linear algebra packages, e.g., TAUCS,
which does not have separate symbolic and numeric LDLT factorization at the
moment but plans to have it in the future. Another possibility is to get Mc-
SML to the state of the art by implementing Bunch-Parlett or Bunch-Kaufman
techniques. The other option is porting McIPM to C language for improving its
speed.
54
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Table 3.1: McIPM Performance on QO Test Set (I).
bor
PD
PM
PB
evit
cej
bO
retI
em
aN
.c
es
UP
Ce
vitc
ejb
Or
etI.
ce
s U
PC
evit
cej
bO
.di
se
Rr
etIe
vitc
ejb
OU
PC
.c
es
reti
#
1>
q.r
effiD
.n
giS
stigi
d.r
effiD
.n
giS
stigi
d.r
effiD
.n
giS
stigi
d
820
-E2
8.4
01
40-
E00
.5
01
30-
E0
1.
10
03
.5
16
0+E
815
711
478
6.
10
1-
E54
.6
860+
E8
114
786
.1
01
.4
60
+E3
257
114
786
.1
51
5.0
60
+E9
257
114
786
.1
1d
2gu
a
82
0-E
94.
311
00
+E0
0.0
01
40
-E
00
.5
045
.41
60+
E15
60
86
381
8.1
01-
E4
2.7
86
0+E
18
63
818
.1
09
.3
60
+E
156
086
381
8.1
59
4.
06
0+E
656
0863
81
8.
11
cd
2g
ua
92
0-
E79.
511
40
-E
00.3
01
40-E
00.
8-
36
8.4
160
+E3
047
431
89
4.
60
1-E
24.
761
60
+E8
431
89
4.
600
.5
60
+E6
047
431
894
.6
31
72
.3
60
+E5
937
431
894
.6
71
pqc
d2g
ua
820-
E19
.6
01
30-
E03
.1-
01
30
-E
03
.5-
48
7.4
160
+E
90
302
107
32.
69
0-E
08.
17
16
0+
E12
107
32.
604
.5
60
+E6
920
210
732
.6
51
38
.3
60
+E6
52
02
107
32
.6
02
pq
d2
gua
96
0-
E03.
401
70
-E
03
.3
-0
180-
E00
.9
06
6.1
20+
E07
527
760
45.
50
1-E
82
.4
62
0+E
377
604
5.5
99
.0
20+
E73
52
77
604
5.5
430
.0
20
+E9
752
776
045
.5
1d
3gu
a
960-
E63
.1
01
70
-E08
.3-
11
80
-E
00
.6
01
7.
120
+E4
683
426
21
7.
70
1-E
76.
36
20+
E44
262
17.
70
0.
12
0+
E62
834
262
17.
74
70
.0
20
+E
078
342
621
7.7
1cd
3gu
a
96
0-
E80
.4
01
70-E
01.
90
17
0-E0
1.6
004
.1
20
+E
295
412
633
9.9
90
-E
52
.4
01
20
+E
512
633
9.9
59
.0
20+
E3
86
412
6339
.9
11
92
.0
20
+E3
564
126
339
.9
31
pqc
d3g
ua
96
0-E
65.
1-
970
-E0
7.7
01
70
-E0
8.2
-0
95
.1
20
+E
65
17
673
257
.6
90-
E95
.1
21
20
+E7
673
257
.6
79
.0
20
+E
332
767
325
7.
62
113
.0
20
+E8
217
673
257
.6
61
pq
d3g
ua
E9
5.
3-
01
20-
E0
4.
21
100
+E0
0.0
22
1.
701
70+
E14
691
25
37
1.6
-90
-E1
6.2
42
70+
E0
22
537
1.6
-24
.31
70
+E
716
912
537
1.6
-7
21
9.8
270
+E1
469
125
371
.6-
04
1d
yob
91
0-
E6
9.6-
10
0+E
69.
6-
10
0+E
69.
6-
43
17
1.9
522
10
+E
155
29
02
128
.2
50-
E69
.8
05
11
0+
E7
676
521.
23
9.
56
10+
E15
966
765
21.
204
76.
98
10+
E76
276
765
21.
23
52
dy
ob
100
+
E09
.3
11
01
-E0
0.4
-1
10
1-E
00.4
-1
94
.3
00+
E93
090
583
65.
4-
90
-E
05.
11
10
0+
E90
583
65.
4-
07
.1
00
+E
340
905
836
5.
4-
11
26
.0
00
+E3
409
058
365
.4-
21
050
-tn
oc
01
90-
.3
-11
01-
E00
.1-
11
01
-E0
0.1
-0
82
.91
00
+E7
868
793
446
.4-
01-
E74
.5
21
00
+E9
793
446
.4-
00
.1
100
+E8
868
793
446
.4-
21
66
.6
00
+E8
868
793
446
.4-
41
001
-tn
oc
980-
E31
90-E
23.
511
11-E
00.
2-
11
11-
E00
.2-
258
.02
10-
E86
42
37
255
9.1
90-
E6
8.1
31
10-
E3
37
255
9.1
01.
910
-E6
642
372
559
.1
21
45
.5
10
-E
664
237
255
9.1
21
10
1-t
noc
9
21
101
-E0
0.1-
11
01-
E00
.1
-0
48.
831
00+
E30
295
784
86.
4-
90
-E8
7.2
21
00+
E95
784
86
.4
-00.
08
00+
E40
295
784
86.
4-
21
81.
74
00
+E
40
29
578
486
.4
-3
10
02
-t
no
c9
80-
E30
.
-E5
0.
3-
11
01
-E
08.
1-
990
-E9
3.4
15
2.4
41
10
-E5
037
338
429
.1
90-
E19
.4
21
10
-E7
338
429.
10
0.
47
10-
E78
273
384
29.
121
70
.6
41
0-
E44
773
384
29.
12
11
02-
tno
c9
90
E9
3.3
11
01
-E
06.
1-
880
-E4
9.5
30
2.0
37
10
-E7
068
221
519
.1
01-
E80
.4
61
10
-E2
321
519
.1
00.
072
10
-E1
958
221
519
.1
31
86.
751
10
-E1
554
321
519
.1
61
003
-tn
oc
880-
2.
26
20
+E5
2.2
620
+E4
2.2
21
18.
894
28
0+E
545
754
078
0.1
50
-E2
3.3
95
80
+E
084
078
0.1
00
.00
61
80+E
110
084
078
0.1
11
70
.1
192
80
+E0
799
740
78
0.
191
l_1
pqx
vc
620
+E5
820-
E81
.3
93
0-
E04
.2
01
40
-E0
0.
10
60
.2
60
+E
286
511
578
0.1
90-
E1
5.1
11
60
+E
611
578
0.1
09.
260
+E6
075
115
78
0.
10
10
7.8
60
+E
386
511
578
0.1
11
m_
1pq
xvc
840
-E0
2.1
-11
60
-E
00
.1
11
60
-E
00.
1-
062
.0
40
+E
021
817
095
1.1
11
-E8
8.5
940
+E8
170
95
1.1
26
.0
40
+E
121
817
0951
.1
84
0.
040
+E
91
181
709
51.
11
1s
_1p
qxv
c
90
.3
-9
20
-E
09.
801
20-E
00.
3-
02
9.
644
70
+E
903
854
248
1.8
90
-E1
9.1
21
70
+E
854
248
1.8
00
.04
27
0+E
8938
54
24
81
.8
11
87
.80
67
0+
E97
285
424
81.
82
1l
_2p
qxv
c9
10
-E
93
0-E
60.
1-
01
40
-E
06.
311
50-
E00
.4-
07
5.1
50+
E60
134
55
10
2.8
01
-E4
2.2
11
50+
E34
551
02.
803
.1
50+
E24
134
551
02.
801
44
.3
50+
E20
134
551
02
.8
9m
_2
pqx
vc
95
0-
E1
2.2
950
-E3
1.2
11
70
-E0
0.6
-0
62
.0
30
+E9
774
049
021
.8
90-
E77
.1
930
+E5
049
021
.8
26
.0
30
+E2
994
049
021
.8
92
0.0
30
+E3
774
04
90
21.
811
s_2
pq
xv
c
74.
4-
710
+E0
0.2
910
-E0
0.4
031
.93
780
+E7
440
111
75
1.
16
0-E
72.
161
80
+E0
111
751
.1
00
.00
84
80
+E
054
211
175
1.1
01
74.
899
28
0+E
784
011
175
1.1
31
l_
3pq
xvc
800
+E
82
0-
E61
.4-
01
40
-E0
0.4
82
0-E
45
.5-
05
7.2
60+
E61
478
282
63.
10
1-E
71.
341
60+
E78
282
63.
106
.7
60
+E
02
47
828
26
3.
111
15.
51
60+
E26
868
282
63.
10
1m
_3p
qxv
c
84
0-E
22.
2-
950
-E0
9.1
-9
50
-E0
0.2
-0
52.
040
+E2
222
343
491
.1
90-E
43.
19
40+
E23
434
91.
146
.0
40
+E
30
223
434
91.
101
50
.0
40+E
20
223
434
91.
12
1s
_3p
qxv
c
01
90
-E9
8.2
11
01
-E
00.3
-21
00
+E
00.
00
81
.0
10
-E
117
126
900
7.3
11-
E9
6.
36
10
-E
226
900
7.3
36
.0
10
-E
186
126
900
7.3
41
0.
01
0-E
11
71
269
007
.3
51
olk
pd
960
-E5
0.1
-11
80-
E00
.2
-9
70
-E0
2.5
-0
54
.6
20
+E
501
842
625
3.2
11-
E8
4.4
72
0+E
842
625
3.2
01
.3
20
+E
301
842
625
3.2
50
3.
020
+E
35
084
262
53.
21
3c
ot
d
11
01
-E5
2.1
990
-E
59
.3
11
01
-E2
4.1
-0
94.
020
-E5
785
692
105
.3
01-
E99
.4
41
20
-E6
692
105
.3
36
.0
20
-E
428
96
92
105
.3
31
60
.0
20
-E3
375
692
105
.3
61
1l
aud
11
01-
E87
.1-
11
11
-E0
4.5
-11
11-
E05
.5-
074
.0
20-
E87
167
633
73.
39
0-E
24.
22
120
-E
67
633
73
.3
56.
020
-E4
216
763
373
.3
21
60
.0
20
-E
321
676
337
3.3
41
2l
au
d
99
0-E
48.
20
101
-E0
6.2
01
01
-E
09
.2
-0
16
.0
10
-E
617
38
55
753
.1
01-
E11
.8
31
10
-E
485
575
3.1
86.
01
0-E
247
385
575
3.1
31
11.
01
0-E
786
38
55
753
.1
71
3l
au
d
01
90-
E01
.2-
11
01-
E04
.5
11
01-
E00
.3-
09
3.
010
-E
01
248
090
64
.7
90-
E43
.3
21
10-
E48
090
64.
766
.0
10
-E
462
480
90
64
.7
21
40.
01
0-E
08
148
090
64.
74
14
la
ud
95
0-
E40
.3-
01
70
-E0
0.9
-0
17
0-E
00.9
-5
36
.0
30
+E
403
805
255
1.6
90-
E9
7.4
42
30+
E80
525
51.
64
6.0
30+
E59
280
525
51.
681
00
.0
30
+E
59
28
052
551
.6
61
1cla
ud
96
0-E
03.
711
70
-E0
0.1
11
00
+E0
0.0
40
5.
030
+E7
296
70
315
5.
30
1-
E09
.5
91
30
+E7
703
155
.3
26
.0
30
+E
829
67
03
155
.3
51
00
.0
30
+E7
296
703
155
.3
51
2cl
aud
960
-E0
2.3
01
80-
E00
.4
11
80
-E0
0.2-
062.
020
+E0
862
32
32
72.
49
0-E
39.
39
20+
E33
23
27
2.4
26.
020
+E4
862
323
272
.4
90
0.0
20
+E
876
232
327
2.4
21
5c
lau
d
94
0-
E76
.1
11
00
+E
00.
011
00
+E0
0.0
237
.0
40
+E
338
853
90
38
.1
90-
E15
.6
02
40
+E
953
903
8.1
66.
04
0+E
338
853
903
8.1
51
00.
04
0+E
338
85
39
038
.1
31
8c
lau
d
64
0-
E08
.4-
64
0-E
28.
4-
07
9.
102
20
+E
640
592
481
4.1
-90-
E9
9.1
31
20+
E34
348
14.
1-
00
.26
20
+E
312
343
481
4.1
-5
1r
orr
e d
aer
SP
Ma
tad
xe
01
90-
E96
.3-
21
11-
E00
.6
21
11-
E00
.8
060
.0
10
-E9
639
637
17
2.
91
1-E
40.
45
10-
E96
371
72
.9
55
.0
10-
E57
396
371
72.
94
00
.0
10
-E7
739
637
172
.9
482
shn
eg
01
01
-E
20.
411
11
-E
55
.2
01
01
-E5
3.4
-0
08
.0
40-
E70
839
472
48.
19
0-
E1
7.4
21
40
-E
435
724
8.1
05.
14
0-
E06
36
94
724
8.1
51
80.
04
0-E
633
054
724
8.1
02
2p
qdl
uog
88
0-E
97.
26
50
-E5
9.1
880
-E9
7.
20
96
.0
00
+E1
279
387
260
.2
90-
E24
.3
01
00
+E0
487
26
0.
201
.1
00
+E
185
430
826
0.2
87
0.
00
0+
E00
004
872
60.
21
13
pqd
luo
g
01
70
-E
01.
401
70
-E
00.
40
170
-E0
1.4
061
.0
20
+E9
594
402
846
.6
90-
E27
.4
01
20
+E
540
284
6.6
67
.0
20
+E
999
440
284
6.6
01
00.
02
0+E0
00
540
284
6.
64
18
11s
h
86
0-
E22
.1
-8
60
-E
22.1
-8
60
-E2
2.1
-0
61
.0
10
+E
187
899
959
9.9
-90-
E6
4.2
21
10+
E00
006
99.
9-
75
.0
10
+E
000
000
06
99
.9-
01
10.
010
+E
000
000
06
99.
9-
22
12
sh
01
70
-E
67.
501
50
-E6
8.1
990
-E7
6.2
07
1.
09
0-E
232
441
937
6.2
-80-
E9
8.2
81
70-
E50
70
13
7.5
95
.0
50-
E13
305
511
68.
141
00
.0
00
+E0
000
000
000
.0
81
86
2sh
880
-E2
8.1
-8
80-
E66
.1-
88
0-
E1
7.1
-0
60
.0
10-
E91
82
11
111
1.1
90-
E7
4.2
51
0-E
11
11
111
.1
85
.0
10-
E95
111
111
11.
15
00
.0
10
-E0
111
111
111
.1
22
53
sh
99
0-
E55
.2-
99
0-E
73.5
99
0-E5
5.2
-0
01
.0
10
-E
552
00
00
005
.2
90-
E06
.3
01
10
-E
000
000
5.2
95.
01
0-E
297
000
000
5.2
01
00.
01
0-E
000
000
000
5.2
62
dom
53s
h
61
41-
E96
.1
51
41-
E59
.1
41
41-
E06
.1
06
0.
04
1-
E55
511
278
95.
1-
01-E
06.
55
61
-E
248
718
8.8
75.
051
-E88
763
172
55.
34
00.0
00
+E
000
0000
00
0.0
01
15s
h
980
-E5
5.3
11
00+
E00
.0
11
11
-E0
0.3
-0
50.
00
0+E
546
574
66
23
.5
01-
E4
0.
25
00
+E6
74
66
23.
57
5.
000
+E5
465
746
623
.5
30
0.0
00
+E
546
574
662
3.5
325
sh
DP
MP
B .p
mo
CK
ES
OM
XE
LP
CK
ES
OM .
pm
oC
XE
LP
C .p
mo
CM
PIc
M
55
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Table 3.2: McIPM Performance on QO Test Set (II).
bor
PD
PM
PB
evit
cej
bO
retI
em
aN
.c
es
UP
Ce
vitc
ejb
Or
etI.
ce
s U
PC
evit
cej
bO
.di
se
Rr
etIe
vitc
ejb
OU
PC
.c
es
reti
#
1>
q.r
effiD
.n
giS
stigi
d.r
effiD
.n
giS
stigi
d.r
effiD
.n
giS
stigi
d
88
0-
E2
4.
41
10
0+E0
0.0
11
00
+E
00
.0
011
.0
00
+E8
552
320
390
.4
90
-E
67.
18
00+
E33
203
90.
49
5.
000
+E
85
523
203
90.
46
00
.0
00
+E
855
232
039
0.4
02
35s
h
980
-E
48.1
-0
19
0-E0
1.4
11
01-E
00.
2-
070
.0
00+
E61
818
181
86.
4-
01-
E7
0.1
60
0+
E28
181
86.
4-
75
.0
00
+E5
771
818
186
.4-
51
0.
00
0+E
818
181
818
6.4
-7
167
sh
62
0+E
62.
201
20
-E0
8.1
11
00+
E0
0.
00
98.
370
+E3
783
644
28
4.
39
0-E
98.
251
70
+E0
964
28
4.
369
.0
70
+E1
983
644
284
.3
21
81
.0
70
+E3
783
644
284
.3
21
dom
-se
uh
66
0+E
62.
20
12
0+E
03.
1-
01
20+
E05
.2-
05
4.
311
+E
89
836
442
84.
37
0-E
62.
13
11
1+
E09
642
84.
389
.0
11
+E5
883
644
284
.3
21
52
.0
11
+E3
78
36
442
84
.3
81
si
ts
euh
01
90-
E70
.1-
21
11
-E00
.4
11
01-
E07
.1
45
5.2
10-
E70
149
797
57.
59
0-E
39.
27
11
0-
E49
797
57.
50
4.
110
-E1
114
979
757.
591
53.
71
10
-E4
214
979
757
.5
61
pis
k
82
0-
E01
.5
81
0-E
12.
19
30
-E
04.
3-
010
.2
60
+E
094
310
690
4.2
90
-E6
1.
21
16
0+E
410
690
4.2
01
.1
60+
E2
07
410
690
4.2
21
81.
06
0+E
654
310
690
4.2
31
re
sa
l
11
0+E
10.
11
10
+E
10.
11
10
+E1
0.1
00
0.5
11
0+
E43
770
820
06.
26
0-E
08.
25
11
0+
E20
422
16.
30
9.
610
+E3
999
142
216
.3
22
19
.1
10+E
00
12
04
221
6.3
32
1t
ew
sil
110+
E93
.2
11
0+E
93.
21
10
+E9
3.
20
42
.5
110
+E5
348
199
755
.2
60-
E59
.3
51
10
+E5
875
849
.4
01
.8
10
+E
360
108
584
9.4
72
43.
31
0+
E00
748
758
49.
41
40
1te
wsi
l
11
0+E
86.
11
10
+E8
6.1
110
+E8
6.1
090
.51
10+
E14
761
250
72.
35
0-E
61
.3
51
10+
E75
932
59.
40
9.
910
+E
69
508
932
59.
463
83
.2
10+
E00
175
932
59.
49
21
1te
wsi
l
130
+E3
5.1
130+
E35
.1
13
0+E
35.
10
60.
51
20
+E
593
69
05
960
.2
40-
E61
.8
51
30
+E
47
29
637.
10
0.8
30
+E
708
58
296
37.
17
201
.5
30
+E1
624
729
637
.1
46
21t
ews
il
20-
E09
.1-
42
0-E
09.
1-
42
0-E0
9.1
-0
25
.6
31
10
+E3
476
407
105
.2
50-
E91
.2
05
110
+E6
708
994
.2
09
.6
10
+E
35
497
08
99
4.2
22
51
.1
10
+E0
016
708
994
.2
31
2te
wsi
l4
10
+E3
7.7
-1
10
+E3
7.7
-1
10+E
37
.7-
663
.64
12
0+
E98
105
303
20.
11
0-E
05.
10
51
10+
E02
210
05.
20
8.
710
+E9
932
221
005
.2
32
58
.1
10
+E
000
022
100
5.2
22
3t
ew
sil
1
50+
E99
.1-
45
0+
E9
9.
1-
45
0+E
99.
1-
31
1.
15
15
0+E
319
999
878
9.1
00+
E9
9.1
05
11
0+E
211
000
5.2
00
.7
10+
E8
69
611
000
5.2
32
67.
11
0+E
090
211
000
5.2
12
4te
wsi
l4
80+E
02.
2-
780
+E0
2.2
-7
80
+E02
.2-
109
.45
180
+E12
71
738
89
1.
20
0+E
00.
205
110
+E3
52
43
05.
203
.7
10
+E
748
092
430
5.2
12
87.
11
0+
E00
035
243
05.
21
25
te
wsi
l7
10
+E3
5.3
-1
10+
E35
.3-
11
0+E
35.
3-
231.
831
10+
E32
17
14
262
0.6
20
-E9
2.3
051
10
+E
847
599
4.2
06.
61
0+
E72
365
759
94.
212
29
.1
10+
E0
06
747
599
4.
23
26t
ews
il
1 12
0+
E95.
41
20
+E
95.
41
20
+E9
5.4
05
0.5
11
0+
E06
239
108
00.
43
0-E
80.
15
12
0+
E98
048
89.
40
0.1
12
0+
E40
64
17
488
9.4
83
85.
120
+E0
809
804
889
.4
81
7te
ws
il
13
0+E
80.
71
20
+E3
5.6
120
+E3
5.6
084
.51
10+
E93
327
242
71.
63
0-E
79.
15
13
0+
E60
074
41.
70
5.
820
+E3
237
207
441
.7
92
65
.4
20
+E
009
500
744
1.7
85
8t
ew
sil
130
+E3
4.1
13
0+
E34
.1
13
0+
E34
.1
04
1.
51
20
+E
225
418
799
2.5
30-
E41
.3
51
30
+E
315
236
9.1
06.
93
0+E
649
415
236
9.1
23
17.
43
0+E
906
215
236
9.1
06
9te
wsi
l
960-
E04
.4
96
0-
E0
6.
3-
96
0-E
02.
4-
011
.0
30
+E
659
851
489
3.2
90
-E3
8.1
11
30
+E
951
489
3.2
16.
030
+E
02
985
14
893
.2
900
.0
30
+E
419
851
489
3.2
31
dhc
sto
l
960
-E3
0.3
960
-E
51.
21
10
0+
E00.
00
46
.0
20
+E3
034
457
825
.9-
11-
E42
.4
01
20
+E4
457
825
.9-
89
.0
20+
E88
044
578
25.
9-
96
3.
02
0+
E30
344
578
25.
9-
61
1p
qr
aso
m
85
0-E
17.
19
60
-E0
3.3
01
70-E
00.
4-
025
.0
30+
E17
112
847
95.
1-
90
-E7
2.1
93
0+
E1
284
79
5.
1-
88
.0
30
+E
83
11
284
795
.1-
981
.0
30
+E5
711
284
795
.1-
51
2pq
ras
om
91
0+
E03
.8-
920+
E17
.2-
01
4.
11
01
+E
380
385
980
2.
580-
E04
.2
11
01
+E
385
980
2.5
ro
rr
e l
ata
f29
.2
01
+E2
182
859
802
.5
73
02
le
wo
p
1-
E7
9.
60
10
1-
E85
.4
11
11-
E06
.3-
00
5.
02
0-E
796
569
21
05
.3-
11-
E24
.9
01
20
-E
569
210
5.3
-0
1.
120
-E9
325
692
105
.3
-0
17
0.0
20
-E
33
756
921
05
.3
-81
1l
am
irp
01
0 11-
E00
.1
11
11
-E0
6.7
-1
10
1-E
31.
1-
046
.0
20-
E01
067
633
73.
3-
01
-E
41.
18
20-
E67
633
73.
3-
03
.1
20
-E
680
676
337
3.3
-8
80
.0
20
-E
321
676
337
3.3
-7
12
la
mir
p21
990
-E4
3.4
-9
90-
E73
.3
01
90
-E0
3.1
-0
23.
11
0-E
665
385
575
3.1
-90-
E36
.1
91
0-
E48
557
53
.1
-08
.1
10
-E9
22
38
557
53.
1-
91
2.
01
0-
E69
638
557
53.
1-
81
3la
mir
p
90-
E28
.9
01
01
-E
08.
801
90-
E89
.1
-0
83
.1
10
-E2
893
809
064
.7-
90
-E
81.
19
10
-E3
809
064
.7-
07
.1
10-
E49
838
090
64.
7-
01
01
.0
10-
E08
148
090
64
.7
-2
14l
ami
rp
9
E4
6.
20
16
0-E
09.
1-
01
60-E
01
.3-
25
4.
03
0+E
462
80
52
551
.6-
01-
E39
.5
02
30
+E
805
255
1.6
-9
9.
030
+E3
828
052
551
.6-
02
10
.0
30
+E5
92
80
525
51.
6-
61
1c
la
mi
rp
95
0- -
E02
.2-
95
0-
E64.
1-
95
0-E
74.
1-
09
2.0
30+
E08
767
031
55.
3-
90-
E90
.4
61
30+
E77
031
55.
3-
00
.1
30
+E
629
670
315
5.3
-9
110
.0
30
+E
729
670
315
5.3
-8
12c
lam
irp
95
0
85
.3
-01
70-
E03.
3-
01
70
-E0
6.3
-0
22.
02
0+E
246
232
327
2.4
-01
-E3
8.8
11
20+
E33
232
72.
4-
98
.0
20
+E
57
62
323
27
2.
4-
01
10.
020
+E
87
623
232
72.
4-
71
5c
la
mir
p9
60
-E E
42.
2-
01
50
-E
02.
101
50-
E02
.1-
04
3.0
40
+E6
779
249
038
.1-
01-
E5
9.
63
140
+E0
349
038
.1-
88
.0
40+
E46
792
490
38.
1-
11
30
.0
40+
E88
792
490
38.
1-
41
8c
la
mir
p8
40
-
91
0-
E40
.1
92
0-
E04
.1
11
30-
E00
.1-
089
.11
70
+E
698
744
447
3.1
01-
E6
3.4
82
70+
E84
444
73.
10
5.
97
0+
E01
974
444
73.
162
36
.6
70+E
598
744
44
73
.1
91
74
vf5
2q
93
0-
E74
.1
94
0-
E07
.8-
11
50-
E00
.1
09
1.0
50+
E35
858
813
08.
41
1-E
52.
42
150
+E6
881
308
.4
19.
050
+E6
675
881
308
.4
11
10
.0
50+
E45
85
88
130
8.4
21
ltt
ild
aq
0-
E32
.1-
01
90
-E
02.1
-9
90
-E0
2.6
-0
51.
00
0+E
778
71
870
95.
1-
90-
E4
2.4
31
00
+E
818
709
5.1
-3
9.
000
+E9
887
187
095
.1-
11
00
.0
00
+E9
397
187
095
.1-
81
ori
faq
98
95
0-
E05
.3-
01
60-
E00
.8
01
60
-E0
0.
21
55
.0
40+
E53
024
325
36.
19
0-E
16.
41
24
0+
E24
325
36
.1
00.
140
+E3
402
432
536
.1
91
60
.0
40
+E
730
243
253
6.1
62
md
nab
q
40-
E07
.2-
01
50
-E0
0.6
-01
40-
E0
2.1
-0
63
.0
50+
E72
060
21
74
6.1
90-
E5
6.1
51
50+
E60
217
46.
159
.0
50+
E12
060
217
46.
131
10
.0
50+E
51
060
21
74
6.1
41
fn
oc
aeb
q9 0
16
0-
E0
8.2
-01
60
-E
01.
20
16
0-E0
0.1
-1
45
.0
30
+E
820
800
200
1.3
01-
E2
4.2
91
30+
E80
020
01.
31
9.
03
0+
E94
080
020
01.
321
20
.0
30
+E
810
80
02
001
.3
81
d3
ero
bq
94
0-E
83.
101
60-
E00
.5
-0
160
-E0
0.5
-0
73.
04
0+E
268
411
573
8.2
90-
E5
3.
17
140
+E5
115
738
.2
69
.0
40
+E
758
411
57
38
.2
71
40.
040
+E7
584
115
738
.2
12
yd
na
rbq
53
0+E
77.
35
30
+E
77.
35
30
+E77
.3
081
.1
70
+E8
003
259
876
.6
90-
E03
.2
92
70
+E
392
397
6.6
01
.1
70
+E
462
392
397
6.6
82
61.
070
+E6
62
392
39
76.
60
4ir
pac
q
860
-E1
0.3
-0
170
-E0
6.1
01
70
-E0
4.1
-0
15
.0
20
+E
10
33
435
621
.2
90-
E34
.1
71
20
+E
343
562
1.2
99.
02
0+E
713
343
562
1.2
61
90.
02
0+E
782
343
562
1.2
42
62
2e
q
94
0-E
68.
301
50
-E
00.
70
150
-E0
2.1
92
9.
14
0+E
416
963
067
6.8
90-
E6
5.2
03
40
+E0
730
676
.8
04
.1
40+
E48
696
306
76.
82
344
.0
40
+E6
269
630
676
.8
23
rca
mat
eq
83
0-
E93
.9
93
0-
E14
.3
01
50-
E00
.9-
06
5.
15
0+
E16
064
741
37.
89
0-E
29.
27
250
+E7
474
137
.8
03
.1
50
+E2
046
474
137
.8
42
04
.0
50
+E2
506
474
13
7.8
23
08f
fff
fq
910
+E2
9.3
91
0+
E65
.1
15
1.
190
+E8
06
41
366
54.
71
0+E
78.
313
90
+E5
136
654
.7
01
.1
90
+E4
674
136
654
.7
52
ro
rre
da
er
SPM
na
lpr
ofq
83
0+
E3
1.
59
20
+E
06.
31
50
.2
11+
E78
485
097
00.
10
1-E
54.
13
31
1+E
950
970
0.1
01.
111
+E3
258
50
97
00.
15
2ro
rre
da
er
SPM
np
xd
rfg
q
10
-E0
5.4
91
0-
E03
.1
01
20
-E0
0.2
-0
53.
18
0+E
540
463
961
0.1
-90-
E1
0.1
12
80+
E4
63
961
0.
1-
04
.1
80+
E23
046
396
10.
1-
02
11.
08
0+
E74
046
396
10.
1-
91
51w
org
q9
00+
E54
.3
91
0-E
00.
401
20-
E00
.2-
00
1.
28
0+E
54
35
982
69
4.
1-
01
-E
20.
642
80
+E5
982
694
.1-
07
.1
80+
E50
359
826
94.
1-
32
91
.0
80+
E74
359
826
94.
1-
32
22
wor
gq
8
-E
46
.1
-0
120
-E0
5.1
-0
12
0-E
07
.3-
03
8.0
70+
E63
831
789
72.
4-
90
-E
09.
41
270
+E4
178
972
.4-
01
.1
70+
E15
83
17
897
2.4
-12
50
.0
70+
E37
83
178
972
.4-
81
7wo
rgq
91
0
MPI
cM
KE
SO
MD
PM
PB .
pm
oC
KE
SO
M .p
mo
CX
EL
PC .
pm
oC
XE
LP
C
56
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Table 3.3: McIPM Performance on QO Test Set (III).
bor
PD
PM
PB
evit
cej
bO
retI
em
aN
.c
es
UP
Ce
vitc
ejb
Or
etI.
ce
s U
PC
evit
cej
bO
.di
se
Rr
etIe
vitc
ejb
OU
PC
.c
es
reti
#
1>
q.r
effiD
.n
giS
stigi
d.r
effiD
.n
giS
stigi
d.r
effiD
.n
giS
stigi
d
91
0-E5
0.2
520
+E09
.3
01
30
-E0
0.6
-2
47
.0
70
+E59
77
387
435
.2
01-
E54
.6
13
70
+E8
387
435.
201
.1
70
+E
014
722
84
35
.2
42
01.
07
0+
E98
773
874
35.
25
2l
ea
rsi
q
E8
6.8
11
01
-E0
1.7-
01
90-E
13.
1-
042
.0
30-
E08
671
452
48.
7-
90
-E
31.
36
13
0-
E90
452
48.
7-
89
.0
30
-E
587
424
52
48
.7-
81
10.
030
-E2
470
345
248
.7-
81
dne
lbc
pq
01
01-
920
-E0
6.1
-9
20-
E0
7.
10
130
-E00
.6-
26
2.
170
+E6
104
193
051
.1
90
-E
29.
132
70
+E4
193
05
1.
100
.1
70
+E3
304
193
051
.1
02
90
.0
70
+E0
104
193
051
.1
82
1ie
obc
pq
920
-E2
6.4
93
0-
E07
.8-
93
0-E
05.
9-
01
7.0
60+
E83
52
26
917
1.8
90-
E9
2.4
32
60+
E3
26
917
1.8
01.
16
0+
E15
422
691
71.
85
25
0.0
60
+E
344
226
917
1.8
54
2i
eob
cpq
920
-E2
2.2
01
30
-E0
5.2
01
30
-E0
7.1
-0
78
.0
60
+E8
774
783
402
.6
01-
E04
.7
32
60
+E5
783
402.
60
2.1
60
+E
308
478
3402
.6
32
91.
06
0+
E1
674
78
34
02.
66
2r
ia
ts
cp
q
01
30-
E04
.1
92
0-
E97
.1-
11
40-
E00
.4
29
7.4
60+
E68
986
858
27.
40
1-E
36.
31
36
0+
E96
858
27.
40
3.
360
+E7
088
685
82
7.
415
49
.0
60+
E09
986
858
27.
47
3o
nto
lip
q
80
0+
E00
.0
11
01-E
00
.3
11
00
+E0
0.0
001
.0
00
+E
000
05
78
173
.4
11-
E86
.5
700
+E0
578
17
3.
468
.0
00
+E3
000
578
173
.4
60
0.
00
0+
E00
005
781
73.
40
2t
set
pq
0-E
06.
2-
970
-E0
3.5
01
70
-E
06.
2-
044
.0
20+
E47
999
516
66.
2-
90
-E6
5.2
71
20+
E00
616
66.
2-
99
.0
20
+E1
299
951
666
.2-
71
20
.0
20
+E0
000
061
666
.2-
32
ep
ice
rq
01
7
0-
E95
.1
11
11-
E43
.9-
11
11
-E1
4.9
-0
23
.0
30-
E24
933
59
31
8.5
-90-
E0
0.3
71
30-
E8
15
931
8.5
-79
.0
30
-E6
784
359
318
.5-
71
20
.0
30
-E3
884
359
318
.5-
91
502
csq
01
9
90
0+
E26.
111
20-
E00
.1-
11
20-
E00
.1-
03
6.
080
+E8
383
973
710
.2
01-
E43
.1
22
80
+E4
973
710.
20
0.
18
0+E
738
397
3710
.2
02
50.
080
+E7
38
39
737
10.
29
15
2rg
acs
q
81
0-
E69.
301
20-
E0
1.
1-
01
20
-E05
.1-
06
3.
070
+E40
68
495
686
.2
90-
E01
.2
22
70
+E9
495
686.
201
.1
70
+E
395
849
568
6.2
81
20.
07
0+
E98
584
956
86.
27
17
rg
acs
q
81
0-
E3
6.3
01
30-
E00.
401
30
-E
00.
21
78
.0
70+
E73
619
628
86
.1
90
-E2
0.2
82
70+
E29
628
86.1
00.
17
0+
E14
619
628
86.
142
21
.0
70+
E93
619
628
86.
18
21m
xfc
sq
81
0-
E32.
401
30-E
00.
311
30
-E0
0.2
306
.1
70
+E
77
51
616
777
.2
01-
E24
.2
23
70
+E2
616
777
.2
03
.1
70+
E08
516
167
77.
20
372
.0
70
+E9
751
616
777
.2
73
2m
xf
cs
q
81
0-E9
3.5
01
30
-E0
0.8
01
20-
E00
.1
49
3.
27
0+
E16
445
361
80.
39
0-E
00.
14
37
0+
E55
361
80.
305
.1
70
+E9
644
536
180
.3
23
63
.0
70+E
17
44
53
618
0.3
53
3m
xf
csq
85
0-E
17.
401
70
-E0
0.2
11
70
-E0
0.1
07
3.
030
+E9
255
905
088
.1
11-
E10
.6
51
30
+E6
905
088
.1
49
.0
30
+E
135
590
508
8.1
21
20.
030
+E
03
559
050
88.
15
1o
ipr
ocs
q
960-
E22
.7-
96
0-E
77.
19
60-E
73.
3-
761
.1
20
+E
227
100
654
0.9
90
-E1
8.
44
22
0+E
100
654
0.9
01.
12
0+E
998
100
654
0.9
32
80.
02
0+E
583
100
654
0.9
62
8s
rcs
q
540
-E04
.2
54
0-
E0
4.
25
40
-E0
4.
293
18
9.
600
+E4
626
624
666
.8
50-
E71
.2
05
100
+E7
666
666
.8
19
.0
00
+E
647
66
66
666
.8
01
50
.0
00
+E3
476
666
666
.8
51
1ds
csq
98
0-
E06
.8
-9
70
-E
97.1
-9
70
-E7
8.1
-0
07
.0
10
+E6
804
128
080
.5
90-
E82
.4
41
10
+E4
128
080
.5
99
.0
10+
E70
931
28
08
0.5
41
11
.0
10
+E9
983
128
080
.5
12
6ds
csq
960
-E1
4.
4-
01
70
-E
03.
4-
01
70-
E00
.2-
03
1.1
20+
E14
475
367
04.
90
1-E
02.
62
12
0+
E75
367
04.
90
2.
120
+E8
937
536
704
.9
41
42
.0
20+
E1
24
753
670
4.9
61
8d
scs
q
950
-E2
1.1
-8
50-
E10
.2
11
70
-E00
.1-
12
5.
030
+E21
11
168
514
.1
01-
E63
.1
71
30
+E1
16
85
14.
199
.0
30
+E
313
116
851
4.1
71
50.
03
0+
E11
111
685
14.
11
21
pa
tcs
q
960
-E0
3.2
85
0-
E85
.5
11
00
+E
00.
00
91
.1
30+
E77
946
205
37
.1
11
-E1
6.3
31
30+
E56
205
37.
103
.1
30+
E53
556
205
37.
151
32
.0
30+
E77
946
205
37.
19
12p
atc
sq
85
0-
E08
.1
01
70-
E00
.9-
01
60
-E0
1.1
-0
15
.1
30
+E
02
86
457
834
.1
01-
E45
.9
31
30
+E7
457
834
.1
04
.1
30+
E11
864
578
34.
15
123
.0
30
+E9
086
457
834
.1
02
3p
atc
sq
91
0-E
25.
69
10
-E
62
.2
01
30
-E0
0.9
21
8.
170
+E8
430
081
841
.8
90-
E03
.1
62
70
+E1
081
841
.8
00
.1
70+
E47
50
08
184
1.
86
14
0.
070
+E7
530
081
841
.8
12
ab
esq
93
0-E
90.
101
40
-E0
1.7
940
-E0
6.7
-0
64.
050
+E1
981
387
002
.7
90-
E64
.1
62
50
+E
238
700
2.7
00
.1
50
+E
269
138
700
2.7
52
40.
050
+E
51
813
870
02.
79
2b
1er
ahs
q
840
-E0
9.2
95
0-
E02
.1
01
50-
E01
.1
08
2.0
40+
E01
719
630
71.
19
0-E
32.
371
40+
E29
630
71.
179
.0
40
+E2
27
19
630
71.
171
20
.0
40+E
12
719
630
71.
17
1b
2er
ahs
q
840
+E1
3.4
-9
30+
E02
.7
11
20
+E00
.1-
65
0.
321
+E1
348
63
62
75.
10
1-E
40.
183
21
+E8
63
62
75.
103
.1
21
+E
305
863
627
5.1
33
43.
02
1+E
03
486
362
75.
193
ll
eh
sq
82
0-E
14.
3-
01
40-
E00.
3-
11
00+
E00
.0
07
0.
160
+E1
43
55
100
24.
20
1-E
33.
151
60
+E5
510
024
.2
01
.1
60
+E8
335
510
024
.2
41
70
.0
60
+E1
43
55
100
24.
251
l4
0p
ihs
q
82
0-
E07
.2
11
00+
E00
.0
11
00
+E0
0.
00
47
.0
60
+E
037
639
942
4.2
01-E
30.
15
16
0+E
739
942
4.2
99.
060
+E0
376
39
94
24.
24
15
0.
06
0+E
037
639
942
4.2
51
s4
0pi
hsq
92
0-
E08
.1-
01
30
-E
00
.2
01
30
-E0
4.1
-0
11.
660
+E0
816
040
673
.2
90-
E85
.1
41
60
+E
604
067
3.
201
.2
60
+E
002
604
067
3.2
31
65.
16
0+
E66
160
406
73.
25
1l
80p
ihs
q
82
0-
E17
.4
93
0-E
09.8
01
30-
E09
.1-
07
7.1
60+
E92
588
275
83.
20
1-E
34
.8
51
60+
E98
275
83.
203
.1
60
+E
81
688
275
83.
231
24
.0
60+E
01
588
275
83.
27
1s
80p
ihs
q
92
0-
E38
.1
01
30-
E08
.2-
93
0-E
07
.4
-1
87.
11
60
+E
718
56
78
81
0.3
90-
E9
8.1
81
60
+E
66
78
810
.3
00
.3
60+
E98
756
788
10.
37
136
.2
60
+E0
775
678
810
.3
91
l21
pih
sq
82
0-
E30
.5
93
0-
E05
.4
01
40-
E00
.4-
19
2.
26
0+
E79
422
696
50.
30
1-E
77.
19
160
+E3
269
650
.3
04
.1
60
+E2
452
269
650
.3
61
45
.0
60
+E
39
422
696
50
.3
81
s21
pih
sq
91
0-
E20
.2-
01
20
-E
00.2
-01
20-
E03
.2-
01
8.3
70+
E20
285
405
73.
20
1-E
08.
31
270
+E8
540
573
.2
03.
170
+E2
818
540
573
.2
81
12
.0
70
+E
971
854
057
3.2
91
ar
re
isq
92
0-
E33
.4
70
0+E
55.1
11
40-
E00
.4-
098
.0
60+
E76
572
545
89.
79
0-E
69.
11
260
+E8
25
45
89.
70
3.
160
+E3
013
454
589
.7
42
31
.0
60
+E
365
725
458
9.7
81
ri
ats
q
01
70
-E0
0.8
-9
60-
E08
.9
95
0-
E91
.1
-1
99
.0
30+
E8
00
48
381
14.
69
0-
E1
6.2
71
30+
E48
381
14.
669
.0
30+
E60
148
381
14.
631
60
.0
30+E
98
838
381
14.
68
1t
adn
ats
q
97
0-E
25.
59
50
-E
68.
18
80
-E4
1.2
-0
81
.0
80
-E1
311
794
341
.2
80-
E08
.1
81
70
-E
507
013
7.5
49
.0
50
-E
133
055
116
8.1
41
10.
000
+E0
00
00
000
00.
08
186
2s
13
0-E
00.
67
00
+E
04.6
920
-E0
9.3
-1
86
.6
70+
E60
046
862
58.
2-
60
-E
61.
26
17
0+E
468
625
8.2
-02
.2
70
+E4
067
586
258
.2-
41
53
.0
70
+E5
404
686
258
.2-
21
1t
ada
ts
0
70
-E7
2.1
-5
30
-E0
1.1
11
00+
E00
.0
14
1.
810
+E3
784
666
262
.3-
90-
E52
.5
02
10
+E
566
626
2.3
-0
2.
210
+E0
22
86
552
62.
3-
61
75
.0
10+
E3
78
466
626
2.3
-22
2t
ad
ats
9
E0
9.
4-
11
90-
E0
0.
111
90-
E00
.2-
03
5.
41
10
+E1
592
549
775
.3-
11
-E
96.
981
10+
E35
497
75.
3-
06
.4
10
+E
059
254
977
5.3
-0
22
5.
11
0+E
359
254
977
5.3
-1
23
ta
dat
s9
80-
23
0+
E02
.2
23
0+E
02.
22
30
+E0
2.2
019
.1
50
+E
876
922
492
5.1
90-
E0
3.1
01
50+
E6
53
415
5.1
00
.1
50+
E77
455
341
55.
18
54
.0
50
+E0
745
534
155
.1
11
1pq
cts
830
-E9
0.2
-8
30
-E1
8.1
-8
30-
E28
.1-
021
.4
40
+E1
905
137
232
.2
90-
E27
.4
84
0+
E31
372
32.
20
1.1
40+
E67
231
372
32.
29
87
.61
40+
E27
231
37
232
.2
11
2pqc
ts
80
0+
E00
.0
11
00
+E
00.
011
00
+E0
0.0
040
.0
00+
E0
00
000
00
00
.0
90-
E90
.4
400
+E0
000
00
0.
058
.0
00
+E
000
000
000
0.0
30
0.
00
0+
E00
000
000
00.
08
em
at
66
0-E
19.
8-
660
-E2
9.8
-1
10-
E88
.4
31
80
.3
60
0+
E23
17
90
061
1.1
90-
E6
5.8
44
00+
E80
006
11.
103
.8
00+
E86
970
006
11.
104
44.
40
0+
E39
717
883
06.
19
1hb
u
880
-E1
4.
40
10
1-
E00
.5
09
5.
000
+E1
441
126
693
.1-
01-
E46
.5
41
00
+E
112
669
3.1
-3
9.
000
+E6
34
11
266
93
.1
-3
1ro
rre
da
er
SPM
se
ula
v
12
0+
E10
.1
20
0+
E84
.9
12
0+E
10.
11
17
.3
10+
E19
19
01
766
6.9
30-
E7
1.1
91
20+
E62
407
79.
105
.1
20+
E64
213
941
60.
131
52
.0
20+
E49
552
407
79.
15
2o
ay
800
+E
00
.0
11
00+
E00
.0
11
00
+E0
0.0
00
1.0
00
+E0
000
000
521
.4-
11-
E12
.4
70
0+
E00
005
21.
4-
58
.0
00
+E
000
000
052
1.4
-5
00
.0
00
+E
000
000
052
1.4
-7
12c
ive
cez
KE
SO
MX
EL
PC
DP
MP
B .p
mo
CK
ES
OM .
pm
oC
XE
LP
C .p
mo
CM
PIc
M
57
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Table 3.4: McIPM Performance on Difficult QO Problems
DP
MP
B.
bor
P
evit
cej
bO
retI
em
aN
UP
C
.c
es
evit
cej
bO
re tI
UP
C
.c
es
evit
cej
bO
.di
se
Rr
e t Ie
vitc
e jb
OU
PC
.c
es
reti
#
1>
q.r
effiD
.n
giS
stigi
d.r
effiD
.n
giS
stigi
d.r
effiD
.n
giS
stigi
d
E26
3.2
-4
30-
E2
63
.2-
43
0-
E1
63.
2-
34
5.76
34
10
+E1
358
21
95
21
.2
80
-E
46
.4
49
10
+E
76
76
52
1.
23
9.
56
10+
E15
966
7652
1.2
04
76
.9
81
0+
E762
767
652
1.2
35
2d
yo
b4
30
-
86
0-E
433
.1
75
0-
E3
39.
18
60
-E
434
.1
02
7.
22
10
+E6
660
04
22
16.
39
0-E
44.
10
21
0+
E20
422
16.
309
.6
10
+E3
999
14
221
6.3
22
19
.1
10+
E001
204
221
6.3
32
1tew
sil
0-
E5
19.
1-
750
-E
51
4.1
86
0-
E51
2.2
-3
74.
33
10
+E
51
96
87
58
49
.4
01
-E
68
.8
42
10
+E
58
75
849
.4
01
.8
10
+E3
601
08
58
49.
472
43.3
10+
E00
748
758
49
.4
14
01t
ews
il
86
-E2
89.
2-
76
0-
E3
22
.6
-7
50
-E2
79.
2-
055.
03
10
+E9
186
89
32
59
.4
80
-E
12
.1
32
10
+E
75
93
25
9.
409
.9
10+
E6
95
089
325
9.4
63
83
.2
10+
E00
1759
325
9.4
92
11te
wsi
l7
50
50-
E0
58.
2-
73
0-E
251
.1
96
0-
E0
93
.2
-2
91
.33
30
+E
58
24
72
96
37.
19
0-E
64.
21
33
0+
E47
2963
7.
10
0.
83
0+
E70
858
29
63
7.1
72
1.
53
0+E
162
472
963
7.1
46
21te
wsi
l8 8
70
-E0
84
.6
-7
60
-E
508
.2
87
0-
E084
.5
-0
1.8
11
0+E
846
670
899
4.2
01
-E1
4.8
51
10
+E
67
08
994
.2
09
.6
10
+E
354
97
08
994
.2
22
51.
110
+E0
016
708
994.
23
12te
wsi
l
60
-E9
82
.2
-9
70
-E
001
.1
86
0-
E9
82
.2
-2
49
.32
10
+E
98
22
22
10
05.
29
0-E
46.
20
21
0+
E02
2100
5.
20
8.
71
0+
E99
322
2100
5.2
32
58.1
10+
E00
00
221
005
.2
22
3te
wsi
l8
60
-E3
16
.1-
76
0-
E55
3.3
86
0-
E32
5.1
-1
26.1
21
0+
E3
16
31
10
00
5.
29
0-
E7
4.
29
11
0+
E2
110
005
.2
00.
710
+E8
696
11
00
05.
232
67.1
10+
E09
021
10
005
.2
12
4t
ews
il
8
60-
E7
20
.6-
65
0-
E2
81.
37
60
-E
720
.6
-0
19.
32
10
+E
720
952
430
5.2
01
-E9
0.4
12
10
+E
35
24
305
.2
03
.7
10
+E
748
092
4305
.2
12
87.1
10+
E00
035
243
05.
212
5t
ews
il
7
60
-E0
31
.5
-7
60
-E
791
.3
76
0-
E0
35
.5
-2
47
.12
10
+E
03
13
57
59
94.
29
0-E
02.
39
11
0+
E84
75
99
4.2
06
.6
10+
E72
36
57
599
4.2
12
29
.1
10+E
006
7475
994
.2
32
6te
wsi
l7 8
60
-E0
40
.5
53
0-
E16
2.6
86
0-
E0
48
.5
06
4.
22
20
+E
694
880
488
9.4
90
-E2
5.8
52
20
+E
980
488
9.4
00
.1
120
+E4
064
174
889
.4
83
85.
120
+E0
809
804
889
.4
81
7t
ews
il
13
0+E
034
.6
74
0-
E2
31.
29
70
-E
009
.9
-2
80.
13
20
+E
999
500
744
1.7
01
-E5
7.4
13
30
+E6
007
441
.7
05
.8
20
+E
32
37
207
441
.7
92
65.
420
+E0
0950
074
41.
78
58te
wsi
l
850
-E
00
0.4
74
0-E
643
.2
01
70
-E
00
9.8
063
.72
30
+E
00
62
15
23
69.
19
0-E
45.
44
33
0+
E31
523
69.
10
6.
93
0+
E64
941
5236
9.1
23
17.4
30+
E90
62
152
369
.1
06
9te
wsi
l
91
0-E
065
.2
-01
30
-E
00
0.
801
20
-E
000
.1
06
8.
17
0+E
652
392
397
6.
601
-E8
2.
20
37
0+E
392
397
6.6
01
.1
70
+E
462
392
3976
.6
82
61.0
70+
E66
239
239
76.
604
irpa
cq
76
0-
E38
.1
-7
60
-E
58
.1
-7
60
-E
58.
1-
18
1.7
00
+E
87
25
86
66
66
.8
70
-E
50.
99
00
+E
766
666
6.8
19
.0
00
+E
647
666
6666
.8
01
50.
000
+E3
476
666
666.
85
11
dsc
sq
830
-E5
98
.4
94
0-
E8
43
.3-
940
-E8
40.
4-
02
5.
472
50
+E
01
555
341
55
.1
90
-E9
3.4
950
+E6
534
155
.1
00
.1
50
+E7
745
534
155
.1
854
.0
50+E
074
553
415
5.1
11
1p
qct
s
86
0-
E8
80
.4
11
0+
E5
51
.9-
01
80-
E01
8.2
14
4.
320
+E
19
55
240
779
.1
60-
E78
.5
02
20
+E
624
07
79
.1
05
.1
20
+E
642
139
416
0.1
31
52
.0
20+E
495
524
077
9.1
52
oa
y
MPI
cM
DP
MP
B .p
mo
CK
ES
OM .
pm
oC
KE
SO
MX
EL
PC
XE
LP
C .p
mo
C
58
Chapter 4
Parametric Quadratic
Optimization
In this chapter we present an IPM and optimal partition based technique and
provide a polynomial time algorithm for conducting parametric analysis of convex
Quadratic Optimization problems. The novelty of our results is that we allow
simultaneous variation in the coefficient vector of the linear term of the objective
function and in the right-hand side vector of the constraints. The resulting
problem we intend to solve is:
φ(λ) = min f(x, λ)s.t. Ax = b + λ△b
x ≥ 0,
where f(x, λ) is linear or quadratic function of x, and λ is the perturbation
parameter.
A method described in this chapter performs division of the parameter
space into so-called invariancy intervals and provides complete description of
the behavior of φ(λ) and x∗(λ) on each interval. The resulting algorithm allows
solving parametric quadratic (as well as parametric linear) optimization problems
efficiently in polynomial time.
59
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
4.1 Origins of Quadratic and Linear Parametric
Optimization and the Existing Literature
As you already know, research on parametric optimization was triggered when
a variant of parametric convex QO problems was considered by Markowitz [17].
He developed the critical line method to determine the optimal value function
of his parametric problem and applied it to mean-variance portfolio analysis.
The basic result for parametric quadratic programming obtained by Markowitz
is that the optimal value function (efficient frontier in financial terminology) is
piecewise quadratic and can be obtained by computing successive corner portfo-
lios while in between these corner portfolios the optimal solutions vary linearly.
Non-degeneracy was assumed and a variant of the simplex method was used for
computations.
Difficulties that may occur in parametric analysis when the problem is de-
generate are studied extensively in the LO literature. In case of degeneracy, the
optimal basis does not need to be unique and multiple optimal solutions may
exist. While simplex methods were used to perform the computations in earlier
studies (see e.g., Murty [24] for a comprehensive survey), recently research on
parametric analysis was revisited from the point of view of IPMs. For degener-
ate LO problems, the availability of strictly complementary solutions produced
by IPMs and use of optimal partitions associated with such solutions allow one
to overcome many difficulties associated with the use of bases. Alder and Mon-
teiro [1] pioneered the use of IPMs in parametric analysis for LO (see also Jansen
et al. [16]). Berkelaar, Roos and Terlaky [4] emphasized shortcomings of using
optimal bases in parametric LO showing by an example that different optimal
bases computed by different LO packages give different optimality intervals.
Naturally, results obtained for parametric LO were extended to the convex
QO. Berkelaar et al. [3] showed that the optimal partition approach can be gen-
eralized to the quadratic case by introducing tripartition of variables instead of
bipartition. They performed the sensitivity analysis for the cases when perturba-
60
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
tion occurs either in the coefficient vector of the linear term of the objective value
function or in the right-hand side of the constraints. We show that the results
obtained in Berkelaar, Roos and Terlaky [4] and Berkelaar et al. [3] can be gener-
alized further to accommodate simultaneous perturbation of the data even in the
presence of degeneracy. The theoretical results allow us to present a universal
computational algorithm for the parametric analysis of LO/QO problems.
It is worthwhile to mention that some encouraging results already exist
for parametric convex Conic Optimization (CO). For convex conic problems a
feasible point is restricted to be in a closed, convex, solid and pointed cone.
Yildirim [34] extended the concept of the optimal partition to such conic prob-
lems. He proved that the optimal value function is quadratic and presented
auxiliary conic problems for computing derivatives and boundaries of invariancy
intervals.
The chapter is organized as follows. In Section 4.2 some elementary con-
cepts related to convex QO problems are reviewed. Simple properties of the
optimal value function are summarized in Section 4.3. Section 4.4 is devoted to
deriving properties of the optimal value function in order to formulate a com-
putational algorithm. It is shown that the optimal value function is continuous
and piecewise quadratic, and an explicit formula is presented to identify it on
the subintervals. Criteria for convexity, concavity or linearity of the optimal
value function on these subintervals are derived. We investigate the first and
second order derivatives of the optimal value function as well. Auxiliary LO
problems can be used to compute the left and right derivatives. It is shown that
the optimal partition on the neighboring intervals can be identified by solving
an auxiliary self-dual QO problem. The results are summarized in a computa-
tional algorithm. Specialization of the method to LO problems is described in
Section 4.5.
61
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
4.2 Properties of Convex QO Problems
In this section we review the concepts and properties of convex QO problems
that are necessary for development of the parametric analysis. Primal (QP) and
dual (QD) QO problems are already defined by (2.1.1) and (2.1.2), respectively.
The first property we would like to look at directly follows from the com-
plementarity condition xizi = 0 for all i ∈ {1, 2, . . . , n} (zero duality gap).
Lemma 4.2.1 For two optimal solutions (x∗, y∗, z∗) and (x, y, z) of (QP ) and
(QD) it holds that Qx∗ = Qx , cT x∗ = cT x and bT y∗ = bT y and consequently,
xT z∗ = zT x∗ = 0. (4.2.1)
Proof. See e.g., Berkelaar et al. [3] and Dorn [7]. ¤
The concept of optimal partition is used extensively in this chapter. It is
very important to understand its meaning precisely. The optimal partition of the
index set {1, 2, . . . , n} is defined as
B = { i : xi > 0 for an optimal solution x ∈ QP∗},N = { i : zi > 0 for an optimal solution (x, y, z) ∈ QD∗},T = {1, 2, . . . , n} \(B ∪ N ),
and denoted by π = (B,N , T ). Berkelaar et al. [3] and Berkelaar, Roos and
Terlaky [4] showed that this partition is unique. Another concept related very
closely to optimal partition is the support set of a vector v which is defined
as σ(v) = {i : vi > 0}. We can establish a link between the optimal partition
and a maximally complementary solution that was defined in Section 2.1. An
optimal solution (x, y, z) is maximally complementary if it possesses the following
properties:xi > 0 if and only if i ∈ B,zi > 0 if and only if i ∈ N .
For any maximally complementary solution (x, y, z) the relations σ(x) = B and
σ(s) = N hold. The existence of a maximally complementary solution is a
62
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
direct consequence of the convexity of the optimal sets QP∗ and QD∗. It is
known that IPMs find a maximally complementary solution in the limit, see e.g.,
McLinden [19] and Guler and Ye [12].
We are going to consider the parametric problems with only one parameter
in this thesis. But let us look at the general multiparametric convex QO problem
first to understand better what has been done in the area before:
min (c + λc△c)T x + 12xT Qx
(QPλb,λc) s.t. Ax = b + λb△b,
x ≥ 0.
Here △b ∈ Rm and △c ∈ R
n are nonzero perturbation vectors, and λb and λc
are real parameters. The optimal value function φ(λb, λc) denotes the optimal
value of (QPλb,λc) as the function of the parameters λb and λc. As we already
mentioned, Berkelaar, Roos and Terlaky [4] and Berkelaar et al. [3] were the
first to analyze parametric convex QO by using the optimal partition approach
when variation occurs either in the right-hand side or in the linear term of the
objective function data, i.e., either △c or △b is zero. In these cases the domain
of the optimal value function φ(λb, 0) (or φ(0, λc)) is a closed interval of the real
line and the function is piecewise convex (concave) quadratic on its domain. The
authors presented an explicit formula for the optimal value function on these
subintervals and introduced the concept of transition points that separate them.
They proved that the optimal partition is invariant on the subintervals which
are characterized by consecutive transition points. The authors also studied the
behavior of the first and second order derivatives of the optimal value function
and proved that the transition points coincide with the points where the first or
second order derivatives do not exist. It was proven that by solving auxiliary self-
dual QO problems, one can identify the optimal partitions on the neighboring
subintervals.
63
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
4.3 The Optimal Value Function in Simultane-
ous Perturbation Sensitivity Analysis
In this section, we introduce explicitly the perturbed convex QO problem when
perturbation occurs simultaneously in the right-hand side data and in the lin-
ear term of the objective function of (QP ). In the problem (QPλb,λc) that
was mentioned in the pervious section, λb and λc are independent parame-
ters. In this thesis we are only concerned with the case when they coincide,
i.e., λb = λc = λ. Consequently, the perturbation takes the form λh, where
h = (△bT ,△cT )T ∈ Rm+n is a nonzero perturbing direction and λ ∈ R is a
parameter. Thus, we define the following primal and dual perturbed problems
corresponding to (QP ) and (QD), respectively:
min (c + λ△c)T x + 12xT Qx
(QPλ) s.t. Ax = b + λ△b,x ≥ 0,
(4.3.1)
min (b + λ△b)T y − 12xT Qx
(QDλ) s.t. AT y + z − Qx = c + λ△c,z ≥ 0,
(4.3.2)
Let QPλ and QDλ denote the feasible sets of the problems (QPλ) and
(QDλ), respectively. Their optimal solution sets are analogously denoted by
QP∗λ and QD∗
λ. The optimal value function of (QPλ) and (QDλ) is
φ(λ) = (c + λ△c)T x∗(λ) +1
2x∗(λ)T Qx∗(λ) = (b + λ△b)T y∗(λ)− 1
2x∗(λ)T Qx∗(λ),
where x∗(λ) ∈ QP∗λ and (x∗(λ), y∗(λ), z∗(λ)) ∈ QD∗
λ. Further, we define
φ(λ) = +∞ if QPλ = ∅,φ(λ) = −∞ if QPλ 6= ∅ and (QPλ) is unbounded.
Let us denote the domain of φ(λ) by
Λ = {λ : QPλ 6= ∅ and QDλ 6= ∅}.
Since it is assumed that (QP ) and (QD) have optimal solutions, it follows that
Λ 6= ∅. Proving the following property of Λ is the first in the sequence of steps
for subdividing the space of λ into subintervals.
64
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Lemma 4.3.1 Λ ⊆ R is a closed interval.
Proof. Let λ 6∈ Λ. There are two cases: the primal problem (QPλ) is feasible
but unbounded or it is infeasible. We only prove the second case, the first one
can be proved analogously. If the primal problem (QPλ) is infeasible then by the
Farkas Lemma (see e.g., Murty [24] or Roos, Terlaky and Vial [26]) there is a
vector y such that AT y ≤ 0 and (b+λ△b)Ty > 0. Fixing y and considering λ as a
variable, the set S(y) ={
λ : (b + λ△b)T y > 0}
is an open half-line in λ, thus the
given vector y is a certificate of infeasibility of (QPλ) for an open interval. Hence,
the union⋃
y S(y), where y is a Farkas certificate for the infeasibility of (QPλ)
for some λ ∈ R, is open. Consequently, the domain of the optimal value function
is closed. We also need to show that this closed set is connected. Let λ1, λ2 ∈ Λ
be two arbitrary numbers. Let (x(λ1), y(λ1), z(λ1)) ∈ QP∗λ1
×QD∗λ1
. Similarly,
(x(λ2), y(λ2), z(λ2)) ∈ QP∗λ2
× QD∗λ2
. For any λ ∈ (λ1, λ2) and θ = λ2−λλ2−λ1
we
have
λ = θλ1 + (1 − θ)λ2.
Let us define
x(λ) = θx(λ1) + (1 − θ)x(λ2),
y(λ) = θy(λ1) + (1 − θ)y(λ2),
z(λ) = θz(λ1) + (1 − θ)z(λ2).
It is easy to check that (x(λ), y(λ), z(λ)) ∈ QP∗λ × QD∗
λ. This implies that the
set Λ is connected and thus Λ is a closed interval. ¤
4.4 Properties of the Optimal Value Function
In this section we investigate the properties of the optimal value function. These
are generalizations of the corresponding properties that have been proven in
Berkelaar et al. [3] for the case when △c = 0 or △b = 0.
65
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
4.4.1 Basic Properties
For λ∗ ∈ Λ, let π = π(λ∗) denote the optimal partition and let (x∗, y∗, z∗) be
a maximally complementary solution at λ∗. We use the following notation that
generalizes the notation introduced in Berkelaar et al. [3]:
O(π) = {λ ∈ Λ : π(λ) = π} ;
Sλ(π) = {(x, y, z) : x ∈ QPλ, (x, y, z) ∈ QDλ,
xB > 0, xN∪T = 0, zN > 0, zB∪T = 0};Sλ(π) = {(x, y, z) : x ∈ QPλ, (x, y, z) ∈ QDλ,
xB ≥ 0, xN∪T = 0, zN ≥ 0, zB∪T = 0};Λ(π) = {λ ∈ Λ : Sλ(π) 6= ∅} ;
Λ(π) ={
λ ∈ Λ : Sλ(π) 6= ∅}
;
Dπ = {(△x,△y,△z) : A△x = △b, AT△y + △z − Q△x = △c,
△xN∪T = 0,△zB∪T = 0}.
The following theorem resembles Theorem 3.1 from Berkelaar et al. [3] and
presents the basic relations between the open interval, where the optimal parti-
tion is invariant, and its closure.
Theorem 4.4.1 Let π = π(λ∗) = (B,N , T ) for some λ∗ denote the optimal
partition and (x∗, y∗, z∗) denote an associated maximally complementary solution
at λ∗. Then,
(i) Λ(π) = {λ∗} if and only if Dπ = ∅;(ii) Λ(π) is an open interval if and only if Dπ 6= ∅;(iii) O(π) = Λ(π) and clO(π) = cl Λ(π) = Λ(π);
(iv) Sλ(π) = {(x, y, z) : x ∈ QP∗λ, (x, y, z) ∈ QD∗
λ} for all λ ∈ Λ(π).
Proof. First let us recall the characteristics of a maximally complementary
solution. Any maximally complementary solution (x∗, y∗, z∗) associated with a
given λ∗ satisfies Ax∗ = b+λ∗△b, AT y∗+z∗−Qx∗ = c+λ∗△c, x∗B > 0, x∗
N∪T = 0,
66
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
z∗N > 0 and z∗B∪T = 0. Let (△x,△y,△z) ∈ Dπ, and define
x = x∗ + (λ − λ∗)△x, (4.4.1)
y = y∗ + (λ − λ∗)△y, (4.4.2)
z = z∗ + (λ − λ∗)△z. (4.4.3)
If λ is in an ǫ-neighborhood of λ∗ for enough small ǫ, then
Ax = b + λ△b,
AT y + z − Qx = c + λ△c,xN∪T = 0,zB∪T = 0,
xB > 0, zN > 0.
(4.4.4)
(i) [⇒] : Let Λ(π) = {λ∗}, and assume to the contrary that Dπ is not
empty. Then, there exists (△x,△y,△z) such that A△x = △b and AT△y+△z−Q△x = △c with △xN∪T = 0 and △zB∪T = 0. Let (x∗, y∗, z∗) be a maximally
complementary solution associated with λ∗, i.e., Ax∗ = b + λ∗△b, AT y∗ + z∗ −Qx∗ = c + λ∗△c, x∗
N∪T = 0, z∗B∪T = 0, x∗B > 0 and z∗N > 0. Let (x, y, z)
be defined by (4.4.1)–(4.4.3). From (4.4.4), one can conclude λ ∈ Λ(π) that
contradicts to the assumption Λ(π) = {λ∗}.(i) [⇐] : Let Dπ = ∅, and suppose to the contrary that λ, λ∗ ∈ Λ(π), with
λ 6= λ∗ and (x, y, z) is a maximally complementary solution at λ. Thus, from
(4.4.1)–(4.4.3) we can compute (△x,△y,△z) and conclude that (△x,△y,△z) ∈Dπ. This contradicts to the fact that Dπ = ∅ and thus Λ(π) = {λ∗}.
(ii) [⇒] : Let λ∗ ∈ Λ(π). Then, there is a maximally complementary
solution (x∗, y∗, z∗) at λ∗. Moreover, since Λ(π) is an open interval, there exists
a λ in an ǫ-neighborhood of λ∗ with λ 6= λ∗ and λ ∈ Λ(π). Let (x, y, z) denote a
maximally complementary solution at λ. From (4.4.1)–(4.4.3), we can compute
(△x,△y,△z) and conclude that (△x,△y,△z) ∈ Dπ 6= ∅.(ii) [⇐] : Suppose that Dπ is non-empty. Then, there exists (△x,△y,△z)
such that A△x = △b, AT△y + △z − Q△x = △c, △xN∪T = 0 and △zB∪T = 0.
On the other hand, a maximally complementary solution (x∗, y∗, z∗) at λ∗ exists
67
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
such that Ax∗ = b + λ∗△b, AT y∗ + z∗ − Qx∗ = c + λ∗△c, x∗N∪T = 0, z∗B∪T = 0,
x∗B > 0 and z∗N > 0. Consider (x, y, z) as defined in (4.4.1)–(4.4.3). For any
λ ∈ R, (x, y, z) satisfies
Ax = b + λ△b, AT y + z − Qx = c + λ△c,
and
xT z = (λ − λ∗)(△xT z∗ + △zT x∗).
From the definitions of π and Dπ, one can conclude that xT z = 0. Thus (x, y, z) is
a pair of primal-dual optimal solutions of (QPλ) and (QDλ) as long as x ≥ 0 and
z ≥ 0, that gives a closed interval around λ∗. Furthermore, for an open interval
Λ, xB > 0 and zN > 0. Let λ′ < λ∗ < λ, where λ′, λ ∈ Λ. If (x′, y′, z′) and (x, y, z)
are defined by (4.4.1)–(4.4.3), then x′B, xB > 0, x′
B∪T = xB∪T = 0, z′N , zN > 0,
z′N∪T = zN∪T = 0. To prove that λ ∈ Λ(π), we need to show that (x, y, z) is not
only optimal for (QPλ) and (QDλ), but also maximally complementary.
Let us assume that the optimal partition π = (B,N , T ) at λ is not identical
to π, i.e., there is a solution (x(λ), y(λ), z(λ)) such that
xB(λ) > 0, zN (λ) > 0, and xT (λ) + zT (λ) 6= 0. (4.4.5)
Let us define
x =λ − λ∗
λ − λ′x(λ) +
λ∗ − λ′
λ − λ′x′,
y =λ − λ∗
λ − λ′y(λ) +
λ∗ − λ′
λ − λ′y′,
z =λ − λ∗
λ − λ′z(λ) +
λ∗ − λ′
λ − λ′z′.
By definition (x, y, z) is optimal for λ∗, while by (4.4.5) it has a positive xi + zi
coordinate in T , contradicting to the definition of the optimal partition π at λ*.
We still need to show that Λ(π) is a connected interval. The proof follows
the same reasoning as the proof of Lemma 4.3.1 and is omitted.
(iii) Let λ ∈ O(π), then by definition π(λ) = π, and hence for λ ∈ Λ there
is a maximally complementary solution (x, y, z) which satisfies Ax = b + λ△b,
68
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
AT y + z − Qx = c + λ△c, xN∪T = 0, zB∪T = 0, xB > 0 and zN > 0, from which
we conclude that λ ∈ Λ(π). Analogously, one can prove that if λ ∈ Λ(π) then
λ ∈ O(π). Consequently, O(π) = Λ(π). Because Λ(π) is a polyhedral set, it
follows that clSλ(π) = Sλ(π). Since Λ(π) is an open interval, we conclude that
clO(π) = cl Λ(π) = Λ(π).
(iv) This result follows from Lemma 2.3 of Berkelaar et al. [3]. ¤
The following two corollaries are direct consequences of Theorem 4.4.1.
Corollary 4.4.2 Let λ2 > λ1 be such that π(λ1) = π(λ2). Then, π(λ) is constant
for all λ ∈ [λ1, λ2].
Corollary 4.4.3 Let (x(1), y(1), z(1)) and (x(2), y(2), z(2)) be maximally comple-
mentary solutions of (QP λ1), (QDλ1
) and (QP λ2), (QDλ2
), respectively. Then,
for any λ ∈ [λ1, λ2]
x(λ) =λ2 − λ
λ2 − λ1
x(1) +λ − λ1
λ2 − λ1
x(2),
y(λ) =λ2 − λ
λ2 − λ1
y(1) +λ − λ1
λ2 − λ1
y(2),
z(λ) =λ2 − λ
λ2 − λ1
z(1) +λ − λ1
λ2 − λ1
z(2)
is a maximally complementary solution of (QP λ) and (QDλ) if and only if
λ1, λ2 ∈ Λ(π).
The importance of Corollaries 4.4.2 and 4.4.3 is that on the intervals where
the optimal partition is constant we can determine a maximally complementary
optimal solution at any point inside such intervals by taking the convex combi-
nation of maximally complementary solutions at any two points of the interval.
The next theorem shows how to determine the endpoints of the interval
Λ(π). It is a direct consequence of Theorem 4.4.1 as well.
Theorem 4.4.4 Let λ∗ ∈ Λ and let (x∗, y∗, z∗) be a maximally complementary
solution of (QPλ∗) and (QDλ∗) with optimal partition π = (B,N , T ). Then the
69
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
left and right extreme points of the closed interval Λ(π) = [λℓ, λu] that contains
λ∗ can be obtained by minimizing and maximizing λ over Sλ(π), respectively, i.e.,
by solving
λℓ = minλ,x,y,z
{λ : Ax − λ△b = b, xB ≥ 0, xN∪T = 0, (4.4.6)
AT y + z − Qx − λ△c = c, zN ≥ 0, zB∪T = 0 },
and
λu = maxλ,x,y,z
{λ : Ax − λ△b = b, xB ≥ 0, xN∪T = 0, (4.4.7)
AT y + z − Qx − λ△c = c, zN ≥ 0, zB∪T = 0 }.
Proof. We will prove the theorem for λu only. The proof for λℓ goes analo-
gously. Problem (4.4.7) is feasible since problems (QPλ) and (QDλ) are feasible
for the given λ = λ∗. We continue by considering two cases:
(i) Problem (4.4.7) is unbounded. Then for every λ ≥ λ∗ there exists a
feasible solution (x, y, z) for (QPλ) and (QDλ). Further, from the complemen-
tarity property xT z = 0 we conclude that (x, y, z) is also optimal for (QPλ) and
(QDλ). Theorem 4.4.1 imply that λ and λ∗ belong to the same interval Λ(π).
Since this holds for any λ ≥ λ∗, the right boundary of Λ(π) is +∞.
(ii) Problem (4.4.7) has an optimal solution (λ, x, y, z). Similarly to (i),
(x, y, z) is feasible for (QPλ) and (QDλ). From the complementarity property
xT z = 0 we conclude that (x, y, z) is optimal for (QPλ) and (QDλ), and then
Theorem 4.4.1 imply that λ and λ∗ belong to the interval Λ(π) and so λ ≤ λu.
Since for every λ ∈ Λ(π) problem (4.4.7) has a feasible solution and for any
λ ≥ λu the optimal partition is different, the proof is completed. ¤
The open interval Λ(π) is referred to as invariancy interval because the
optimal partition is invariant on it. The points λℓ and λu, that separate neigh-
boring invariancy intervals, are called transition points (see Figure 4.1). The
following theorem shows that the optimal value function is quadratic on an in-
variancy interval (λℓ, λu), where λℓ and λu are obtained by solving (4.4.6) and
70
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Figure 4.1: The Invariancy Intervals and Transition Points
(4.4.7). It presents an explicit formula for the optimal value function as well
as simple criteria to determine convexity, concavity or linearity of the optimal
value function on a specific invariancy interval. Figure 4.2 provides graphical
interpretation of Theorem 4.4.5.
Theorem 4.4.5 Let λℓ and λu be obtained by solving (4.4.6) and (4.4.7), re-
spectively. The optimal value function φ(λ) is quadratic on O(π) = (λℓ, λu).
Proof. If λℓ = λu the statement is trivial, so we may assume that λℓ < λu.
Let λℓ < λ1 < λ < λ2 < λu are given and let (x(1), y(1), z(1)), (x(2), y(2), z(2)) and
(x(λ), y(λ), z(λ)) be pairs of primal-dual optimal solutions corresponding to λ1,
λ2 and λ, respectively. Thus, there is a θ = λ−λ1
λ2−λ1∈ (0, 1) such that
λ = λ1 + θ△λ,
x(λ) = x(1) + θ△x = x(1) +λ − λ1
λ2 − λ1
△x,
y(λ) = y(1) + θ△y = y(1) +λ − λ1
λ2 − λ1
△y,
z(λ) = z(1) + θ△z = z(1) +λ − λ1
λ2 − λ1
△z,
where △λ = λ2 − λ1,△x = x(2) − x(1),△y = y(2) − y(1) and △z = z(2) − z(1). We
also have
A△x = △λ△b, (4.4.8)
AT△y + △z − Q△x = △λ△c. (4.4.9)
71
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
The optimal value function at λ is given by
φ(λ) = (b + λ△b)T y(λ) − 1
2x(λ)T Qx(λ)
= (b + (λ1 + θ△λ)△b)T (y(1) + θ△y) − 1
2(x(1) + θ△x)T Q(x(1) + θ△x)
= (b + λ1△b)T y(1) + θ(△λ△bT y(1) + (b + λ1△b)T△y) + θ2△λ△bT△y
− 1
2x(1)T
Qx(1) − θx(1)T
Q△x − 1
2θ2△xT Q△x. (4.4.10)
From equations (4.4.8) and (4.4.9), one gets
△xT Q△x = △λ(△bT△y −△cT△x), (4.4.11)
x(1)T
Q△x = (b + λ1△b)T△y −△λ△cT x(1). (4.4.12)
Substituting (4.4.11) and (4.4.12) into (4.4.10) we obtain
φ(λ) = φ(λ1 + θ△λ) = φ(λ1) + θ△λ(△bT y(1) + △cT x(1))
+1
2θ2△λ(△cT△x + △bT△y). (4.4.13)
Using the notation
γ1 = △bT y(1) + △cT x(1), (4.4.14)
γ2 = △bT y(2) + △cT x(2), (4.4.15)
γ =γ2 − γ1
λ2 − λ1
=△cT△x + △bT△y
λ2 − λ1
, (4.4.16)
one can rewrite (4.4.13) as
φ(λ) = (φ(λ1) − λ1γ1 +1
2λ2
1γ) + (γ1 − λ1γ)λ +1
2γλ2. (4.4.17)
Since λ1 and λ2 are two arbitrary elements from the interval (λℓ, λu), the claim
of the theorem follows directly from (4.4.17). ¤
It should be mentioned that the sign of △cT△x + △bT△y in (4.4.13) is
independent of λ1 and λ2, because both λ1 and λ2 are two arbitrary numbers in
(λℓ, λu). The following corollary is a straightforward consequence of (4.4.17).
72
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Figure 4.2: The Optimal Value Function on Invariancy Interval
Corollary 4.4.6 For two arbitrary λ1 < λ2 ∈ (λℓ, λu), let (x(1), y(1), z(1)) and
(x(2), y(2), z(2)) be pairs of primal-dual optimal solutions corresponding to λ1 and
λ2, respectively. Moreover, let △x = x(2) − x(1) and △y = y(2) − y(1). Then, the
optimal value function φ(λ) is quadratic on O(π) = (λℓ, λu) and it is
(i) strictly convex if △cT△x + △bT△y > 0;
(ii) linear if △cT△x + △bT△y = 0;
(iii) strictly concave if △cT△x + △bT△y < 0.
Remark 4.4.7 For △c = 0 equation (4.4.17) reduces to the one presented in
Theorem 3.5 in Berkelaar et al. [3].
Remark 4.4.8 Note that π represents either an optimal partition at a transition
point, when λℓ = λu, or on the interval between two consequent transition points
λℓ and λu. Thus Λ =⋃
π Λ(π) =⋃
π Λ(π), where π runs throughout all possible
partitions.
Corollary 4.4.9 The optimal value function φ(λ) is continuous and piecewise
quadratic on Λ.
Proof. The fact that the optimal value function is piecewise quadratic follows
directly from Theorem 4.4.5. We need only to prove the continuity of φ(λ).
73
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Continuity at interior points of any invariancy interval follows from (4.4.17). Let
λ∗ be the left transition point of the given invariancy interval (i.e., λ∗ = λℓ) and
(x(λ∗), y(λ∗), z(λ∗)) be a pair of primal-dual optimal solutions at λ∗. We need
to prove that limλ↓λ∗
φ(λ) = φ(λ∗). For any λ close enough to λ∗, there is a λ such
that for λ∗ < λ < λ with π(λ) = π(λ). Let (x(λ), y(λ), z(λ)) be a maximally
complementary optimal solution at λ and θ = λ∗−λ
λ∗−λ∈ (0, 1). We define
x(λ) = θx(λ) + (1 − θ)x(λ∗),
y(λ) = θy(λ) + (1 − θ)y(λ∗),
z(λ) = θz(λ) + (1 − θ)z(λ∗),
that shows the convergence of subsequence (x(λ), y(λ), z(λ)) to (x(λ∗), y(λ∗),
z(λ∗)) when θ goes to zero. As in our case λ∗ = λℓ, thus it follows from (4.4.6)
that x(λ)T z(λ) = θ(1 − θ)(x(λ)T z(λ∗) + z(λ)T x(λ∗)) = 0. It means that the
subsequence (x(λ), y(λ), z(λ)) is complementary.
We know that x(λ) = x(λ∗) + θ(x(λ) − x(λ∗)) and
φ(λ) = (c + λ△c)T x(λ) +1
2x(λ)T Qx(λ) = (c + λ△c)Tx(λ∗) +
1
2x(λ∗)T Qx(λ∗)
+ θ(c + λ△c)T (x(λ) − x(λ∗)) + θ(x(λ) − x(λ∗))T Qx(λ∗)
+1
2θ2(x(λ) − x(λ∗))T Q(x(λ) − x(λ∗)), (4.4.18)
with θ = λ∗−λ
λ∗−λ∈ (0, 1). When λ ↓ λ∗ (i.e., θ ↓ 0), we have φ(λ) → φ(λ∗) that
proves left continuity of the optimal value function φ(λ) at λ∗ = λℓ. Analogously,
one can prove right continuity of the optimal value function considering λ∗ = λu
by using problem (4.4.7) that completes the proof. ¤
4.4.2 Derivatives, Invariancy Intervals, and Transition
Points
In this subsection, the first and second order derivatives of the optimal value
function are studied. We also investigate the relationship between the invariancy
intervals and neighboring transition points where these derivatives may not exist.
74
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Two auxiliary LO problems were presented in Theorem 4.4.4 to identify
transition points and consequently to determine invariancy intervals. The fol-
lowing theorem allows us to compute the left and right first order derivatives of
the optimal value function by solving another two auxiliary LO problems.
Theorem 4.4.10 For a given λ ∈ Λ, let (x∗, y∗, z∗) be a pair of primal-dual
optimal solutions of (QPλ) and (QDλ). Then, the left and right derivatives of
the optimal value function φ(λ) at λ satisfy
φ′−(λ) = min
x,y,z{△bT y : (x, y, z) ∈ QD∗
λ} + maxx
{△cT x : x ∈ QP∗λ}, (4.4.19)
φ′+(λ) = max
x,y,z{△bT y : (x, y, z) ∈ QD∗
λ} + minx
{△cT x : x ∈ QP∗λ}. (4.4.20)
Proof. Let ε be a sufficiently small real number. For any optimal solution
x(λ) ∈ QP∗λ and (x(λ), y(λ), z(λ)) ∈ QD∗
λ we have
φ(λ + ε) = (b + (λ + ε)△b)T y(λ + ε) − 1
2x(λ + ε)T Qx(λ + ε)
= (b + λ△b)T y(λ) − 1
2x(λ)T Qx(λ) + ε(△bT y(λ) + △cT x(λ + ε))
+ x(λ + ε)T s(λ) +1
2(x(λ + ε) − x(λ))T Q(x(λ + ε) − x(λ))
≥ (b + λ△b)T y(λ) − 1
2x(λ)T Qx(λ) + ε(△bT y(λ) + △cT x(λ + ε))
= φ(λ) + ε(△bT y(λ) + △cT x(λ + ε)), (4.4.21)
where the constraints of (QDλ) and (QPλ+ε) were used. Analogously, the con-
straints of (QPλ) and (QDλ+ε) imply
φ(λ + ε) ≤ φ(λ) + ε(△bT y(λ + ε) + △cT x(λ)). (4.4.22)
We prove only (4.4.20). The proof of (4.4.19) goes analogously. Using (4.4.21)
and (4.4.22) for a positive ε we derive the following inequality
△bT y(λ)+△cTx(λ+ ε) ≤ φ(λ + ε) − φ(λ)
ε≤ △bT y(λ+ ε)+△cTx(λ). (4.4.23)
75
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
For any ε small enough, there is a λ such that λ < λ + ε < λ and λ + ε and λ
are in the same invariancy interval. Let (x(λ), y(λ), z(λ)) be an optimal solution
at λ and for θ = λ−λ
λ+ε−λ> 1, we define
x(θ) = θx(λ + ε) + (1 − θ)x(λ),
y(θ) = θy(λ + ε) + (1 − θ)y(λ),
z(θ) = θz(λ + ε) + (1 − θ)z(λ),
which shows that when ε ↓ 0 (θ ↓ 1) the subsequence (x(λ+ε), y(λ+ε), z(λ+ε))
converges to (x(1), y(1), z(1)) = (x, y, z). Since λ + ε and λ are in the same
invariancy interval, thus xT z = θ(1 − θ)(x(λ + ε)T z(λ) + z(λ + ε)T x(λ)) = 0
shows that x is a primal optimal solution of (QPλ) and (x, y, z) is a dual optimal
solution of (QDλ). Letting ε ↓ 0 we get
limε↓0
△cT x(λ + ε) = △cT x and limε↓0
△bT y(λ + ε) = △bT y. (4.4.24)
From (4.4.23) and (4.4.24) one can easily obtain the inequality
△bT y(λ) + minx(λ)∈QP∗
λ
△cT x(λ) ≤ φ′+(λ) ≤ max
y(λ)∈QD∗
λ
△bT y(λ) + △cT x(λ). (4.4.25)
Since x(λ) is any optimal solution of (QPλ) and (x(λ), y(λ), z(λ)) is any optimal
solution of (QDλ), from (4.4.25) we conclude that
maxy(λ)∈QD∗
λ
△bT y(λ) + minx(λ)∈QP∗
λ
△cT x(λ) ≤ φ′+(λ)
and
φ′+(λ) ≤ max
y(λ)∈QD∗
λ
△bT y(λ) + minx(λ)∈QP∗
λ
△cT x(λ).
Then
φ′+(λ) = max
y(λ)∈QD∗
λ
△bT y(λ) + minx(λ)∈QP∗
λ
△cT x(λ),
completing the proof. ¤
76
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Remark 4.4.11 It is worthwhile to make some remarks about Theorem 4.4.10.
We consider problem (4.4.20) only. Similar results hold for problem (4.4.19).
Let
QPD = {(x, y, z) : Ax = b + λ△b, x ≥ 0, xT z∗ = 0,
AT y + z − Qx = c + λ△c, z ≥ 0, zT x∗ = 0}.
First, in the definition of the set QPD the constraints x ≥ 0, xT z∗ = 0 and
z ≥ 0, zT x∗ = 0 are equivalent to xB ≥ 0, xN∪T = 0 and zN ≥ 0, zB∪T = 0,
where (B,N , T ) is the optimal partition in the transition point λ. Second, let
us consider the first and the second subproblems of (4.4.20). Observe that the
optimal solutions produced by each subproblem are both optimal for (QPλ) and
(QDλ) and so the vector Qx, appearing in the constraints, is always identical
for both subproblems (see, e.g., Dorn [7]). This means that we can maximize the
first subproblem over the dual optimal set QD∗λ only and minimize the second
subproblem over the primal optimal set QP∗λ only. In other words, instead of
solving two subproblems in (4.4.20) separately, we can solve the problem
minx,y,z
{△cT x −△bT y : (x, y, z) ∈ QPD} (4.4.26)
that produces the same optimal solution (x, y, z) as problem (4.4.20). There-
fore, the right derivative φ′+(λ) can be computed by using the values (x, y, z)
as φ′+(λ) = △bT y + △cT x. Consequently, we refer to the optimal solutions of
problems (4.4.20) and (4.4.26) interchangeably.
The following theorem shows that if λ is not a transition point, then the
optimal value function is differentiable and the derivative can be given explicitly.
Theorem 4.4.12 If λ is not a transition point, then the optimal value function
at λ is a differentiable quadratic function and its first order derivative is
φ′(λ) = △bT y(λ) + △cT x(λ).
77
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Proof. The result can be established directly by differentiating the optimal
value function given by (4.4.17). ¤
If we have non-degenerate primal and dual optimal solutions at transition
point λ, the left and right first order derivatives coincide and can be computed by
using the formula given in Theorem 4.4.12 as well. It follows from the fact that
in such situation, the optimal solution of both the primal and the dual problems
(QPλ) and (QDλ) are unique.
The next lemma shows an important property of strictly complementary
solutions of (4.4.19) and (4.4.20) and will be used later on in this chapter.
Lemma 4.4.13 Let λ∗ be a transition point of the optimal value function. Fur-
ther, assume that the (open) invariancy interval to the right of λ∗ contains λ
with the optimal partition π = (B,N , T ). Let (x, y, z) be an optimal solution of
(4.4.20) with λ = λ∗. Then, σ(x) ⊆ B and σ(z) ⊆ N .
Proof. Let (x, y, z) be a maximally complementary solution at λ and let
(λ∗, x, y, z) be an optimal solution of (4.4.6) where the optimal partition is π.
First, we want to prove that
△cT x = △cT x and △bT y = △bT y, (4.4.27)
cT x = cT x and bT y = bT y. (4.4.28)
For this purpose we use equation (4.4.13). In (4.4.13) let λ2 = λ, x(2) = x,
y(2) = y. The continuity of the optimal value function, proved in Corollary 4.4.9,
allows us to establish that equation (4.4.13) holds not only on invariancy in-
tervals, but also at their endpoints, i.e., at the transition points. Thus, we are
allowed to consider the case when λ1 = λ∗ and (x(1), y(1), z(1)) is any optimal
solution at the transition point λ∗.
78
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Computing φ(λ) at the point λ (where θ = λ−λ1
λ2−λ1= λ−λ∗
λ−λ∗= 1) by (4.4.13)
gives us
φ(λ) = φ(λ∗) + (λ − λ∗)(△bT y(1) + △cT x(1))
+1
2(λ − λ∗)[△cT (x − x(1)) + △bT (y − y(1))] (4.4.29)
= φ(λ∗) +1
2(λ − λ∗)[△cT (x + x(1)) + △bT (y + y(1))].
One can rearrange (4.4.29) as
φ(λ) − φ(λ∗)
λ − λ∗= △cT
(
x + x(1)
2
)
+ △bT
(
y + y(1)
2
)
.
Let λ ↓ λ∗, then we have
φ′+(λ∗) = lim
λ↓λ∗
φ(λ) − φ(λ∗)
λ − λ∗= △cT
(
x + x(1)
2
)
+ △bT
(
y + y(1)
2
)
. (4.4.30)
Since (x(1), y(1), z(1)) is an arbitrary optimal solution at λ∗ and φ′+(λ∗) is in-
dependent of the optimal solution choice at λ∗, one may choose (x(1), y(1), z(1)) =
(x, y, z) and (x(1), y(1), z(1)) = (x, y, z). From (4.4.30) we get
φ′+(λ∗) = △cT
(
x + x
2
)
+ △bT
(
y + y
2
)
= △cT
(
x + x
2
)
+ △bT
(
y + y
2
)
. (4.4.31)
Equation (4.4.31) reduces to △cT (x+x
2) = △cT x from which it follows that
△cT x = △cT x. Furthermore, let us consider (x(1), y(1), z(1)) = (x, y, z) and
(x(1), y(1), z(1)) = (x, y, z). From (4.4.30) we obtain △bT y = △bT y.
Now, since both (x, y, z) and (x, y, z) are optimal solutions in QP∗λ∗×QD∗
λ∗ ,
it holds that (c + λ∗△c)T x = (c + λ∗△c)T x and (b + λ∗△b)T y = (b + λ∗△b)T y
(see e.g., Dorn [7]). Consequently, it follows from (4.4.27) that cT x = cT x and
bT y = bT y.
79
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
As a result we can establish that
xT z = xT (c + λ△c + Qx − AT y) = cT x + λ△cT x + xT Qx − (b + λ∗△b)T y
= cT x + λ△cT x + xT Qx − (Ax)T y = xT (c + λ△c + Qx − AT y)
= xT z = 0, (4.4.32)
and
xT z = xT (c + λ∗△c + Qx − AT y) = xT (c + λ∗△c + Qx) − bT y − λ△bT y
= xT (c + λ∗△c + Qx) − bT y − λ△bT y = xT (c + λ∗△c + Qx − AT y)
= xT z = 0. (4.4.33)
For θ ∈ (0, 1) and λ = (1 − θ)λ∗ + θλ, let us consider
x = (1 − θ)x + θx,
y = (1 − θ)y + θy, (4.4.34)
z = (1 − θ)z + θz.
Utilizing equations (4.4.34) and the complementarity properties (4.4.32) and
(4.4.33), we obtain that x and (x, y, z) are feasible and complementary, and
thus optimal solutions of (QPλ) and (QDλ), respectively. Noting that (B,N , T )
is the optimal partition at (x, y, z), it follows from (4.4.34) that xB ≥ 0, xN = 0,
xT = 0 and zB = 0, zN ≥ 0, zT = 0. Then we can conclude that σ(x) ⊆ B and
σ(z) ⊆ N . ¤
Utilizing two auxiliary linear optimization problems we can also calculate
the left and right second order derivatives of φ(λ) [13].
The following theorem, that summarizes the results we got up to now, is
a direct consequence of Theorem 4.4.1 (equivalence of (i) and (ii)), the defi-
nition of a transition point (equivalence of (ii) and (iii)), and Corollary 4.4.3
80
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Figure 4.3: Neighboring Invariancy Intervals
and Lemma 4.4.13 (equivalence of (iii) and (iv)). The proof is identical to the
proof of Theorem 3.10 in Berkelaar et al. [3] and it also shows that in adjacent
subintervals φ(λ) is defined by different quadratic functions.
Theorem 4.4.14 The following statements are equivalent:
(i) Dπ = ∅;(ii) Λ(π) = {λ∗};(iii) λ∗ is a transition point;
(iv) φ′ or φ′′ is discontinuous at λ∗.
By solving an auxiliary self-dual quadratic optimization problem one can obtain
the optimal partition in the neighboring invariancy interval. The result is given
by the next theorem. Figure 4.3 provides graphical interpretation of this result.
Theorem 4.4.15 Let λ∗ be a transition point of the optimal value function.
Let (x∗, z∗) be an optimal solution of (4.4.20) for λ∗. Let us assume that the
(open) invariancy interval to the right of λ∗ contains λ with optimal partition
π = (B,N , T ), and (x, y, z) is a maximally complementary solution at λ. Define
T = σ(x∗, z∗) = {1, 2, . . . , n} \ (σ(x∗) ∪ σ(z∗)). Consider the following self-dual
quadratic problem
minξ,ρ,η
{−△bT η + △cT ξ + ξT Qξ : Aξ = △b, AT η + ρ − Qξ = △c,
ξσ(z∗) = 0, ρσ(x∗) = 0, ξσ(x∗,z∗) ≥ 0, ρσ(x∗,z∗) ≥ 0},(4.4.35)
81
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
and let (ξ∗, η∗, ρ∗) be a maximally complementary solution of (4.4.35). Then,
B = σ(x∗) ∪ σ(ξ∗), N = σ(z∗) ∪ σ(ρ∗) and T = {1, . . . , n} \ (B ∪N ).
Proof. For any feasible solution of (4.4.35) we have
−△bT η + △cT ξ + ξT Qξ = ξT (Qξ − AT η + △c) = ξT ρ = ξTT ρT ≥ 0.
The dual of (4.4.35) is
maxδ,ξ,γ,ζ
{ △bT δ −△cT ζ − ξT Qξ : Aζ = △b, AT δ + γ + Qζ − 2Qξ = △c,
γσ(z∗) = 0, ζσ(x∗) = 0, γT ≥ 0, ζT ≥ 0}.
For a feasible solution it holds
△bT δ −△cT ζ − ξT Qξ = ξT Aζ −△cT ζ − ξT Qξ = −ζT γ − (ζ − ξ)T Q(ζ − ξ) ≤ 0.
So, the optimal value of (4.4.35) is zero. Let us assign
ξ = ζ = x − x∗, ρ = γ = z − z∗, η = δ = y − y∗, (4.4.36)
that satisfy the first two linear constraints of (4.4.35). Then, problem (4.4.35) is
feasible and self-dual.
Using the fact that by Lemma 4.4.13 σ(x∗) ⊆ B and σ(z∗) ⊆ N , it follows
that
ξσ(z∗) = xσ(z∗) − x∗σ(z∗) = 0, ξT = xT − x∗
T = xT ≥ 0
and
ρσ(x∗) = zσ(x∗) − z∗σ(x∗) = 0, ρT = zT − z∗T = zT ≥ 0.
From the proof of Lemma 4.4.13 we have xT z∗ = zT x∗ = 0, implying that (4.4.36)
is an optimal solution. The fact that (x, y, z) is maximally complementary shows
that (4.4.36) is maximally complementary solution in (4.4.35) as well. For x =
x∗ + ξ, we need to consider four cases to determine B:
82
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
1. x∗i > 0 and ξi > 0;
2. x∗i > 0 and ξi = 0;
3. x∗i > 0 and |ξi| < x∗
i ;
4. x∗i = 0 and ξi > 0.
One can easily check that B = σ(x∗) ∪ σ(ξ∗). Analogous arguments hold
for N , that completes the proof. ¤
4.4.3 Computational Algorithm
In this subsection we summarize the results in a computational algorithm. This
algorithm is capable of finding the transition points; the right first order deriva-
tives of the optimal value function at transition points; and optimal partitions at
all transition points and invariancy intervals. Note that the algorithm computes
all these quantities to the right from the given initial value λ∗. One can easily
outline an analogous algorithm for the transition points to the left from λ∗. It
is worthwhile to mention that all the subproblems used in this algorithm can be
solved in polynomial time by IPMs.
Algorithm: Transition Points, First-Order Derivatives
of the Optimal Value Function and Optimal Partitions
at All Subintervals for Convex QO
input:
a nonzero direction of perturbation: r = (△b,△c);
a maximally complementary solution (x∗, y∗, z∗) of (QPλ) and (QDλ) for λ∗;
π0=(B0,N 0, T 0), where B0=σ(x∗), N 0=σ(z∗), T 0={1, . . . , n}\ (B0∪N 0);
k := 0; x0 := x∗; y0 := y∗; z0 := z∗;
83
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
ready:= false;
begin
while not ready do
begin
solve λk = maxλ,x,y,z{λ : Ax − λ△b = b, xBk ≥ 0, xN k∪T k = 0,
AT y + z−Qx−λ△c = c, zN k ≥ 0, zBk∪T k = 0};if this problem is unbounded, ready:=true
else let (λk, xk, yk, zk) be an optimal solution;
begin
let x∗ := xk and z∗ := zk;
solve minx,y,z{△cT x −△bT y : (x, y, z) ∈ QPDk}if this problem is unbounded, ready:= true;
else let (xk, yk, zk) be an optimal solution;
begin
let x∗ := xk and z∗ := zk;
solve minξ,ρ,η{−△bT η + △cT ξ + ξT Qξ : Aξ = △b,
AT η + ρ − Qξ = △c, ξσ(z∗) = 0,
ρσ(x∗) = 0, ξσ(x∗,z∗) ≥ 0, ρσ(x∗,z∗) ≥ 0};Bk+1 = σ(x∗) ∪ σ(ξ∗), N k+1 = σ(z∗) ∪ σ(ρ∗),
T k+1 = {1, . . . , n} \ (Bk+1 ∪N k+1);
k := k + 1;
end
end
end
end
84
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
4.5 Simultaneous Perturbation in Linear
Optimization
The case, when perturbation occurs in the objective function vector c or the right-
hand side vector b of an LO problem was extensively studied. A comprehensive
survey can be found in the book of Roos, Terlaky and Vial [26]. Greenberg [10]
has investigated simultaneous perturbation of the objective and right-hand side
vectors when the primal and dual LO problems are formulated in canonical form.
He proved some properties of the optimal value function in that case and showed
that the optimal value function is piecewise quadratic.
Theorems 4.5.1–4.5.3 resemble Greenberg’s findings, but they are presented
in the manner more suitable for implementation.
We start this section by emphasizing the differences in the optimal parti-
tions of the optimal value function in LO and QO problems and then proceed to
specialize our results to the LO case. Let us define the simultaneous perturbation
of a LO problem as
(LPλ) min { (c + λ△c)T x : Ax = b + λ△b, x ≥ 0 } .
Its dual is
(LDλ) max { (b + λ△b)T y : AT y + z = c + λ△b, z ≥ 0 } .
The LO problem can be derived from the convex QO problem by substitut-
ing zero matrix for Q. As a result, vector x does not appear in the constraints
of the dual problem, and the set T in the optimal partition is always empty.
The following theorem shows that to identify an invariancy interval, we
don’t need to solve problems (4.4.6) and (4.4.7) as they are formulated for the
85
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
QO case. Its proof is based on the fact that the constraints in these problems
separate when Q = 0 and is omitted.
Theorem 4.5.1 Let λ∗ ∈ Λ be given and let (x∗, y∗, z∗) be a strictly com-
plementary optimal solution of (LPλ∗) and (LDλ∗) with the optimal partition
π = (B,N ). Then, the left and right extreme points of the interval Λ(π) = [λℓ, λu]
that contains λ∗ are λℓ = max {λPℓ, λDℓ
} and λu = min {λPu, λDu
}, where
λPℓ= min
λ,x{λ : Ax − λ△b = b, xB ≥ 0, xN = 0},
λPu= max
λ,x{λ : Ax − λ△b = b, xB ≥ 0, xN = 0},
λDℓ= min
λ,y,z{λ : AT y + z − λ△c = c, zN ≥ 0, zB = 0},
λDu= max
λ,y,z{λ : AT y + z − λ△c = c, zN ≥ 0, zB = 0}.
We also state the following lemma that does not hold for QO problems but only
for LO case.
Lemma 4.5.2 Let λℓ and λu be obtained from Theorem 4.5.1 and (x(ℓ), y(ℓ), z(ℓ))
and (x(u), y(u), z(u)) be any strictly complementary solutions of (LPλ) and (LDλ)
corresponding to λℓ and λu, respectively. Then it holds that
△bT△y = △cT△x,
where △y = y(u) − y(ℓ) and △x = x(u) − x(ℓ).
Proof. Subtracting the constraints of (LPλℓ) from (LPλu
) and the constraints
of (LDλℓ) from (LDλu
) results in
A△x = △λ△b, (4.5.1)
AT△y + △z = △λ△c, (4.5.2)
86
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
where △λ = λu − λℓ and △z = z(u) − z(ℓ). Premultiplying (4.5.1) by △yT and
(4.5.2) by △xT , the result follows from the fact that △xT△z = 0, completing
the proof. ¤
Utilizing Lemma 4.5.2 and using the same notation as in (4.4.14)–(4.4.16),
we can state the following theorem that gives explicit expressions for computing
the objective value function. The theorem also gives the criteria to determine
convexity, concavity and linearity of the objective value function on its subinter-
vals.
Theorem 4.5.3 Let λ1 < λ2 and π(λ1) = π(λ2) = π, let (x(1), y(1), z(1)) and
(x(2), y(2), z(2)) be strictly complementary optimal solutions of problems (LPλ)
and (LDλ) at λ1 and λ2, respectively. The following statements hold:
(i) The optimal partition is invariant on (λ1, λ2).
(ii) The optimal value function is quadratic on this interval and is given by
φ(λ) = (φ(λ1) − λ1γ1 +1
2λ2
1γ) + (γ1 − λ1γ)λ +1
2γλ2
= φ(λ1) + θ△λ(△bT y(1) + △cT x(1)) + θ2△λ△cT△x
= φ(λ1) + θ△λ(△bT y(1) + △cT x(1)) + θ2△λ△bT△y
(iii) On any subinterval, the objective value function is
• strictly convex if △cT△x = △bT△y > 0,
• linear if △cT△x = △bT△y = 0,
• strictly concave if △cT△x = △bT△y < 0.
Computation of derivatives can be done by solving smaller LO problems
than the problems introduced in Theorem 4.4.10. The following theorem sum-
marizes these results.
87
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Theorem 4.5.4 For a given λ ∈ Λ, let (x∗, y∗, z∗) be a pair of primal-dual opti-
mal solutions of (LPλ) and (LDλ). Then, the left and right first order derivatives
of the optimal value function φ(λ) at λ are
φ′−(λ) = min
y,z{△bT y : AT y + z = c + λ△c, z ≥ 0, zT x∗ = 0}
+ maxx
{△cT x : Ax = b + λ△b, x ≥ 0, xT z∗ = 0},
φ′+(λ) = max
y,z{△bT y : AT y + z = c + λ△c, z ≥ 0, zT x∗ = 0}
+ minx
{△cT x : Ax = b + λ△b, x ≥ 0, xT z∗ = 0}.
88
Chapter 5
Implementation of Parametric
Quadratic Optimization
Implementation details of the parametric LO/QO package McParam are de-
scribed in this chapter. Section 5.1 illustrates the desired output of the paramet-
ric solver on a simple parametric QO problem. Implementation of the algorithm
and optimal partition determination are subjects of Secton 5.2. The structure of
the McParam package is considered in Section 5.3. Sections 5.4 and 5.5 contain
computational results and their analysis.
5.1 Illustrative Example
Here we present an illustrative numerical example that shows the desired output
of a parametric solver based on the algorithm outlined in Section 4.4.3. Com-
putations related to finding optimal solutions of auxiliary subproblems can be
performed by using any IPM solver for LO and convex QO problems. Let us
consider the following convex QO problem with x, c ∈ R5, b ∈ R
3, Q ∈ R5×5
being a positive semidefinite symmetric matrix, A ∈ R3×5 with rank(A) = 3.
89
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Table 5.1: Transition Points, Invariancy Intervals, and Optimal Partitions forthe Illustrative Problem
Inv. Intervals and Tr. Points B N T φ(λ)λ = −8.0 {3,5} {1,4} {2}
−8.0 < λ < −5.0 {2,3,5} {1,4} ∅ 68.0λ + 8.5λ2
λ = −5.0 {2} {1,3,4,5} ∅−5.0 < λ < 0.0 {1,2} {3,4,5} ∅ −50.0 + 35.5λ + 4λ2
λ = 0.0 {1,2} ∅ {3,4,5}0.0 < λ < 1.739 {1,2,3,4,5} ∅ ∅ −50.0 + 35.5λ − 6.9λ2
λ = 1.739 {2,3,4,5} ∅ {1}1.739 < λ < 3.333 {2,3,4,5} {1} ∅ −40.0 + 24.0λ − 3.6λ2
λ = 3.333 {3,4,5} {1} {2}3.333 < λ < +∞ {3,4,5} {1,2} ∅ 0
The problem data are
Q =
4 2 0 0 02 5 0 0 00 0 0 0 00 0 0 0 00 0 0 0 0
, c =
−16−20
000
, △c =
76000
,
A =
2 2 1 0 02 1 0 1 02 5 0 0 1
, b =
118
20
, △b =
111
.
With this data the perturbed convex QO problem (QPλ) is
min (−16 + 7λ)x1 + (−20 + 6λ)x2 + 2x21 + 2x1x2 + 5
2x2
2
s.t. 2x1 + 2x2 + x3 = 11 + λ2x1 + x2 + x4 = 8 + λ2x1 + 5x2 + x5 = 20 + λx1, x2, x3, x4, x5 ≥ 0.
(5.1.1)
The computational results we are interested in are presented in Table 5.1.
The set Λ for the optimal value function φ(λ) is [−8, +∞). Figure 5.1 depicts the
graph of φ(λ). Transition points and the optimal partitions at each transition
point and on the invariancy intervals are identified by solving the problems in
90
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
−8 −6 −4 −2 0 2 4 6−140
−120
−100
−80
−60
−40
−20
0
20
λ
φ(λ)
Figure 5.1: The Optimal Value Function for Illustrative Problem
Theorem 4.4.4, Remark 4.4.11 and Theorem 4.4.15 according to the algorithm
from Section 4.4.3. The optimal value function on the invariancy intervals is
computed by using formula (4.4.17). Convexity, concavity or linearity of the
optimal value function can be determined by the sign of the quadratic term of
the optimal value function (see Table 5.1). As shown in Figure 5.1, the optimal
value function is convex on the first two invariancy intervals, concave on the
third and fourth and linear on the last one. The first order derivative does not
exists at transition point λ = −5.
91
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
5.2 Implementing the Parametric Algorithm
for QO
Theoretical developments presented in Chapter 4 allowed us to develop an ef-
ficient algorithm for solving parametric QO problems. As we have seen, the
algorithm has polynomial time complexity as all the subproblems can be effi-
ciently solved with an IPM.
5.2.1 Implementation Algorithm
In this section one can find the algorithm which represents an extended version of
that in Section 4.4.3 but adapted for implementation purposes. This algorithm
defines the structure of the McParam package.
Implementation Algorithm for Parametric QO
input:
a quadratic problem (QP ) in the standard form: A, Q, c and b;
a nonzero direction of perturbation: r = (△b,△c);
initial value λ0 for the parameter λ (optional);
begin
solve the parametric problem for λ = λ0 to find a maximally complementary
solution (x0, y0, z0) of (QPλ) and (QDλ);
recover the optimal partition π0 = (B0,N 0, T 0) at λ = λ0, where B0 = σ(x0),
N 0 = σ(z0) and T 0 = {1, . . . , n} \ (B0 ∪N 0);
Solve parametric QO for λ > λ0:
k := 0;
stop:= false;
while not stop do
begin
92
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
solve problem (PQO1):
λk = maxλ,x,y,z{λ : Ax − λ△b = b, xBk ≥ 0, xN k∪T k = 0,
AT y + z − Qx − λ△c = c, zN k ≥ 0, zBk∪T k = 0};if (PQO1) is unbounded then stop:=true
else let (λk, x, y, z) be an optimal solution of (PQO1);
begin
solve problem (PQO2):
minx,y,z{△cT x −△bT y :
Ax = b + λk△b, xσ(x)∪σ(x,z) ≥ 0, xσ(z) = 0,
AT y + z−Qx = c+λk△c, zσ(z)∪σ(x,z) ≥ 0, zσ(x) = 0};if (PQO2) is infeasible then stop:= true;
else let (x∗, y∗, z∗) be an optimal solution of (PQO2);
begin
solve problem (PQO3):
minξ,ρ,η{−△bT η + △cT ξ + ξT Qξ : Aξ = △b,
AT η + ρ − Qξ = △c, ξσ(s∗) = 0,
ρσ(x∗) = 0, ξσ(x∗,z∗) ≥ 0, ρσ(x∗,z∗) ≥ 0};let (ξ∗, ρ∗, η∗) be an optimal solution of (PQO3);
Bk+1 = σ(x∗) ∪ σ(ξ∗), N k+1 = σ(z∗) ∪ σ(ρ∗),
T k+1 = {1, . . . , n} \ (Bk+1 ∪N k+1);
k := k + 1;
end
end
Solve parametric QO for λ < λ0:
k := 0;
stop:= false;
Proceed analogously as for λ > λ0;
end
end
93
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
As the implementation algorithm for the parametric QO problem requires
some more explanations, we provide them in the next paragraphs. McParam
chooses an initial value for λ if it is not specified by a user. Usually λ0 = 0
is good enough as we already know that the non-parametric quadratic problem
(λ = 0) has an optimal solution. In the situations when the quadratic solver
produces imprecise solution at λ0 = 0 and it is difficult to recover the optimal
partition based on this solution, we choose another value of λ0. It is wise to try
values of λ0 being close to zero as the problem can become infeasible if we move
far from it. Such a situation is discussed in Section 5.2.2.
For determining the invariancy intervals and transition points, we first find
the optimal partition π0 = (B0,N 0, T 0) at the initial point. After that, we solve
the parametric problem for the values of λ > λ0 until the parametric problem
becomes unbounded or infeasible. Analogously, we repeat the same procedure
for the values of λ to the left from the initial point λ0.
Problem (PQO1) does not require much explanation. It allows us to get
the next transition point λk and an optimal solution (x, y, z) in that point. We
need this optimal solution to get the support sets σ(x), σ(z) and σ(x, z) =
{1, . . . , n} \ (σ(x) ∪ σ(z)) which are used as inputs for the problem (PQO2).
The derivative subproblem (PQO2) can be more challenging than it seems.
The difficulties here are caused by the fact that we want to solve the derivative
subproblem without knowing the optimal partition at the current transition point
λk. This is actually the reason why we have nonnegativity constraints xσ(x,z) ≥ 0
and zσ(x,z) ≥ 0 in the problem (PQO2). Presence of these constraints reflects
the fact that we do not actually know to which tri-partition Bk, N k or T k the
indices σ(x, z) will belong. It is the consequence of not having a maximally com-
plementary solution at the current transition point λk. This imply that we need
94
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
to somehow enforce the hidden constraint (xσ(x,z))j (zσ(x,z))j = 0 ∀ j ∈ σ(x, z)
for the problem (PQO2). In contrast, if we know maximally complementary so-
lution of the parametric problem for λk and consequently the optimal partition
(Bk, N k, T k) at the transition point λk, the problem (PQO2) becomes:
minx,y,z
△cT x −△bT y
(PQO′2) s.t. Ax = b + λk△b, xBk ≥ 0, xN k∪T k = 0,
AT y + z − Qx = c + λk△c, zN k ≥ 0, zBk∪T k = 0.
Our computational experience shows that if (xσ(x,z))j > 0 and (zσ(x,z))j > 0 for
some j, then B = xσ(x) and N = zσ(z) in that transition point and we just use
this partition while solving (PQO2).
Finally, if we obtain an optimal solution for the problem (PQO2), we can
use the union of its support set with the support set for the problem (PQO3) to
find the optimal partition on the invariancy interval to the right from the current
one. The steps for the points to the left from λ0 are derived analogously.
5.2.2 Determining Optimal Partitions and Support Sets
Determination of the optimal partition for a given maximally complementary
optimal solution or determination of the support set for a given optimal solution
is a challenging task because of numerical reasons. From Chapter 4 we know
that for a given maximally complementary solution (x∗, y∗, z∗):
i ∈ B if z∗i = 0 and x∗i > 0,
i ∈ N if x∗i = 0 and z∗i > 0,
i ∈ T if x∗i = 0 and z∗i = 0.
Unfortunately, numerical solution produced by a LO/QO solver may not
allow to determine the optimal partition or support set in 100% of cases. So, we
introduce a zero tolerance parameter tol zero (the default value is 10−4, which
95
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
−40 −20 0 20 40 60 80 100 120 140 1602.684
2.686
2.688
2.69
2.692
2.694
2.696
2.698
2.7x 10
7
λ
φ(λ)
Figure 5.2: Optimal Partition Determination Counterexample
performs quite well in practice), and compare the entries of the vectors x∗ and
z∗ to it. As a result, we adopt the following strategy for determining the optimal
partition (support set):
if xi ≤ tol zero and zi ≤ tol zero then i ∈ T
elseif xi > tol zero and zi < tol zero then i ∈ B
elseif xi < tol zero and zi > tol zero then i ∈ N
elseif xi ≥ zi then i ∈ B
else i ∈ N
96
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
The methodology described above does not give the desired results in all
cases. Even in the linear case when the partition consists of two sets B and
N only, the task is not easy. In the quadratic case the tri-partition introduces
even more complications as there are 3 sets and the differences between the
entries are smaller. We do not provide any probabilistic analysis of the partition
determination in this thesis, but it can be performed.
Below we show an example of the case when determination of the optimal
partition in a correct way is difficult for a range of starting points. Consider the
problem qscagr7 from the Maros and Meszaros test set and apply the following
perturbation vectors △c = (1, 1, 0, 1,−1, 0, . . . , 0)T and △b = (1,−1, 0, . . . , 0)T .
For that perturbation vectors this problem has one invariancy interval (see Fig-
ure 5.2). Only one coordinate (i = 152) causes problems on the parameter in-
terval λ ∈ (−0.14, 3.98). On this interval the variable x152 becomes numerically
slightly larger than z152 = 0.0016667 and that causes i = 152 to be moved from
the set N to B for the mentioned above interval. This, in turn, causes the op-
timal partition to be determined wrongly and, consequently, leads to computing
incorrect parametric interval, if any.
5.3 Structure of the McParam Package
5.3.1 McParam Arguments and Output
The McParam package for solving parametric quadratic optimization problems
is implemented in Matlab and can be called from it by the following command
[output,exitflag] = mcparam(A,b,db,c,dc,Q,l,u,du,lambda0,options),
where all input arguments except for A, b, c and dc or db are optional. If matrix
Q is not provided, the problem is treated as linear. If no lower bounds l and/or
97
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
upper bounds u are provided, then the lower bound is set to zero for all variables
and the upper bound is set to +∞. You can notice that upper bound u can be also
perturbed by du. In that case we have one more pair of primal and dual variables
s and w and the size of the index set becomes (n + mf ). Consequently, the
partition and support sets are determined based on this enlarged set consisting
of the variables (xT , sT )T and (zT , wT )T . The argument lambda0 is the initial
value for the parameter λ. The description of options parameter, that allows
us to select the LO/QO solver employed for solving auxiliary subproblems and
specify the output file for saving results, is given in Appendix A. The output of
the package contains the indicator exitflag that gives us the solution status of
the parametric problem (0 corresponds to infeasible problem for initial λ and 1 to
successfully terminated algorithm). The structure output contains the following
fields:
output(i).type type of the invariancy interval i (’invariancyinterval’ or ’transition point’);
output(i).B optimal partition set B;output(i).N optimal partition set N ;output(i).T optimal partition set T ;output(i).lambdaleft left end of the invariancy interval i;output(i).lambdaright right end of the invariancy interval i;output(i).lambda value of λ in the transition point i;output(i).obj 3D vector representing the optimal value function
φ(λ) = aλ2 + bλ + c on the invariancy interval i;output(i).lderiv value of the left derivative in the transition point i
(if available);output(i).rderiv value of the right derivative in the transition point i
(if available);output(i).x optimal solution vector x in the transition point i;output(i).y optimal solution vector y in the transition point i;output(i).z optimal solution vector z in the transition point i.
Optimal value function output(i).obj is computed based on the objec-
tive function values at the endpoints of the invariancy interval and left (right)
98
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
derivative in one of the endpoints.
It is worthwhile to make the following notes about the output produced by
the McParam package:
• The problem is preprocessed using the same preprocessing techniques as
described in Section 3.1.2. All the subproblems solved later on are utilizing
the results of preprocessing. The optimal partitions are computed by the
package in terms of the preprocessed problem.
• Note that the optimal partition reported in the transition points may not
be complete. The determination of the optimal partition in a transition
point is based on the union of the support sets obtained from solving prob-
lems (PQO1) and (PQO2), which does not necessarily gives the optimal
partition, but provides its best ”approximation” based on the available
information. This allows avoiding solving an additional parametric QO
problem in the transition point to compute the optimal partition. But
even though such problem is solved, the optimal partition can be deter-
mined only subject to numerical values of the optimal solution and may
not be complete as well.
• Optimal partition reported for parametric LO problems in the transition
points may include set T . As we know for LO problems the optimal par-
tition π = (B,N ) consists of two sets only. So, all the indices appearing
in the set T are the indices which are not determined to be in either B or
N as we do not want to solve the additional optimization problem in the
transition point.
99
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
5.3.2 McParam Flow Chart
Taking into account all above information we can summarize the actions taken
by McParam in the flow chart on Figure 5.3. As we see from Figure 5.3, the
precautions are taken against numerical difficulties which can arise while solving
LO/QO auxiliary subproblems. First, the difficulties can occur while solving
problem (PQO1) using incorrectly determined optimal partition (Section 5.2.2).
In that case we choose a random value of λ close to the initial λ0 and try to resolve
problem (QPλ) in attempt to recover the optimal partition correctly. Second, if
numerical difficulties occur while solving problems (PQO2) or (PQO3) we follow
the same strategy: choose a value of λ close to the current one, for instance,
λk+1 = λk + ǫ (where ǫ > 0 is a small number), if we are moving to the right
from λ0, and solve problem (QPλk+1) to determine the optimal partition in order
to proceed further.
5.4 Computational Results
In this section we show the performance of our McParam package on a set of
parametric convex QO problems. As there are no other solvers that can perform
analogous analysis, we cannot compare our software with another packages. That
is why we only present the computational results produced by McParam on
selected QO/LO problems for the illustration purposes. All computations are
performed on a Windows PC with Pentium IV 3.0 GHz processor and 1Gb RAM.
Variable t in the McParam printouts plays the role of λ, phi(t) gives the the
optimal value function, type is the interval type (invariancy interval or transition
point).
Tables 5.2 and 5.3 contain the McParam output for the perturbed quadratic
100
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Solve (QP ) for
Error computing optimal partition
Choose another
Find optimal partition (Bk,Nk,Tk)
Solve (PQO1)
(PQO1) is primal infeasible
Parametric interval is
unbounded
No
Yes
(PQO1) is dual infeasible
Yes
No (PQO1) is optimal
Solve (PQO2)
Yes
(PQO2) is optimal
Yes
Solve (PQO3)
No(PQO2) is infeasible
End of parametric
interval
Yes No
(PQO3) is optimal
Yes
No(PQO3) is infeasibleYes
No
End of parametric interval. Stop
End of parametric
interval
Numerical difficulties. Stop
Input datak = 0
No
k = k + 1
Recover from numerical difficulties:
, k = k + 1
0l
kll
ell +=+ kk 1
Figure 5.3: Flow Chart of the McParam Package
101
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Table 5.2: McParam Output for Perturbed lotschd Problem (QO Formulation)
type tl tu B N T phi(t)
------------------------------------------------------------------------------------------------
tr. point -19.38488 -19.38488 1 3 5 7 9 11 2 6 8 10 12 4 3076.87621
inv. interv -19.38488 -9.11811 1 3 4 5 7 9 11 2 6 8 10 12 2.5t2+27.9t+2671.6
tr. point -9.11811 -9.11811 1 3 4 5 7 9 11 2 8 10 12 6 2626.26944
inv. interv -9.11811 -7.31937 1 3 4 5 6 7 9 11 2 8 10 12 -3.6t2-84.4t+2159.5
tr. point -7.31937 -7.31937 1 3 5 6 7 9 11 2 8 10 12 4 2582.09702
inv. interv -7.31937 +0.30244 1 3 5 6 7 9 11 2 4 8 10 12 0.8t2-19.1t+2398.4
tr. point +0.30244 +0.30244 1 3 5 6 7 9 11 2 4 8 10 12 2392.71830
inv. interv +0.30244 +11.00000 1 3 5 6 7 9 11 12 2 4 8 10 0.3t2-18.8t+2398.4
tr. point +11.00000 +11.00000 3 5 6 7 9 11 12 1 2 4 8 10 2223.41981
inv. interv +11.00000 +18.06373 2 3 5 6 7 9 11 12 1 4 8 10 1.6t2-11.6t+2157.9
tr. point +18.06373 +18.06373 2 3 5 6 7 9 11 1 4 8 10 12 2469.53014
inv. interv +18.06373 +44.49132 2 3 5 6 7 9 11 1 4 8 10 12 2.9t2-58.2t+2578.8
tr. point +44.49132 +44.49132 2 3 5 7 9 11 1 4 8 10 12 6 5706.58820
Table 5.3: McParam Output for Perturbed lotschd Problem (LO Formulation)
type tl tu B N T phi(t)
------------------------------------------------------------------------------------------------
tr. point -19.38488 -19.38488 1 3 5 7 9 11 2 6 8 10 12 4 75.45527
inv. interv -19.38488 -0.00000 1 3 4 5 7 9 11 2 6 8 10 12 0.3t2+0.9t+0.0
tr. point -0.00000 -0.00000 1 3 4 5 7 9 11 12 2 6 8 10 0.00000
inv. interv -0.00000 +11.00000 1 3 5 6 7 9 11 2 4 8 10 12 -0.9t2-21.3t-0.0
tr. point +11.00000 +11.00000 3 5 6 7 9 11 1 2 4 8 10 12 -343.02008
inv. interv +11.00000 +44.49132 2 3 5 6 7 9 11 1 4 8 10 12 1.0t2-42.2t+0.0
tr. point +44.49132 +44.49132 2 3 5 7 9 11 1 4 8 10 12 6 99.96501
and linear formulations of lotschd problem from the Maros and Meszaros test
set. Optimal value functions graphed by McParam are shown at Fugures 5.4
and 5.5. Problem lotschd is a small-size problem, with 12 variables and 7 con-
straints and that is why McParam prints optimal partition information. For
larger problems, only intervals and corresponding optimal value functions are
shown.
Tables 5.4 and 5.5 provide the printout of McParam output for the per-
turbed quadratic and linear formulations of medium-size qsc205 problem
(317 variables and 205 constarints). Fugures 5.6 and 5.7 graph the optimal
value functions for it.
Small and medium-size problem are handled pretty well by McParam. For
102
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
−20 −10 0 10 20 30 402000
2500
3000
3500
4000
4500
5000
5500
6000
λ
φ(λ)
Figure 5.4: Optimal Value Function for Perturbed lotschd QO Problem
−20 −10 0 10 20 30 40−500
−400
−300
−200
−100
0
100
λ
φ(λ)
Figure 5.5: Optimal Value Function for Perturbed lotschd LO Problem
103
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Table 5.4: McParam Output for Perturbed qsc205 Problem (QO Formulation)
type tl tu phi(t)
--------------------------------------------------------------------
tr. point -200.00000 -200.00000 -0.00000
inv. interval -200.00000 -198.05825 27.0t2+10600.0t+1040000.0
tr. point -198.05825 -198.05825 -286.54916
inv. interval -198.05825 -189.47368 5.95t2+2261.9t+214285.7
tr. point -189.47368 -189.47368 -664.81994
inv. interval -189.47368 -13.50000 -0.02t2-0.0t-0.0
tr. point -13.50000 -13.50000 -3.37500
inv. interval -13.50000 -0.02326 -0.02t2-0.0t-0.01
tr. point -0.02326 -0.02326 -0.00581
inv. interval -0.02326 +0.30814 -0.0t2+0.0t-0.01
tr. point +0.30814 +0.30814 -0.00581
inv. interval +0.30814 +0.49057 -0.1t2+0.1t-0.0
tr. point +0.49057 +0.49057 -0.00926
inv. interval +0.49057 +94.54545 -0.1t2+0.0t-0.0
tr. point +94.54545 +94.54545 -343.80165
inv. interval +94.54545 +100.00000 12.9t2-2444.4t+115555.6
tr. point +100.00000 +100.00000 -0.00000
−200 −150 −100 −50 0 50 100−700
−600
−500
−400
−300
−200
−100
0
λ
φ(λ)
Figure 5.6: Optimal Value Function for Perturbed qsc205 QO Problem
104
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Table 5.5: McParam Output for Perturbed qsc205 Problem (LO Formulation)
type tl tu phi(t)
------------------------------------------------------------
tr. point -200.00000 -200.00000 -0.00000
inv. interval -200.00000 -100.00000 1.0t2+200.0t+0.0
tr. point -100.00000 -100.00000 -10000.00000
inv. interval -100.00000 -1.00000 0.0t2+100.0t-0.0
tr. point -1.00000 -1.00000 -100.00000
inv. interval -1.00000 -0.66667 2.0t2+102.0t-0.0
tr. point -0.66667 -0.66667 -67.11111
inv. interval -0.66667 -0.26282 -1.0t2+28.3t-47.8
tr. point -0.26282 -0.26282 -55.30182
inv. interval -0.26282 -0.25537 -0.7t2+25.3t-48.6
tr. point -0.25537 -0.25537 -55.11111
inv. interval -0.25537 -0.24484 -0.6t2+17.7t-50.5
tr. point -0.24484 -0.24484 -54.92099
inv. interval -0.24484 +0.03057 -0.4t2+11.0t-52.2
tr. point +0.03057 +0.03057 -51.86571
inv. interval +0.03057 +0.50000 1.1t2-31.4t-50.9
tr. point +0.50000 +0.50000 -66.33333
inv. interval +0.50000 +100.00000 1.3t2-133.3t-0.0
tr. point +100.00000 +100.00000 0.00000
−200 −150 −100 −50 0 50 100
−10000
−9000
−8000
−7000
−6000
−5000
−4000
−3000
−2000
−1000
0
λ
φ(λ)
Figure 5.7: Optimal Value Function for Perturbed qsc205 LO Problem
105
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Table 5.6: McParam Output for Perturbed aug3dc Problem (QO Formulation)
type tl tu phi(t)
---------------------------------------------------------
tr. point +0.00000 +0.00000 -1165.23756
inv. interval +0.00000 +1.10980 -0.3t2-0.3t-1165.2
tr. point +1.10980 +1.10980 -1165.91114
inv. interval +1.10980 Inf 0.1t2-1.1t-1164.8
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5−1169
−1168.5
−1168
−1167.5
−1167
−1166.5
−1166
−1165.5
−1165
λ
φ(λ)
Figure 5.8: Optimal Value Function for Perturbed aug3dc QO Problem
106
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
the large-size parametric problems (number of variables n ≥ 1000), numerical
difficulties occur often, especially while solving quadratic self-dual subproblem
(PQO3) due to problems with determination of optimal partitions and support
sets. The example of such case is shown in Table 5.6 and Fugure 5.8. For the
large-size perturbed QO problem aug3dc (7746 variables and 1000 constarints)
the quadratic self-dual subproblem (PQO3) is not solved in the transition point
λ = 1.1098 because of numerical difficulties and so, the strategy for recovering
the optimal partition described in Section 5.3.2 is used.
5.5 Analysis of Results
Summarizing shortly the computational experience of the McParam package the
following conclusions can be drawn.
• Performance of the parametric quadratic solver depends on the number of
variables. For small- and medium-size problems it performs well, but large-
size problems represent significant challenge. It mostly happens because
of numerical troubles occurring when solving auxiliary subproblems or due
to difficulties with determination of the optimal partition or support sets.
From our computational experience, it especially applies to the self-dual
quadratic problem (PQO3). Note that problem (PQO3) seems to be twice
as large as the original QO problem, but because of its self-duality property
it contains only two more rows than the original problem.
• Robustness of the McParam package is an important issue. It was designed
to handle unexpected situations. The package tries to find the invariancy
intervals, optimal value function, etc. on the whole set Λ. McParam makes
attempts to recover when some of the auxiliary subproblems are not solved
107
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
by a QO/LO solver. McParam mostly fails only if QO/LO solver fails.
• Difficulties in determining the optimal partition can be overcome by the
strategy described in Section 5.3. In addition, we can use other strategies
for determining the optimal partition, such as doing heuristic analysis of
the problematic indices or solving the parametric problem of the reduced
size by eliminating primal and dual variables, which are determined to be
in either set B or N .
• Both QO/LO solvers, MOSEK and McIPM, currently used for solving aux-
iliary subproblems, perform comparably well. Still, MOSEK seems to pro-
vide better solution precision for some problems and is faster on many
problems.
108
Chapter 6
Conclusions and Further Work
In this thesis we considered techniques for solving quadratic optimization prob-
lems and their parametric counterparts. We modified the linear optimization
software package McIPM to be able to solve quadratic problems. Consequently,
McIPM became the tool for solving auxiliary problems in our parametric algo-
rithm, but any other IPMs solvers can be used for that purpose as well. Testing
results show that our implementation is competitive with non-commercial and
even commercial quadratic optimization software.
We extended the existing theory of parametric quadratic optimization to
simultaneous perturbation sensitivity analysis when the variation occurs in both
the right-hand side vector of the constraints and the coefficient vector of the
objective function linear term. In our analysis the rate of variation, represented
by the parameter λ, was identical for both perturbation vectors △b and △c.
One of the main results is that the invariancy intervals and transition points can
be determined by solving auxiliary linear or quadratic problems. This means
that we should not worry about ”missing” short-length invariancy intervals. As
we already mentioned, all auxiliary problems can be solved in polynomial time.
Finally, we developed and implemented the algorithm that represents a sequence
109
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
of linear and quadratic auxiliary problems to identify all invariancy intervals and
graph the optimal value function.
The implementation of the algorithm for parametric optimization and test-
ing results suggest that our implementation is well suitable for solving small- and
medium-size parametric quadratic problems and can be used for large-scale prob-
lems as well with some caution connected to the precision of solutions produced
by a quadratic solver. As none of the existing software packages can perform
parametric analysis for quadratic optimization, we illustrated our implementa-
tion on our own set of parametric quadratic problems.
As it was mentioned in the introduction, the most famous application of
the convex QO sensitivity analysis is the mean-variance portfolio optimization
problem introduced by Markowitz [17]. Our method allows us to analyze not
only the original Markowitz model, but also some of its extensions. One possible
extension is when the investors’ risk aversion parameter λ influences not only
risk-return preferences, but also budget constraints. One more practical problem,
where our methodology can be used, is coming from engineering. It consists in
optimizing power distribution between users of digital subscriber lines (DSL).
The variation of λ represents variation of noisiness.
There are many possible extensions and future work in both theoretical
and implementation directions. I just want to mention some of them.
(i) As we know the number of transition points for the parametric quadratic
problem is finite, but can grow exponentially in the number of variables.
The finite number of transition points is determined by the finite number
of partitions of the index set. It would be interesting to establish an upper
bound on the number of transition points which can possibly depend on
the problem dimension in primal and dual spaces.
110
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
(ii) There are some results recently developed for parametric Convex Conic Op-
timization, where the variables are restricted to belong to cones. Yildirim
[34] extended the concept of the optimal partition to conic problems. Con-
sequently, one of the further research directions can include generalizing
the analysis of this thesis to Second-Order Cone Optimization (SOCO) and
Semidefinite Optimization (SDO) problems. The first priority here would
be to generalize the methodology and develop an algorithm for simultane-
ous perturbation sensitivity analysis for Second-Order Cone Optimization
problems. It is also worthwhile exploring parametric SOCO applications
to financial models.
(iii) All the analysis in this thesis is done for single parameter perturbation. It
may be also interesting to look at the problem (QPλb,λc) where the param-
eter λ is different for the objective function and for the constraints. This
case can be generalized further to multiparametric quadratic optimization.
(iv) A number of studies exist for multiparametric quadratic optimization where
λ is a vector (see, e.g., [28]). Multiparametric QO is widely applied in
optimal control. No doubt that the analysis is becoming more complicated
when we work in multidimensional parameter space λ. We would like to
look at the possibility of extending our results and the algorithm to n-
dimensional parametric optimization.
From the practical or implementation side we want to suggest the following
future directions:
(iv) Make the implementation of McParam more robust and improve the solu-
tion precision of the linear system in McIPM using iterative refinement.
(v) Experiment with warm-start strategy.
111
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
(vi) Extend the implementation to SOCO case. We already have a SOCO
package (McSOCO) developed in the Advanced Optimization Laboratory
by B. Wang [30] that can become the base solver for such implementation.
112
Appendix A
McIPM and McParam Options
This appendix contains the description of the command-line options of the
McIPM and McParam packages. Options are specified according to Matlab
conventions. Furthermore, McIPM output parameter exitflag values are also
described.
113
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
McIPM Options
Display Level of display. ’off’ displays no output; ’iter’ (de-fault) displays output at each iteration; ’final’ displaysthe final output only.
Diagnostics Print diagnostic information such as MPS/QPS readerstatistics, preprocessing information, ordering time, etc.:’No’ or ’Yes’ (default).
MaxIter Maximum number of iterations allowed (default is 150).
McParam Options
Solver LO/QO solver to be used for solving auxiliary subprob-lems. Currently two solvers are available, ’MOSEK’ and’McIPM’, the first one is default.
Outfile Filename for saving computational results.
McIPM exitflag Values
2 NONFIRM OPTIMAL Non-firm optimal solution found.1 FIRM OPTIMAL Optimal solution found.0 MAXITER UNKNOWN Maximum number of iterations ex-
ceeded or status unknown.-1 PRIMAL INFEAS Primal infeasibility detected.-2 DUAL INFEAS Dual infeasibility detected.-3 PRIMDUAL INFEAS Primal-dual infeasibility detected.-4 NONFIRM PRIMAL INFEAS Non-firm primal infeasibility detected.-5 NONFIRM DUAL INFEAS Non-firm dual infeasibility detected.-6 NONFIRM PRIMDUAL INFEAS Non-firm primal-dual infeasibility de-
tected.
114
Appendix B
Maros and Meszaros Test Set
This appendix contains the description of the Maros and Meszaros [18] test set
of convex quadratic optimization problems. The set currently contains 138 prob-
lems. In the tables the following notation is used:
Name – problem name;m – number of rows in matrix A;n – number of variables;NZ A – number of nonzeros in A;NZ Q diag – number of diagonal entries in Q;NZ Q off-diag – number of off-diagonal entries in the lower
triangular part of Q;Prepr. Rows – number of rows in matrix A after McIPM
preprocessing;Prepr. Cols – number of variables after McIPM preprocessing.
115
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Table B.1: The Maros and Meszaros QO Test Set (I)
Problem Statistics McIPM Preprocessing
Name m n NZ A NZ Q diag NZ Q off-diag Prepr. Rows Prepr. Colsaug2d 10000 20200 40000 19800 0 10000 40400aug2dc 10000 20200 40000 20200 0 10000 40400aug2dcqp 10000 20200 40000 20200 0 10000 20200aug2dqp 10000 20200 40000 19800 0 10000 20200aug3d 1000 3873 6546 2673 0 1000 7746aug3dc 1000 3873 6546 3873 0 1000 7746aug3dcqp 1000 3873 6546 3873 0 1000 3873aug3dqp 1000 3873 6546 2673 0 1000 3873boyd1 18 93261 558985 93261 0 18 93261boyd2 186531 279785 423784 2 0 186531 279785cont-050 2401 2597 12005 2597 0 2401 2597cont-100 9801 10197 49005 10197 0 9801 10197cont-101 10098 10197 49599 2700 0 10098 10197cont-200 39601 40397 198005 40397 0 39601 40397cont-201 40198 40397 199199 10400 0 40198 40397cont-300 90298 90597 448799 23100 0 90298 90597cvxqp1 l 5000 10000 14998 10000 29984 5000 10000cvxqp1 m 500 1000 1498 1000 2984 500 1000cvxqp1 s 50 100 148 100 286 50 100cvxqp2 l 2500 10000 7499 10000 29984 2500 10000cvxqp2 m 250 1000 749 1000 2984 250 1000cvxqp2 s 25 100 74 100 286 25 100cvxqp3 l 7500 10000 22497 10000 29984 7500 10000cvxqp3 m 750 1000 2247 1000 2984 750 1000cvxqp3 s 75 100 222 100 286 750 100dpklo1 77 133 1575 77 0 77 266dtoc3 9998 14999 34993 14997 0 9998 29994dual1 1 85 85 85 3473 1 85dual2 1 96 96 96 4412 1 96dual3 1 111 111 111 5997 1 111dual4 1 75 75 75 2724 1 75dualc1 215 9 1935 9 36 215 223dualc2 229 7 1603 7 21 229 235dualc5 278 8 2224 8 28 278 285dualc8 503 8 4024 8 28 503 510exdata 3001 3000 7500 1500 1124250 3001 6000genhs28 8 10 24 10 9 8 20gouldqp2 349 699 1047 349 348 349 699gouldqp3 349 699 1047 698 697 349 699hs118 17 15 39 15 0 17 32hs21 1 2 2 2 0 1 3hs268 5 5 25 5 10 5 15hs35 1 3 3 3 2 1 4hs35mod 1 3 3 3 2 1 3hs51 3 5 7 5 2 3 10hs52 3 5 7 5 2 3 10
116
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Table B.2: The Maros and Meszaros QO Test Set (II)
Problem Statistics McIPM Preprocessing
Name m n NZ A NZ Q diag NZ Q off-diag Prepr. Rows Prepr. Colshs53 3 5 7 5 2 3 5hs76 3 4 10 4 2 3 7hues-mod 2 10000 19899 10000 0 2 10000huestis 2 10000 19899 10000 0 2 10000ksip 1001 20 18411 20 0 1001 1041laser 1000 1002 3000 1002 3000 1000 3004liswet1 10000 10002 30000 10002 0 10000 30004liswet10 10000 10002 30000 10002 0 10000 30004liswet11 10000 10002 30000 10002 0 10000 30004liswet12 10000 10002 30000 10002 0 10000 30004liswet2 10000 10002 30000 10002 0 10000 30004liswet3 10000 10002 30000 10002 0 10000 30004liswet4 10000 10002 30000 10002 0 10000 30004liswet5 10000 10002 30000 10002 0 10000 30004liswet6 10000 10002 30000 10002 0 10000 30004liswet7 10000 10002 30000 10002 0 10000 30004liswet8 10000 10002 30000 10002 0 10000 30004liswet9 10000 10002 30000 10002 0 10000 30004lotschd 7 12 54 6 0 7 12mosarqp1 700 2500 3422 2500 45 700 1457mosarqp2 600 900 2930 900 45 600 1233powel20 10000 10000 20000 10000 0 10000 30000primal1 85 325 5815 324 0 85 734primal2 96 649 8042 648 0 96 1393primal3 111 745 21547 744 0 111 1600primal4 75 1489 16031 1488 0 75 3052primalc1 9 230 2070 229 0 9 254primalc2 7 231 1617 230 0 7 240primalc5 8 287 2296 286 0 8 304primalc8 8 520 4160 519 0 8 545q25fv47 820 1571 10400 446 59053 798 1854qadlittl 56 97 383 17 70 55 137qafiro 27 32 83 3 3 27 51qbandm 305 472 2494 25 16 269 436qbeaconf 173 262 3375 18 9 148 270qbore3d 233 315 1429 28 50 199 300qbrandy 220 249 2148 16 49 149 259qcapri 271 353 1767 56 838 267 476qe226 223 282 2578 67 897 220 469qetamacr 400 688 2409 378 4069 357 692qfffff80 524 854 6227 278 1638 501 1005qforplan 161 421 4563 36 546 135 463qgfrdxpn 616 1092 2377 54 108 600 1144qgrow15 300 645 5620 38 462 300 645qgrow22 440 946 8252 65 787 440 946qgrow7 140 301 2612 30 327 140 301
117
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
Table B.3: The Maros and Meszaros QO Test Set (III)
Problem Statistics McIPM Preprocessing
Name m n NZ A NZ Q diag NZ Q off-diag Prepr. Rows Prepr. Colsqisrael 174 142 2269 42 656 174 316qpcblend 74 83 491 83 0 74 114qpcboei1 351 384 3485 384 0 347 722qpcboei2 166 143 1196 143 0 140 279qpcstair 356 467 3856 467 0 356 538qpilotno 975 2172 13057 94 391 951 2242qptest 2 2 4 2 1 2 4qrecipe 91 180 663 20 30 85 177qsc205 205 203 551 11 10 205 317qscagr25 471 500 1554 28 100 471 671qscagr7 129 140 420 8 17 129 185qscfxm1 330 457 2589 56 677 322 592qscfxm2 660 914 5183 74 1057 644 1184qscfxm3 990 1371 7777 89 1132 966 1776qscorpio 388 358 1426 22 18 388 358qscrs8 490 1169 3182 33 88 485 1270qscsd1 77 760 2388 54 691 77 760qscsd6 147 1350 4316 96 1308 147 1350qscsd8 397 2750 8584 140 2370 397 2750qsctap1 300 480 1692 36 117 300 660qsctap2 1090 1880 6714 141 636 1090 2500qsctap3 1480 2480 8874 186 861 1480 3340qseba 515 1028 4352 96 550 515 1036qshare1b 117 225 1151 18 21 112 248qshare2b 96 79 694 10 45 96 162qshell 536 1775 3556 405 34385 496 1487qship04l 402 2118 6332 14 42 356 2162qship04s 402 1458 4352 14 42 268 1414qship08l 778 4283 12802 940 34025 688 4339qship08s 778 2387 7114 538 11139 416 2171qship12l 1151 5427 16170 2023 60205 838 5329qship12s 1151 2763 8178 1042 16361 466 2293qsierra 1227 2036 7302 122 61 1222 2715qstair 356 467 3856 66 952 356 538qstandat 359 1075 3031 138 666 359 1258s268 5 5 25 5 10 5 15stadat1 3999 2001 9997 2000 0 3999 8001stadat2 3999 2001 9997 2000 0 3999 8001stadat3 7999 4001 19997 4000 0 7999 16001stcqp1 2052 4097 13338 4097 22506 770 1368stcqp2 2052 4097 13338 4097 22506 1881 2644tame 1 2 2 2 1 1 2ubh1 12000 18009 48000 6003 0 12000 29991values 1 202 202 202 3620 1 202yao 2000 2002 6000 2002 0 2000 5999zecevic2 2 2 4 1 0 2 4
118
Bibliography
[1] I. Adler and R. Monteiro. A geometric view of parametric linear program-
ming. Algorithmica, 8:161–176, 1992.
[2] E. D. Anderson and Y. Ye. On a homogeneous algorithm for the monotone
complementarity problem. Mathematical Programming, 84:375–399, 1999.
[3] A. B. Berkelaar, B. Jansen, C. Roos, and T. Terlaky. An interior point
approach to parametric convex quadratic programming. Working paper,
Erasmus University Rotterdam, Rotterdam, The Netherlands, 1997.
[4] A. B. Berkelaar, C. Roos, and T. Terlaky. The optimal set and opti-
mal partition approach to linear and quadratic programming. Chapter 6 in
Advances in Sensitivity Analysis and Parametric Programming, T. Gal and
H. J. Greenberg, eds., Kluwer Academic Publishers, Boston, USA, 1997.
[5] J. C. G. Boot. Quadratic Programming: Algorithms, Anomalies and Appli-
cations. North-Holland Publishing Company, Amsterdam, The Netherlands,
1964.
[6] T. Davis. Algorithm 8xx: a concise sparse Cholesky factorization packages.
Submitted to ACM Trans. on Math. Software, 2004.
119
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
[7] W. S. Dorn. Duality in quadratic programming. Quarterly of Applied Math-
ematics, 18:155–162, 1960.
[8] D. E. Farrar. The Investment Decision Under Uncertainty. Prentice-Hall,
New York, 1965.
[9] Financial Glossary ; http://www.investorwords.com, accessed 26 July
2004.
[10] H. J. Greenberg. Simultaneous primal-dual right-hand-side sensitivity
analysis from a strictly complementary solution of a linear program. SIAM
Journal of Optimization, 10:427–442, 2000.
[11] H. J. Greenberg. Mathematical Programming Glossary, 1996-2004, s.v. ”Sen-
sitivity analysis”; http://carbon.cudenver.edu/~hgreenbe/glossary/,
accessed 26 July 2004.
[12] O. Guler and Y. Ye. Convergence behavior of interior-point algorithms.
Mathematical Programming, 60:215–228, 1993.
[13] A. Ghaffari Hadigheh, O. Romanko, and T. Terlaky. Sensitivity Analysis
in Convex Quadratic Optimization: Simultaneous Perturbation of the Ob-
jective and Right-Hand-Side Vectors. Technical Report #2003/6, Advanced
Optimization Laboratory, McMaster University, Hamilton, Canada, 2003.
Submitted to Optimization.
[14] N. Gould and P. Toint. Preprocessing for quadratic programming. Mathe-
matical Programming, 100:95–132, 2004.
120
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
[15] X. Huang. Preprocessing and Postprocessing in LO. M. Sc. Thesis, Advanced
Optimization Laboratory, Department of Computing and Software,
McMaster University, Hamilton, Canada, 2004.
[16] B. Jansen, J. J. de Jong, C. Roos, and T. Terlaky. Sensitivity analysis
in linear programming: just be careful! European Journal of Operational
Research, 101:15–28, 1997.
[17] H. M. Markowitz. The optimization of a quadratic function subject to linear
constraints. Naval Research Logistics Quarterly, 3:111–133, 1956.
[18] I. Maros and C. Meszaros. A repository of convex quadratic optimiza-
tion problems. Departmental Technical Report DOC 97/6, Department of
Computing, Imperial College, London, UK, 1997.
[19] L. McLinden. An analogue of Moreau’s proximation theorem, with appli-
cation to the nonlinear complementary problem. Pacific Journal of Mathe-
matics, 88:101–161, 1980.
[20] S. Mehrotra. On the implementation of a primal-dual interior point method,
SIAM Journal on Optimization, 2:575-601, 1992.
[21] C. Meszaros. The separable and non-separable formulations of con-
vex quadratic problems in interior-point methods. Working paper 98-3,
Hungarian Academy of Sciences, Budapest, Hungary, 1998.
[22] C. Meszaros. The BPMPD interior point solver for convex quadratic prob-
lems. Working paper 98-8, Hungarian Academy of Sciences, Budapest,
Hungary, 1998.
121
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
[23] B. Murtagh. Advanced Linear Programming: Computation and Practice.
McGraw-Hill, New York, USA, 1981.
[24] K. G. Murty. Linear Programming. John Wiley & Sons, New York, USA,
1983.
[25] J. Peng, C. Roos, and T. Terlaky. Self-Regularity: A New Paradigm for
Primal-Dual Interior-Point Methods. Princeton University Press, 2002.
[26] C. Roos, T. Terlaky, and J.-Ph. Vial. Theory and Algorithms for Linear
Optimization: An Interior Point Approach. John Wiley & Sons, Chichester,
UK, 1997.
[27] T. Seol. Solving Linear Equations in Interior-Point Methods. Intermedi-
ate Technical Report, Advanced Optimization Laboratory, Department of
Computing and Software, McMaster University, Hamilton, Canada, 2004.
http://www.cas.mcmaster.ca/~oplab/seminar/students/slides/seol.pdf
[28] P. Tondel, T. A. Johansen, and A. Bemporad. Further results on multipara-
metric quadratic programming. Proc. 42th IEEE Conf. on Decision and
Control, Maui, Hawaii, USA, 3173-3178, 2003.
[29] R. J. Vanderbei. Symmetric quasi-definite matrices. SIAM Journal on
Optimization, 5(1):100113, 1995.
[30] B. Wang. Implementation of Interior-Point Methods for Second-Order
Conic Optimization. M. Sc. Thesis, Advanced Optimization Laboratory,
Department of Computing and Software, McMaster University, Hamilton,
Canada, 2003.
122
M.Sc. Thesis - Oleksandr Romanko McMaster - Computing and Software
[31] N. Yamashita and Z.-Q. Luo. A Nonlinear Complementarity Approach
to Multiuser Power Control for Digital Subscriber Lines. To appear in
Optimization Methods and Software, 2004.
[32] Y. Ye, M. J. Todd, and S. Mizuno. An O(√
nL) iteration homogeneous
and self-dual linear programming algorithm. Mathematics of Operations
Research, 19:52–67, 1994.
[33] Y. Ye. Interior Point Algorithms, Theory and Analysis. John Wiley,
Chichester, UK, 1997.
[34] E. A. Yildirim. A unifying optimal partition approach to sensitivity analysis
in conic optimization. To appear in Journal of Optimization Theory and
Applications, 2004.
[35] X. Zhu. Implementing the New Self-Regular Proximity Based IPMs. M. Sc.
Thesis, Advanced Optimization Laboratory, Department of Computing and
Software, McMaster University, Hamilton, Canada, 2002.
[36] X. Zhu, J. Peng, T. Terlaky, and G. Zhang. On Implementing Self-Regular
Proximity Based Feasible IPMs. Technical Report #2004/3, Advanced
Optimization Laboratory, McMaster University, Hamilton, Canada, 2004.
Submitted to INFORMS Journal on Computing.
123