wind farm layout optimization using approximate inference in graphical models · 2018-04-24 ·...
TRANSCRIPT
WIND FARM LAYOUT OPTIMIZATION USING
APPROXIMATE INFERENCE IN GRAPHICAL MODELS
by
Aditya Dhoot
A thesis submitted in conformity with the requirements for the degree of Master of Applied Science
Graduate Department of Mechanical and Industrial Engineering University of Toronto
© Copyright 2016 by Aditya Dhoot
ii
Wind farm layout optimization using approximate inference in
graphical models
Aditya Dhoot
Master of Applied Science
Graduate Department of Mechanical and Industrial Engineering
University of Toronto
2016
Abstract
Wind farm layout optimization (WFLO) determines the optimal location of wind turbines
within a fixed geographical area to maximize the total power capacity of the wind farm,
under stochastic wind conditions and non-linear aerodynamic interferences between
turbines. This thesis develops optimization approaches to fast approximate (sub-optimal)
turbine layouts to aide engineers make design decisions. Building on previous work in
discrete quadratic WFLO models, we recast the program as a probabilistic graphical
model incorporating spatial dependencies (i.e., aerodynamic interferences, proximity
constraints, and maximum number of turbines) between the variables. Turbine layouts
are estimated using message passing inference (BP, TRW-S), which exploit the
problem’s graph-theoretic structure using decomposition and factorization. We perform
an exhaustive computational study comparing TRW-S with branch-and-cut algorithms
under varying wind-regime complexity and problem resolutions. We demonstrate the
broad applicability of techniques we develop by solving a suite of benchmark quadratic
knapsack problems, a general class of problems that arise in many settings.
iii
Acknowledgements
I sincerely thank Professor Cristina Amon for believing in my academic abilities and giving me an
opportunity to pursue my research interests. Her support and guidance have helped me to grow
as a researcher, and have made the last two years academically and personally enriching. I am
particularly grateful to her for being incredibly supportive of my decisions when I chose to defer
my graduate studies to pursue entrepreneurial projects.
I owe a great debt of gratitude to Dr. David Romero for being a supportive research advisor.
His open-mindedness and patience towards exploring new ideas and concepts encouraged me
to think outside the box and find inspiration when it is least expected. I’ve become a better
researcher and engineer because of David.
I am thankful to Professor Daniel Frances and Professor Roy Kwon for serving on my thesis
committee. Their insights into optimization theory and engineering practices helped me improve
this thesis. I would also like to acknowledge Professor Brendan Frey, his graduate class and
research partially formed the inspiration to conduct this research.
I am grateful to my undergraduate thesis advisor, Professor Timothy Chan for giving me the
opportunity to explore the world of operations research. I really enjoyed his LP class, and his clear
and organized teaching helped me understand and conceptualize ideas in the formative stages
of my research.
I am thankful to the many friends I’ve made while sharing an often overcrowded office space
– Juan Stockle, Enrico Antonini, Jim Kuo, Carlos Da Silva, Sami Sorkhabi, Matthew Doyle, David
Guirguis, Armin Taheri, Fernan Saiz, Francisco Contreras, and Sean Crawford. In particular, I am
thankful to Jim for helping me navigate through the many research ideas. I would also like
acknowledge Peter Zhang for introducing me to the research group and taking the time to explain
his work.
I am grateful to my parents for their support and encouragement, and to whom this thesis is
dedicated to. I would also like to acknowledge my sister for her help and from whom I’ve learned
the art of rigorously applying logical fallacies to get your way. And now, last but not least, I would
like to thank my furry friend, Frodo for always sticking by my side through thick and thin. His
determination and dedication to fetching the ball has always been a source of inspiration.
iv
Contents
1 Introduction ................................................................................................................ 1
1.1 Motivation ............................................................................................................. 1
1.2 Contribution .......................................................................................................... 1
1.3 Organization ......................................................................................................... 3
2 Literature Review ....................................................................................................... 4
2.1 Probabilistic Graphical Models ............................................................................. 5
2.2 Literature Review ................................................................................................. 6
2.2.1 Evolutionary Algorithms................................................................................ 7
2.2.2 Mathematical Programs ............................................................................... 8
2.2.3 Message-Passing Algorithms ..................................................................... 11
2.2.4 Loopy Belief Propagation ........................................................................... 12
2.2.5 Max-Product Linear Programming ............................................................. 13
2.2.6 Tree-Reweighted Message-Passing .......................................................... 14
2.2.7 Tightening LP Relaxations using Message-Passing Algorithms ................. 15
2.3 Discussion .......................................................................................................... 16
3 Quadratic Knapsack Problem ................................................................................. 18
3.1 Introduction ........................................................................................................ 19
3.1.1 Quadratic Knapsack Problem ..................................................................... 19
3.2 Related Work ..................................................................................................... 20
3.2.1 Message Passing Algorithms ..................................................................... 21
3.3 Method ............................................................................................................... 23
3.3.1 Penalty Method .......................................................................................... 23
3.4 Experiments and Discussion .............................................................................. 24
3.4.1 Benchmark Experiments ............................................................................ 25
3.5 Conclusion ......................................................................................................... 27
4 Wind Farm Layout Optimization ............................................................................. 28
4.1 Introduction ........................................................................................................ 29
4.2 Literature Review ............................................................................................... 30
v
4.3 Background ........................................................................................................ 32
4.3.1 Wake Modelling .......................................................................................... 32
4.3.2 Wind Farm Layout Optimization Mathematical Program ............................ 33
4.3.3 WFLO as a Markov Random Field ............................................................. 35
4.3.4 Message-Passing Algorithms ..................................................................... 38
4.4 Experimental Results and Discussion ................................................................ 40
4.4.1 Case 1 – 100 cells ...................................................................................... 42
4.4.2 Case 2 – 400 cells ...................................................................................... 47
4.4.3 Case 3 – 2,500 cells ................................................................................... 50
4.4.4 Discreteness Analysis ................................................................................ 53
4.5 Conclusions........................................................................................................ 55
5 Concluding Remarks ............................................................................................... 58
5.1 Conclusions ........................................................................................................ 58
5.2 Future work ........................................................................................................ 58
5.2.1 Wind Farm Layout Optimization ................................................................. 59
5.2.2 Message passing algorithms for inference and optimization ...................... 60
Bibliography ................................................................................................................ 62
1
Chapter 1
Introduction
1.1 Motivation
Increasing levels of greenhouse emissions threaten to irreversibly damage the climate
and eco-system. There is an immediate need to produce sustainable energy to meet the
world’s growing energy demand. In 2014, a total of $270 billion was invested in renewable
energy technologies that resulted in an addition of 128GW power capacity, of which 37%
was due to new wind farm installations [1]. Therefore, capturing wind energy will play a
major role in our shift to sustainable energy generation. A common problem encountered
while designing a wind farm is to determine the optimal placement of wind turbines within
a fixed geographical area to maximize the total power capacity of the wind farm under
stochastic wind conditions and non-linear aerodynamic interferences between the
turbines. This is known as the wind farm layout optimization (WFLO) problem. In this
thesis, we investigate mathematical models and optimization techniques that can fast
approximate turbine layouts to aide engineers to quickly assess and plan infrastructure
design decisions. While the principal goal of this thesis is to build optimization methods
for the WFLO problem, we demonstrate that the methods and techniques we develop are
widely applicable to optimal resource allocation problems constrained with a limited
budget capacity that are encountered in many fields, such as computer vision, natural
language processing, computational biology, telecommunications, and finance, among
others.
1.2 Contribution
In this thesis, we discretize the wind farm and use a quadratic formulation of the WFLO
problem developed by Turner et al. [2] to model a probabilistic undirected graphical
model, also known as Markov random fields (MRFs), which are an useful abstraction that
2
succinctly capture the spatial dependencies between the variables such as aerodynamic
interferences caused by wind regimes, and turbine proximity constraint. Furthermore, the
modelled WFLO problem has a single linear inequality constraint (budget constraint) that
restricts the number of turbines that can be placed on the terrain. We use Jensen’s wake
model [3] to determine the pairwise wake interactions between turbines. However, in
contrast with previous works that use Jensen’s wake model, we do not simplify the shape
and distance of the wake interactions.
We demonstrate that the modelled WFLO problem is structurally similar to the
quadratic knapsack problem (QKP), which is a well-studied problem with NP-Hard time
complexity and for which some algorithms that seek an exact solution have been
developed [4]. Rader et al. [5] established that the QKP has graph-theoretic properties
that can be exploited to better understand the availability of polynomial approximability,
or lack thereof under various problem conditions. Using this insight, we review message
passing algorithms, a class of decentralized and asynchronous probabilistic inference and
optimization methods that operate on graphical models by iteratively passing beliefs
locally along its edges to determine the maximum-a-posteriori (optimal) configuration of
the graph’s random variables. Belief propagation (BP) [6], Sequential Tree-Reweighted
Message Passing (TRW-S) [7], and Max-Product Linear Programming (MPLP) [8] are all
types of message-passing algorithms that have been successful in practice at
determining a good configuration (sub-optimal) of dense and large graphs in problems
that arise in computer vision [9] – [10], machine learning [11], and computational biology
[12]. While much remains to be investigated about the algorithms’ theoretical guarantees,
their deep rooted relationship with linear programming, duality theory, and probabilistic
graphical models forms the basis for our motivation to apply these algorithms to the WFLO
problem.
As message passing algorithms typically operate on an unconstrained objective
function, we derive an unconstrained form of QKP using Lagrangian augmentation and
penalty functions to incorporate the budget constraint into the graphical model. As a
number of number of important sub-routines in many fields can be described as a QKP,
we apply BP and TRW-S on benchmark QKP datasets of varying graph density and
3
number of random variables, and then compare the obtained results with the ones from
exact optimization methods (e.g., branch-and-bound, cutting planes, etc.). We also
conduct a thorough computational study to compare the effectiveness of turbine layouts
produced using TRW-S with computationally exhaustive branch-and-cut algorithm under
varying wind-regime complexity and discrete resolution. Additionally, to the best of our
knowledge, we generate results for the WFLO problem at a higher discrete resolution
than any existing works in literature.
As QKPs and the modelled WFLO problem are computationally intractable and in
certain cases in-approximable, we don’t intend to find optimal solutions using BP and
TRW-S. We want to assess the empirical performance of these algorithms to generate
good wind farm layouts in comparison with exact algorithms within a given time budget.
1.3 Organization
In Chapter 2, we review the literature on WFLO models and the variety of optimization
methods used to solve them. We also review probabilistic graphical models, exact
inference, and message passing algorithms such as BP, TRW-S, and MPLP. In Chapter
3, we build an undirected graphical model to represent QKPs and solve them using TRW-
S. In Chapter 4, we perform a thorough computational study comparison between TRW-
S and branch-and-cut methodology to solve the WFLO problem using varying wind-
regime complexity and problem resolution. Finally, in Chapter 5 we provide concluding
remarks and directions for future work.
4
Chapter 2
Background
As wind farm layout optimization (WFLO) is a problem that consists of optimal turbine
placement based on an accurate model of the non-linear aerodynamic interferences
(wake effects) between the turbines, existing work focuses primarily on two key areas –
constructing accurate wake models and improving layout optimization algorithms.
However, sophisticated models that try to accurately portray physical and mechanical
properties make the problem non-convex, non-linear, and computationally intractable. In
this thesis, we follow previous work and use a simple analytical wake model (Jensen’s
wake model [3]) to calculate the non-linear aerodynamic interferences, and work on
developing novel optimization approaches to solve the WFLO problem. In this chapter,
we thoroughly review the various heuristic and exact optimization techniques that have
been used to solve the WFLO problem.
In this thesis, we make an extensive use of message passing algorithms, graphical
models, and probabilistic inference to solve the WFLO problem and quadratic knapsack
problems (QKPs). Therefore, we initially derive the underlying mathematical framework
describing probabilistic graphical models. Subsequently, we compare the various
message passing algorithms by discussing their advancements, benefits, and limitations.
5
2.1 Probabilistic Graphical Models
A graph G = (𝒱, ℰ) is a set of nodes or vertices 𝒱 connected by edges ℰ, where the edges
between the nodes represent an underlying mathematical relationship between the
nodes. A probabilistic graphical model is a graphical representation of a joint probability
distribution of a set of random variables [13]. Therefore, in a probabilistic graphical model,
the nodes represent random variables in the probability distribution and the edges
connecting the nodes capture the dependencies between the random variables.
Probabilistic graphical models can be directed or undirected. Directed graphs are
acyclic and the edges have a directionality that encapsulates further conditional
dependency between the random variables. In undirected graphs or Markov random
fields (MRF), the edges have no directionality and maybe cyclic. MRF can be
decomposed as a product of factors composed of a set of random variables XC over the
set of maximal cliques C of graph G [13].
Given a set of random variables 𝐗 = (X1, X2, … , Xn), the joint distribution of a pairwise
MRF, shown in Eq. (2.1) is proportional to the product of potential functions θC(𝐗𝐂) over
maximal cliques C, where Z is the partition function denoted by Eq. (2.2). The partition
function normalizes the joint distribution.
p(𝐗) =1
Z∏ θC(𝐗𝐂)C (2.1)
Z = ∑ ∏ θC(𝐗𝐂)C𝐗 (2.2)
The positive potential functions θC(𝐗𝐂) encode the problem specific relationship
between the random variables. As many combinatorial optimization problem formulations
can be represented as a MRF, the potential function can be thought of as encoding the
optimization objective function [14]. Furthermore, the potential function is generally
expressed as an exponential function as denoted in Eq. (2.3), where E(𝐗𝐜) represents the
energy function. Thus, random variable configuration with a higher probability have a
lower overall energy [15].
θC(𝐗𝐂) = exp(−E(𝐗𝐜)) (2.3)
6
Furthermore, the product of the exponential functions results in the sum of factors of
energy functions as shown by Eq. (2.4), where φs(Xs) represent all unary potentials over
the model vertices 𝒱 and φst(Xs, Xt) represent the pairwise potentials over the edges ℰ.
E(𝐗) = ∑ φs(Xs)s∈𝒱 + ∑ φst(Xs, Xt)(s,t)∈ℰ (2.4)
Then, the joint distribution of all the random variables is given by Eq. (2.5), which is
generally known as the Gibbs distribution [15].
p(𝐗) =1
Zexp(− ∑ φs(Xs)s∈𝒱 − ∑ φst(Xs, Xt)(s,t)∈ℰ ) (2.5)
The problem of determining the configuration of random variables in a graphical model
that yields the maximum probability over the joint distribution is known as maximum-a-
posteriori (MAP) inference problem as formulated in Eq. (2.6). This is equivalent to
minimizing the energy function as shown in Eq. (2.7).
𝐗∗ = argmax𝐗
p(𝐗) (2.6)
𝐗∗ = argmin𝐗
E(𝐗) = argmin𝐗
(∑ φs(Xs)s∈𝒱 + ∑ φst(Xs, Xt)(s,t)∈ℰ ) (2.7)
In many cases, the decomposable factors within the graphical model can be exploited
to efficiently compute the optimization problem.
We propose to use a MRF based representation to model the relationship between all
pairwise discrete locations on the wind farm by encoding the appropriate wake
interactions for any given wind speed and directions within the pairwise edge potentials.
An optimal wind farm configuration can be obtained by determining the MAP estimation
of the resulting graphical model.
2.2 Literature Review
The overall goal of WFLO research community is to generate turbine layouts that produce
maximum energy, while adhering to several project specific constraints such noise,
7
infrastructure costs, etc. WFLO is a challenging problem because it is not only difficult to
accurately model wind turbine behaviour, but also a difficult discrete optimization task.
Therefore, existing work on wind turbine layout optimization focuses primarily on two key
areas – improving the model accuracy, and improving optimization algorithms.
In order to improve the model accuracy researchers are studying the physical and
mechanical relationships between turbine properties and environmental factors.
Sophisticated models that try to accurately portray physical and mechanical properties
make the problem non-convex, non-linear, and generally intractable. In this review, we
initially focus on the success and limitations of the optimization algorithms that have been
applied to various formulations of the WFLO problem. We then review a class of message
passing algorithms that have been widely applied within the field of computer vision and
computational biology with much success.
2.2.1 Evolutionary Algorithms
Two decades ago when the WFLO problem started to garner the interest of the
engineering research community, evolutionary algorithms were the first set of methods
applied to solve the optimization problem. Evolutionary algorithms are biologically
inspired metaheuristic optimization methods that generate populations of candidate
solutions and evolve them until a convergence criterion is met, in the hope that the final
population of solutions contains the optimal solution to the problem. Depending on the
search space, problem representation, and evolution operators used (e.g., the crossover
strategy used to generate new candidate solutions), evolutionary algorithms can in many
cases find the globally optimal solution for problems that are difficult to solve using other
methods.
In 1994, Mosetti et al. [16] were the first group of researchers to formulate the WFLO
problem and use a genetic search algorithm to find optimal turbine placements on a
simple 10 x 10 square cell grid that maximizes energy, while minimizing installation costs
and wake interactions for single directional and multi-directional wind scenarios. Even
though simple assumptions regarding the turbine type, wake modeling, and terrain data
are made, the promising results encouraged the entire wind engineering community to
8
broadly investigate the wide applicability of using evolutionary algorithms to solve realistic
WFLO problems.
In 2004, Grady et al. [17] also used genetic search algorithm and modeled a similar
objective function that was presented by Mosetti et al. in [16]. Grady et al. obtained better
layouts with higher power output and lower installation costs compared to the results
published in [16] by increasing the simulation population size and number of generations.
This shows that by fine tuning the parameters involved in genetic algorithms such as
crossover type, diversity percentage, and population size the overall results of the
algorithm can be drastically improved.
Huang et al. [18] further enhanced earlier approaches by applying a distributed genetic
algorithm to decompose a large wind farm terrain search space into local search spaces
to maximize annual wind farm profits. It is shown that a distributed approach yields better
solution quality and is computationally faster than traditional genetic algorithms. Huang
et al. also suggests the use of local search techniques to further enhance the distributed
genetic algorithm.
The evolutionary methods described here used to solve the WFLO problem work well
with small-scale non-linear and non-convex problems. These methods, however, do not
provide any optimality guarantees; therefore, solutions generated maybe sub-optimal and
further methods need to be investigated to achieve globally optimal solutions.
2.2.2 Mathematical Programs
By discretizing the wind farm terrain into grid cells and representing each cell as a binary
decision variable, mathematical programming models can be developed for the WFLO
problem. Individual decision variables take on a binary state and determine whether at
every location a turbine is placed or not placed. The relationship between the decision
variables (e.g., wake interactions, infrastructure costs, etc.) can be encoded within the
objective function and hard constraints can be developed for budget, spacing, proximity,
terrain usage, noise, and other infrastructure constraints. Existing research on formulating
the WFLO problem as a mixed integer program (MIP) and quadratic integer program
9
(QIP) has been conducted with promising results. Integer programs can be solved exactly
using well-designed algorithms [19] that are generally implemented in an off-the-shelf
commercial solver such as CPLEX (IBM Corp., Armonk, NY), and have several theoretical
guarantees, which is one of the biggest advantages to model WFLO problem as a
mathematical program. However, in many cases, models need to be simplified to make
the problem tractable and convex.
Donovan [20] [21] used a MIP model to formulate the WFLO problem and applied
branch-and-bound method to solve the problem. Donavan exploits the inherent graph
based structure of the problem to model the problem as a vertex packing problem. Edges
in the graph represent turbine proximity and wake interference relationship between
potential turbine placements. Furthermore, maximal cliques are used limit local placement
of turbines and satisfy turbine proximity constraints. Donovan notes the limitations of the
constraints used within the model, and suggests that a hybrid heuristic algorithm and MIP
can be applied together to handle more realistic non-linear constraints and improve non-
optimal solutions generated by the MIP.
Donovan in [21] explored varying branching strategies to reduce the time taken to
reach optimality. The paper dynamically generates violated constraints as needed to
significantly reduce the time required to find the optimal solution. A brief analysis on the
implications of linear programming (LP) relaxation of the integer constraints is conducted.
Donovan also claims that the branching strategy will garner benefits up to a maximum
placement of 50 turbines beyond which more efficient techniques need to be developed.
Nevertheless, this approach illustrates that good solutions can be generated for simplified
models that can be used along with hybrid heuristic algorithms with non-linear constraints.
Similarly, Archer et al. [22] formulated a WFLO MIP model that minimizes the wake
interactions between turbines. Wake interactions are modelled using the PARK model
[23] and a Weibull distribution is implemented to handle variability in the wind velocity.
The authors were able to place up to 25 wind turbines and the resulting solution quality
outperforms those generated using other heuristic algorithms.
Turner et al. [2] formulate the discrete wind farm layout problem as a quadratic integer
program (QIP). Jensen’s wake model is used to calculate the wake interactions between
10
turbines, and the overall formulation determines turbine placement that minimizes the
overall kinetic energy deficit. Furthermore, the paper also uses Adams-Sherali zero-one
quadratic program linearization technique to formulate a MIP. CPLEX 12.1 (IBM Corp.,
Armonk, NY) was used to find optimal turbine placements for a multi-directional wind
problem on a 10 x 10 (2km by 2km) grid. Solution quality for placing 10 to 40 turbines
was better than solutions from Mosetti et al. [16] and Grady et al [17]. Nevertheless, it
could take up to 24 hours to solve the problem, which is non ideal as a realistic WFLO
problem would consist of further constraints and a higher dimensional objective function.
Thus, there is a need for developing algorithms that quickly produce good solutions
(approximate) for mathematical program formulations of single and multi-objective WFLO
problems.
Recognizing the limitation of existing WFLO MIP models to incorporate realistic
nonlinearities in the problem, in 2014, Zhang et al [24] proposed constraint programming
(CP) and MIP models that takes into account the inherent non-linear aerodynamic
behaviour of the wake interactions among multiple wind turbines. The paper showed that
the MIP based decomposition models outperforms CP models. Additional models with
noise and land-owner participation constraints were also solved. The results from [24] are
promising and further research in mathematical programs for the WFLO problem needs
to be conducted in this direction.
In general, the application of MIP models and constraint programming to WFLO
problem has been recent but very promising. These models offer exact solutions and
optimality guarantees as opposed to evolutionary algorithms. Nevertheless, further
research in this area is required to develop models that consider the full non-linearity of
the wake interactions, stochastic wind conditions, difficult constraints, and are also
capable of encoding multiple objectives, while solving the problems in a reasonable
amount of time. Furthermore, a hybrid approach using mathematical models alongside
with evolutionary algorithms is also an area of research worth exploring.
11
2.2.3 Message-Passing Algorithms
A large class of discrete optimization problems can be posed as a maximum-a-posteriori
(MAP) inference problem in a probabilistic graphical model by recasting the objective cost
function as a graph structured probability distribution known as Markov random fields
(MRFs). Vertex cover (e.g. [25]), facility location (e.g. [26]), quadratic assignment (e.g.
[27]), and resource allocation (e.g. [28]) are some examples of discrete optimization
problems that have been reformulated and solved as MAP inference problems.
A class of algorithms known as message passing algorithms approximate the MAP
assignment by iteratively passing beliefs along the edges of the graphical model in a
distributed, decentralized, and asynchronous manner. Message passing algorithms
exploit the decomposability of the graphical model to find the optimal configuration. MIP,
QIP, and CP WFLO models with a graph-theoretic structure can be posed as a MAP
inference problem. Therefore, it is hypothesized that message passing algorithms can be
used to efficiently approximate layout configurations of discretized WFLO model with a
graph based structure.
Pearl [6] introduced a type of message passing algorithm called the max-product belief
propagation (BP) that is guaranteed to converge to the optimal MAP configuration for
graphical models that contain only trees. BP is not guaranteed to converge or find optimal
configurations for graphical models that include cycles. Nevertheless, empirical evidence
has shown that BP can produce good results even for graphs with cycles in the fields of
computer vision (e.g. [9] and [10]) and computational biology (e.g. [12]). Even though BP
cannot be directly applied to graphical models with cycles, understanding their
convergence properties has helped researchers develop several variants of BP
algorithms that have been empirically successful at finding optimal MAP configurations.
As performing exact inference in graphical models is an NP-Hard problem, Wainwright
et al. [29] argue that fixed points, the stationary points of the energy function Eq. (2.7), of
message passing algorithms do not generally yield globally optimal solutions;
nevertheless, these algorithms in practice yield good sub-optimal MAP configurations.
12
2.2.4 Loopy Belief Propagation
Loopy Belief Propagation (LBP) is simply BP applied over graphical models with cycles
(“loops”) to estimate the MAP configuration. In [30] Sanghavi et al. used Loopy Belief
Propagation to determine the MAP estimation of the weighted matching problems on
general graphs. In general, weighted matching problems can be formulated as integer
programs; however, the distributed and asynchronous characteristics of message
passing algorithms have been used to empirically solve the problem efficiently. The paper
proves that LBP converges to the optimal weighted matching if the linear programming
relaxation is tight, and does not converge when the linear programming relaxation is not
tight for a given weighted matching problem. Thus, any WFLO problem modelled as a
weighted matching problem can be optimally and efficiently solved using LBP if the
resulting LP relaxation of the problem is tight. However, this is generally not the case.
Therefore, further variants of message passing algorithms need to be generated that work
well with cycles.
Additionally, as many real-world objective functions are non-convex it is always ideal
to develop message passing algorithms that are able to solve these problems to
optimality. For instance, Turner et al. [2] formulated the WFLO problem as a QIP model,
which is generally non-convex and thus linear approximations of the problem yield
solutions that are not globally optimal. In [31] Kumar et al. used the concave-convex
procedure (CCCP) with message-passing algorithm to estimate the MAP inference of the
relaxed quadratic integer formulations of pairwise MRF. The CCCP technique
decomposes a non-convex quadratic objective function into a difference of two convex
functions. Using CCCP with message passing algorithms is guaranteed to converge to a
local minima or a global minimum for a non-convex and convex quadratic programs
respectively. This methodology results in richer solution quality than using standard LP
formulations of max-product for Ising graphs and protein design problems. This technique
can be applied to the QIP model generated in [2], however, the relaxed QIP is not always
tight.
Similarly, Ravikumar et al. [32] formulated a relaxed QIP to estimate the MAP
inference. They show that a quadratic program solves the MAP solution exactly as it
13
accurately represents the energies from a pairwise Markov random field compared to a
more commonly used linear formulation of the MAP problem. Furthermore, simple convex
approximations are made to make non-convex quadratic formulations convex. Standard
solvers and iterative projected conjugate gradient methods are used to solve small and
large problems respectively. It is shown that computing a quadratic relaxation program
exactly is sometimes faster and yields better solutions than LP Relaxation, iterative
conditional modes, and tree-reweighted max product algorithms.
Even though traditional integer, quadratic, and linear programming algorithms from
[33] can be applied to solve MAP problems, it is sometimes more efficient to exploit clique
factors within the graphical model to efficiently compute solutions. Furthermore, in many
cases very large graphs are cannot be solved using techniques in [33] in a limited amount
of time. Thus, it is important to know when message-passing algorithms can be applied
to best exploit graphical model structure and solve for its MAP configuration.
Since this thesis will be using message passing to tackle WFLO MIP models, we need
to further investigate message passing algorithms that are able to efficiently compute
integer programs that are not tight when relaxed. Two such variants of belief propagation
that efficiently compute MAP configuration for MRFs with cycles are Max Product Linear
Programming algorithm (MPLP) and Tree-Reweighted Message Passing (TRW).
However, both these methods relax the integer constraints to solve the problem.
Therefore, we also need to investigate methodologies that can be used to dynamically
generate constraints to make the solution space results tight.
2.2.5 Max-Product Linear Programming
Globerson et al. [8] introduced a new type of max-product message passing algorithm
known as Max Product Linear Programming algorithm (MPLP). MPLP uses block
coordinate descent in the dual of the LP Relaxation of the MAP problem to compute
beliefs between nodes. MPLP is guaranteed to converge and computes the exact solution
to the relaxed LP problem for binary variables. MPLP explicitly exploits the graph structure
of graphical models that traditional LP solvers fail to exploit.
14
2.2.6 Tree-Reweighted Message-Passing
In [29], Wainwright et al. show that by decomposing an arbitrary graphical model with
cycles into a convex combination of tree-structured distributions, an optimal upper bound
can be calculated for the MAP estimation problem. The authors show that an optimal MAP
configuration and a tight lower bound can be obtained if and only if the combination of
tree distributions share a common optimum (known as tree agreement). This property is
derived by initially formulating a LP by relaxing the integer polytope into loose tree-based
consistency conditions and then solving its Lagrangian dual. Furthermore, Wainwright et
al. in [29] apply an iterative tree-reweighted message passing algorithm that
reparametrizes the tree-structured distributions to determine fixed points that specify the
solutions to the Lagrangian dual. Lastly, it is shown that when the relaxed integer
constraints yield fractional values, the relaxation can be tightened by forming additional
clusters of random variables within the graphical model.
As demonstrated in [29], TRW works by maximizing the lower bound of the free energy
objective. However, Kolmogorov in [7] shows that TRW does not guarantee to increase
this lower bound, and in certain situations may actually decrease the lower bound value
during the iterative algorithm. Furthermore, TRW is not guaranteed to converge. In [7],
Kolmogorov develops sequential tree-reweighted message passing algorithm (TRW-S)
that guarantees that the lower bound will never decrease.
Messages are also updated sequentially rather than in parallel. Tested on benchmark
computer vision problems, TRW-S yields better results and is memory efficient compared
to TRW and MPLP. However, it is important to note that TRW-S does not work well when
the resulting relaxed LP formulation is not tight.
In [34], Kolmogorov et al. derive key optimality conditions of TRW-S for problems with
binary variables. The derivation decomposes an arbitrary graph into a convex
combination of trees to illustrate that any fixed point (stationary points of the energy
function) that satisfy the weak tree agreement can be used to specify a subset of a
globally optimum solution [34]. Similar to MPLP, it is shown that for problems with binary
variables a weak tree agreement fixed point always determines the optimum of the
relaxed linear program of the MAP estimation problem. Lastly, the properties derived in
15
[34] are similar to the results found in integer programming literature of Rhys relaxation,
roof duality, and weak persistency.
Thus, given MPLP’s and TRW’s optimality guarantees and computation efficiency this
paper encourages researchers to solve discretized WFLO problems using message
passing algorithms. However, it is important to note that both MPLP and TRW are not
general purpose LP solvers, they both work well when the graph can be decomposed into
factors and sub-problems.
Yanover et al. [35] conducted a thorough empirical study to determine the similarities
and differences in using off-the-shelf LP solvers versus TRW to estimate the MAP
problem for common graphical models generally found in computational biology and
computer vision. The paper demonstrates that a MAP problem can be represented as a
type of combinatorial optimization problem using an integer program with binary indicator
variables. The paper shows that even when the integer constraints are relaxed, barrier
and dual LP solvers in CPLEX (IBM Corp., Armonk, NY) fail to solve problems that have
over 106 variables on standard desktop computers. The paper demonstrates that due to
the sparse nature of the indicator variable constraints, TRW exploits the relaxed LP
structure to obtain good MAP configurations faster than CPLEX (IBM Corp., Armonk, NY).
However, only when the LP solution is non-fractional (tight) that the TRW beliefs had a
unique MAP configuration.
2.2.7 Tightening LP Relaxations using Message-Passing Algorithms
Many message passing algorithms rely on the underlying assumption that the resulting
linear program is tight when the integer constraints are relaxed to continuous constraints.
However, this is hardly the case for general real-world MAP estimation problems. More
specifically, none of the discretized WFLO problems discussed earlier are tight when their
integer constraints are relaxed to continuous constraints. Thus, in order to use the
efficient message passing algorithms we need to further investigate ways in which integer
constraints can be systematically enforced.
Traditionally, cutting-plane algorithms have been employed within the combinatorial
optimization community to enforce violated integer constraints. For instance, Gomory cuts
16
are widely used in traditional integer program solvers to enforce integer constraints.
However, these methods are generic and not problem specific, which results in inefficient
[14].
In [36] and [37] Sontag et al. demonstrate that tighter relaxations can be achieved by
implementing higher order cluster-based LP relaxations that enforce local consistency.
Adding clusters is a lift-and-project method where the goal is to develop a relaxed
polytope that approaches the boundaries of the original integer polytope. This is achieved
by dynamically enforcing relaxations until the relaxed polytope is equal to the integer
polytope. It is expected that by developing a hierarchy of relaxations that the integer
polytope can be approximated without having to enforce all the constraints. Therefore, in
[36] clusters are incrementally added to the graphical model only when they are
guaranteed to improve the approximation, thus, as more clusters are added the relaxation
becomes tighter.
Therefore, it is hypothesized in this thesis that by applying message passing
algorithms to the relaxed WFLO MIP model while enforcing key constraints using cluster
based approach that good sub-optimal can be generated.
2.3 Discussion
WFLO is in general an intractable problem due to stochastic wind conditions, non-linear
aerodynamic wake interferences, and other difficult constraints. Simplified wake models
and linear constraints are used to make the problem tractable. In this review we have
shown that evolutionary algorithms can be used to develop heuristics that enable
researchers to solve the non-convex WFLO problems with nonlinearities. Nevertheless,
these algorithms are extremely slow and generally do not offer any optimality guarantees.
Further simplifications can be performed to formulate the WFLO problem as discretized
MIP, QIP, and CP models. Off-the-shelf exact solvers guarantee optimality but once again
are extremely slow and they fail to solve problem instances with a large number of
variables, for instance, Zhang et. al [24] solved the 20 x 20 WFLO instance, the largest
known instance, using mathematical programming with limited success. Therefore, there
is a need for an optimization methodology that is able to quickly approximate wind farm
17
layouts for a general class of problem instances, considering stochastic wind conditions,
non-linear aerodynamic wake interferences, and other difficult constraints for a given
formulation.
In this chapter, we have illustrated through existing literature that many combinatorial
optimization problems can be modeled as a probabilistic graphical model and MRF, and
then MAP configuration can be found by applying message passing algorithms. Message
passing algorithms work best when the graphical model can be decomposed into factors
of small cliques. More specifically, we show that MPLP and TRW algorithms work well
with MRF with cycles; however, these algorithms relax the underlying integer constraints
to solve the resulting LP. Thus, MPLP and TRW need to be applied alongside with lift-
and-project methods that enforce a series of hierarchical relaxations until the relaxed
polytope approximates the integer polytope near the corner point with optimal solution.
The underlying rationale of the method I propose is to frame the existing WFLO integer
problem formulation from [2] as a maximum-a-posteriori (MAP) inference problem in a
probabilistic graphical model by recasting the objective cost function as a graph structured
probability. Then, I will apply an approximate inference method using message passing
algorithm augmented with cluster based LP relaxations to sub-optimally approximate the
WFLO.
18
Chapter 3
Quadratic Knapsack Problem
Quadratic knapsack problem (QKP) is a binary integer program with a quadratic objective
function bounded by a linear capacity (budget) constraint. QKPs arise in many fields that
require maximizing resource allocation under a constrained budget capacity. While
several techniques that derive tight upper bounds have been extensively studied and are
effective at solving super-modular QKPs, they are computationally slow and memory
intensive to solve general QKPs without making any a priori assumptions. Additionally,
although QKPs have strong graph-theoretic properties, they have not been studied in the
context of approximate inference in graphical models using message passing algorithms
to generate approximate (sub-optimal) solutions to otherwise difficult QKPs.
In this research, we use the graphical model abstraction along with Lagrangian
augmentation to represent a QKP as an undirected graph. We develop a method to solve
a QKP using sequential tree-reweighted message passing (TRW-S) augmented with
triplet clusters to tighten the results. In this chapter, we perform an experimental
comparison between the effectiveness of solving benchmark QKPs using TRW-S and an
exact solver (branch-and-bound) over a varying number of variables and graph densities.
The subsequent chapter performs a similar computational study to solve the WFLO
problem.
19
3.1 Introduction
As discussed in earlier sections, undirected graphical models are a widely used
framework in machine learning to succinctly model pairwise relationships between a set
of random variables. This relationship is expressed in the form of an energy function (Eq.
(3.1)) with unary (φs(xs)) potentials over vertices 𝒱, and pairwise (φst(xs, xt)) potentials
over edges ℰ in a graph G = (𝒱, ℰ). In general, optimal inference over graphical models
is computationally intractable (NP-Hard), and using state-of-the art exact solvers (i.e.
CPLEX) to conduct exact optimization such as branch-and-bound and cutting planes as
a subroutine task is ineffective and time consuming. Recent advancements in messaging
passing algorithms, a class of decentralized and asynchronous optimization methods that
exploit graph factorization, have shown to well approximate MAP configurations with
some optimality guarantees.
E(x, φ) = ∑ φs(xs)s∈𝒱 + ∑ φst(xs, xt)(s,t)∈ℰ (3.1)
In this chapter, we model quadratic knapsack problem, a type of quadratic integer
program, as a graphical model and apply approximate inference with an expectation of
generating good sub-optimal solutions in comparison to computationally exhaustive exact
methods.
3.1.1 Quadratic Knapsack Problem
Despite major advances in message passing algorithms, their behaviour in maximizing
Eq. (3.1) under a global constraint has not been studied. The binary quadratic knapsack
problem (QKP) is a class of constrained combinatorial integer maximization problem with
a quadratic objective function (Eq. (3.2a)) and a global linear inequality constraint (Eq.
(3.2b)). Formally, QKP assumes that there are N items each with a positive weight ai, and
an integral profit di. Furthermore, for each selected item i {i|i ∈ N}, an integral profit cij can
be derived if item j, {j|j ∈ N, j > i} is also selected. Therefore, the problem involves
choosing a subset of items from N items such that the weight of chosen items does not
exceed b, known as the knapsack capacity, while maximizing the overall profit. The
objective function (Eq. (3.2a)) is an undirected graph with unary and pairwise potentials,
which is constrained by a linear capacity constraint (Eq. (3.2b)). The linear constraint,
20
also known as a knapsack polytope, appears in optimization formulations with a limited
budget capacity.
maximize𝐱 𝑓(𝑥) = ∑ ∑ cijxixjj∈Ni∈N + ∑ dixii∈N (3.2a)
subject to ∑ aixii∈N ≤ b (3.2b)
xi ∈ {0,1} ∀i ∈ {1, … , N} (3.2c)
The special case when profits cij = 0, the problem reduces to the standard knapsack
problem, which even though is a NP-hard problem can be solved using dynamic
programming in pseudo-polynomial time complexity O(bN) [38]. QKP is NP-hard in the
strong sense as shown by its reduction from the Clique problem in [39]. Contrary to
standard knapsack problem, no such pseudo-polynomial algorithm exists that can solve
all QKPs.
We motivate our study of QKPs as they commonly appear as sub-routines within a
wide variety of fields such as transportation planning [40], electronic circuit design [41],
finance [42], and wind farm layout optimization (WFLO) [2]. In this thesis, we design
methods that exploit the effectiveness of message passing algorithms on undirected
graphs bounded by a knapsack polytope to approximate a variety of QKPs.
3.2 Related Work
Existing work to solve QKPs mostly focusses on advancing exact algorithms such as
branch-and-bound by developing methods to generate better bounds and branching
strategies that yield a lower duality gap and a tighter integrality gap. Therefore, techniques
such as Lagrangian decomposition [43] [44], Lagrangian relaxation [39], semidefinite
programming [45], upper planes [46], linearization [47] and heuristic methods [48] have
been widely studied in the context of solving super-modular QKPs. A QKP is called super-
modular when all quadratic profits cij ≥ 0. Although these methods yield solutions with
optimality guarantees, they are ineffective at solving QKPs with a very large number of
variables, arbitrary profits, and in cases where QKPs appear as a subroutine as part of a
much larger optimization problem.
21
Therefore, we need to investigate approximate and heuristic algorithms that perform
reasonably well with general QKPs without making any a priori assumptions and when
the size of the problem scales exponentially. More specifically, for practical purposes we
need to test the viability of approximate algorithms that seek good solutions without
exhaustively searching for optimality. Rader et al. [5] illustrated that the graphical
representation of the problem’s objective function gives insight into the problem’s
solvability, that are otherwise difficult to understand. Rader et al. prove that graph’s that
are edge series-parallel (ESP) can be solved in pseudo-polynomial time in O(b2N) time
complexity using dynamic programming. Furthermore, a fully polynomial time
approximation scheme (FPTAS) is available when the graph is ESP and all profits, cij
and di, are non-negative (super-modular). Rader el al. [5] also show that graphs that are
vertex series-parallel (VSP) are strongly NP-Complete without a pseudo-polynomial time
solution unless P = NP. QKPs with positive and negative profits are in-approximable
unless P = NP. Similarly, Rader el al. [5] derive that tree-based graphs with arbitrary
profits do not have polynomial time approximation unless P = NP. It is important to note
that these results have only been studied under conditions with integral costs, cij and di.
3.2.1 Message Passing Algorithms
As Rader el al. [5] shows that the underlying graph-structure of the objective function
yields significant insight into the difficulty of QKPs, it is natural to investigate the
application of message passing algorithms to solve QKPs, as these algorithms have long
been used to factorize graphs to best estimate its MAP assignment. Pearl’s Max-product
Belief Propagation (BP) [6] is a common message-passing scheme that estimates the
MAP assignment of random variables by decomposing the graph G into factors and
iteratively passing beliefs along adjacent nodes over edges in a distributed, decentralized,
and asynchronous manner. Beliefs represent the potential value of the random variable
as determined by the unary and pair-wise potential function of adjacent nodes. BP is
guaranteed to converge to the optimal MAP configuration for acyclic MRFs [10]; although
no such convergence and optimality guarantees exist for cyclical (loopy) graphs, BP has
still shown to produce very good results [10].
22
Variants of BP such as Max-Product Linear Programming (MPLP) [8] and Sequential
Tree-Reweighted Message Passing (TRW-S) [7] conduct MAP estimation by linearizing
Eq. (3.1) into an integer program (MAP-IP) using auxiliary marginal variables {μs(xs)}s∈𝒱
and {μst(xs, xt)}(s,t)∈ℰ for every vertex and edge in the undirected graph G respectively.
These auxiliary variables also satisfy the conditions ∑ μs(xs)Xs= 1 and ∑ μst(xs, xt)Xs,Xt
=
1 [29]. The MAP-IP can be defined by Eq. (3.3) over the binary marginal polytope M(G)
from Eq. (3.4), where p(x, φ) is the graph G’s joint distribution characterizing its
factorization [29].
TRW and MPLP work by relaxing the marginal polytope in Eq. (3.4) and iteratively
minimizing the dual to approximate the lower bound using graph tree decompositions and
re-parameterization. These variants are guaranteed to produce optimal configurations for
marginal polytopes that are tight when relaxed.
MAP − IP: argmax𝛍∈M(G)
𝛗𝐓𝛍 = argmax𝛍∈M(G)
(∑ ∑ μs(xs)φs(xs)Xss∈𝒱 +
∑ ∑ μst(xs, xt)φst(xs, xt)Xs,Xt(s,t)∈ℰ )
(3.3)
M(G) = {𝛍 ∈ ℝd|∃p(x, φ) s. t μs(xs) = ∑ p(x, φ)x𝒱\s
μst(xs, xt) = ∑ p(x, φ)x𝒱\s,t
} (3.4)
Existing literature in the context of graphical models and probabilistic inference
remains inconclusive and uncertain regarding methods to incorporate linear inequality
constraints such as the knapsack polytope (Eq. (3.2a)) into graphical models. Therefore,
it is important to develop procedures that are able to effectively incorporate global linear
inequalities within graphical models so that message passing algorithms can become an
attractive and robust subroutine to generate good approximate feasible solutions within
large-scale optimization problems.
The purpose of this study is to determine methods to solve a variety of QKPs with
positive, negative, and real cost coefficients using message passing algorithm. We
investigate methods to incorporate knapsack inequality constraint within graphical models
to generate good approximate feasible solutions. Lastly, we test the effectiveness of these
23
methods to solve QKPs against exact methods solved using state-of-the-art solvers (i.e.
CPLEX).
3.3 Method
In the context of MRFs, the QKP objective function 𝑓(𝑥) (Eq. (3.2a)) can be reformulated
as an energy function shown in Eq. (3.1), defining the unary potentials (φs) using linear
costs and pairwise potentials (φst) using quadratic costs as illustrated in Eq. (3.5a) and
Eq. (3.5b) respectively.
φs(xs) = dsxs, ∀ s ∈ 𝒱 (3.5a)
φst(Xs, Xt) = cstxsxt, ∀ (s, t) ∈ ℰ (3.5b)
There are two methods that can be used to augment MRFs to incorporate knapsack
polytope feasibility. These two methods are based on penalty methods and auxiliary
variables.
3.3.1 Penalty Method
QKP can be transformed into an unconstrained quadratic integer program (QIP) using
penalty methods based on the augmented Lagrangian function as described in [49]. It is
hypothesized that the unconstrained QIP is generally easier to solve compared to the
constrained QKP. We use a quadratic penalty function to incorporate the knapsack
capacity constraint into the quadratic objective function, and the resulting augmented
Lagrangian function with the penalty parameter, μ is shown in Eq. (3.6).
𝐿(𝑥, 𝜇) = 𝑓(𝑥) + μ [max {0,1 −1
b∑ aixij∈N }]
2
(3.6)
The undirected graph, G𝐿 representing 𝐿(𝑥, 𝜇) can have a higher or equal graph
density compared to the constrained undirected graph G. Nevertheless, both BP and
TRW-S have been applied to complete graphs with some success in [7] .
maximize𝐱 𝐿𝑘(𝑥, 𝜇𝑘) (3.7a)
subject to xi ∈ {0,1} ∀i ∈ {1, … , N} (3.7b)
24
The unconstrained QIP is shown in Eq. (3.7a), where {𝜇𝑘} is a sequence of penalty
parameters such that 0 < μk < μk+1. Therefore, the unconstrained QIP in Eq. (3.7a) is
successively solved for a given penalty parameter μk until Lk(x, μk) converges to a
stationary point within the feasible domain. In general, the problem may become ill-
conditioned if μk → ∞. This procedure is described in Algorithm 1.
Algorithm 1: Penalty Method
1: Inputs: QKP with model parameters: set of cost coefficients ({d}i, {c}ij), set of knapsack
capacity coefficients {𝑎}𝑖, knapsack capacity 𝑏, and problem size 𝑁
2: Result: Approximate MAP: argmax𝒙 (𝐿(𝑥, 𝜇) = 𝑓(𝑥) + μ [max {0,1 −1
b∑ aixij∈N }]
2)
3: Choose step-size sequence {γ(i)}0∞ in descending order
4: Initialize penalty parameter 𝜇(𝑘=0) = 𝛾𝑖=0, iteration 𝑘 = 0
5: while ‖𝑓(𝒙𝒌+𝟏∗ ) − 𝑓(𝒙𝒌
∗ )‖ is below some threshold (convergence-check) do
4: Compute 𝒙𝒌
∗ ← argmax𝒙 𝐿(𝑥, 𝜇𝑘) = 𝑓(𝑥) + μk [max {0,1 −1
b∑ aixij∈N }]
2 using TRW-S
5: if 𝒙𝒌∗ is feasible (∑ ai𝑥𝑘𝑖
∗j∈N ≤ b) then
6:
Update step-size μk+1 ← μk + γi
Update iteration count k ← k + 1
7: else if 𝒙𝒌∗ is not feasible (∑ ai𝑥𝑘𝑖
∗j∈N > b) then
8:
Update step-size by reverting to previous feasible penalty parameter and incrementing
μk ← μk − γi + γi+1 Update step-size
i ← i + 1 end if
9: end while
10: Return Approximate MAP result 𝑓(𝑥𝑘) and arguments 𝑥𝑘
3.4 Experiments and Discussion
We use Algorithm 1 over benchmark dataset to establish a comparison between the
solutions generated using TRW-S and an exact solving algorithm. Results generated by
Algorithm 1 over benchmark dataset gives us insight into the benefits and limitations of
TRW-S as applied to QKPs of varying difficulties. In the subsequent chapter, we perform
a similar computational study in the context of wind farm layout optimization.
25
3.4.1 Benchmark Experiments
Experiments are conducted on benchmark dataset to measure the effectiveness of
message passing algorithms with existing methods. The benchmark dataset as shown in
[46], [44], [47], [39], [50], and others use test cases that randomly generate 10 test
problems for 100, 200, and 300 variables for a range of graph densities (∆d). The set of
cost coefficients c and d for every test case are uniformly distributed integers between 0
and 100, while the set of knapsack capacity coefficients a are uniformly distributed
integers between 0 and 50, and the capacity constraint b is randomly selected integer
between 50 and max(50, ∑ aii∈N ). Billionnet et al. [50] showed these test cases to be
reasonable difficult by measuring the gap of random feasible solutions and optimal
solutions to be between 40% – 60%.
For each set of 10 test problems with a specific number of variables and graph density,
we apply TRW-S via Algorithm 1 to measure the average error (Eμ) and the standard
deviation in the average error (Eσ). We also measure the average time (Tμ) taken to
compute the MAP configuration over a convergence criteria, and the standard deviation
in the average time (Tσ) is also calculated.
In general, MAP configurations generated using TRW-S are not tight since TRW-S
computes fixed-points (extreme points) over a relaxed LP, which is transformed from a
QIP using a linearization technique. Therefore, tighter relaxations are generated by
iteratively generating triplet clusters that enforce local consistency between cluster
marginals based on cutting-plane methods and Sherali-Adams hierarchy as illustrated by
Sontag et al. [36] and [37]. Clusters are iteratively searched and added after termination
of Algorithm 1. Thus, measurements using this approach is also computed, and it is
expected that MAP configurations are tighter and yield better value at convergence.
Table 3.1 shows the results generated using the two approaches and an exact method
that uses Lagrangian decomposition to generate upper bounds [50]. We compare our
results to ones from [50], as generating upper bounds to compute exact solution is
representative of the existing direction of research in generating optimal solutions to
QKPs.
26
As shown in Table 3.1, we can see that applying TRW-S using Algorithm 1 without
clusters yields sub-optimal solutions in time that is several orders of magnitude less than
the time it takes exact method to generate optimal solution. Nevertheless, these solutions
have an average error of 2-4%. Depending on the application type and solution reliability
constraints, a higher threshold of average error maybe acceptable. As searching for
triplet-clusters is a computationally exhaustive task [37], we see that generating a tighter
solution is time consuming, however, it yields better MAP assignments compared to
applying TRW-S without clusters. While both methods are unable to generate optimal
solutions for all test cases as illustrated in Figure 3.1 and Figure 3.2, generating triplet-
clusters consistently generates solutions with same or better MAP assignments as
quantified by a lower error rate.
TRW-S performs well in comparison with an exact method. Applying TRW-S with
clusters over test cases with 200 variables in a fully-connected graph produces an error
margin of less than 1% using 40 times less the amount of time as exact method.
Performance with 100 and 300 variables is also computationally comparable to exact
method. We believe that performance of TRW-S can be further enhanced by exploiting
the distributed and asynchronous properties of message passing algorithms by
implementing parallel processes.
Test Case TRW-S (Via Algorithm 1) TRW-S (Via Algorithm 1) with
triplet-cluster tightening
Exact Solution using Lagrangian
Decomposition [50]
𝐍 ∆d(%) Eμ(%) Eσ Tμ(s) Tσ Eμ(%) Eσ Tμ(s) Tσ Tμ(s) Tσ
100 25 4.00 2.75 7 5 2.71 1.52 394 175 117 122
200 100 2.19 3.34 64 37 0.66 0.93 1,656 811 70,633 167,326
300 25 3.87 3.25 219 120 3.31 3.35 7,252 1,151 7,765 7,600
50 2.41 2.12 321 127 0.69 0.63 6,814 1,291 4,786 4,591 Table 3.1: A comparison of results generated using penalty method, penalty method applied with tightening technique, and Lagrangian decomposition to generate upper bounds.
27
Figure 3.1: Resulting error of MAP assignments generated using TRW with and without triplet-clusters in comparison to exact method. Test cases labelled 100:1-10 correspond to a QKP with 100 variables with 25% graph density. Test cases labelled 200:1-10 correspond to a QKP with 200 variables with 100% graph density.
Figure 3.2: Resulting error of MAP assignments generated using TRW with and without triplet-clusters in comparison to exact method. Test cases labelled 300:1-10 correspond to a QKP with 300 variables with 25% graph density. Test cases labelled 300:11-20 correspond to a QKP with 300 variables with 50% graph density.
3.5 Conclusion
In this chapter, we derived a method to solve QKPs using approximate inference in
graphical models using message passing algorithms. While unconstrained quadratic
objective function have been represented as graphical models and solved using a variety
of message passing algorithms such as BP, MPLP, and TRW-S, they have not been
studied when the quadratic objective function is constrained by a linear capacity
constraint. We incorporate this constraint into the objective function using Lagrangian
augmentation and a quadratic penalty function, thereby making the problem an
unconstrained QIP that can be represented as a graphical model. By applying TRW-S
over a variety of benchmark test cases, we showed that TRW-S can be used as an
alternative method to generate good feasible results to QKPs.
28
Chapter 4
Wind Farm Layout Optimization
As rising greenhouse emissions threaten to harmfully disrupt the climate and eco-system,
there is an immediate need to produce sustainable energy to meet the world’s growing
energy demand. Capturing wind energy will play a major role in our shift to sustainable
energy generation. A common problem encountered while designing a wind farm is to
determine the optimal location of wind turbines within a fixed geographical area to
maximize the total power capacity of the wind farm under stochastic wind conditions and
non-linear aerodynamic interferences between the turbines. This is known as the Wind
Farm Layout Optimization (WFLO) problem. This thesis investigates optimization
techniques that can fast approximate turbine layouts to aide engineers to quickly assess
and plan infrastructure design decisions.
We discretize the wind farm and use a quadratic integer formulation of the WFLO
problem to model an undirected graph that succinctly captures the spatial dependencies
between the variables due to aerodynamic interferences caused by wind regimes. By
performing probabilistic inference using sequential tree-reweighted message passing
(TRW-S) on the undirected graphical model the most probable placement of the turbines
is approximated. Turbine placements are also determined using branch-and-cut algorithm
using CPLEX, a state-of-the-art exact solver. This research conducts a computational
study to compare the effectiveness of turbine layouts produced using approximate
probabilistic inference with computationally exhaustive branch-and-cut algorithm under
varying wind-regime complexity and discrete resolutions. We demonstrate that
probabilistic inference can produce sub-optimal turbine layouts that are within 5% of the
power capacity of optimal layouts for low resolution. Our results also suggest that
probabilistic inference can produce approximate layouts that are equivalent or better than
branch-and-cut in a shorter period of time for higher resolutions.
29
4.1 Introduction
In 2014, a total of $270 billion was invested in renewable energy technologies that
resulted in an addition of 128GW power capacity, of which 37% was due to new wind
farm installations [1]. As the world’s energy demand continues to rise, governments
continue to invest heavily in wind farms to harness wind energy to limit their greenhouse
gas emissions to curtail the threat of climate change and achieve energy security.
Therefore, an important stepping stone in achieving this goal is to build fast computational
tools that help design efficient large-scale wind farms that maximize power generation
and minimize infrastructure costs, while adhering to local land-use, environmental, and
mechanical constraints.
Wind Farm Layout Optimization (WFLO) is a problem that consists of determining the
optimal location of wind turbines within a fixed geographical area to maximize the total
power capacity of the wind farm under stochastic wind conditions and non-linear
aerodynamic interferences (wake effects) between the turbines. Wake effects created by
turbines effectively reduce the wind speed directly downstream of the placed turbines,
which decreases the expected power capacity of any turbines placed in their wakes.
Additionally, wakes can overlap with each other to further decrease the effective wind
speeds. Thus, turbines need to be placed optimally to minimize the total wake effects
caused by complex wind regimes.
Existing work focuses primarily on two key areas – constructing accurate wake models
and improving layout optimization algorithms. Standard wake models such as Jensen [3]
and Eddy Viscosity [51] quickly calculate the wake decay constant through momentum
mixing rate simplifications and ignoring the turbine effects on the planetary boundary
layers (PBL) [52]. Computationally expensive models (i.e. Deep-Array Wake Model,
Fuga, and WindModeller) capturing detailed wake interactions can be generated using
complex Reynolds-averaged Navier-Stokes equations and large eddy simulations [52]. In
this resarch, we use Jensen’s wake model and focus on developing fast optimization
algorithms for WFLO problem with discretized wind farms.
30
In developing better optimization algorithms, the two most important factors of any
wind farm design objective function are minimizing wake effects and power capacity. A
more comprehensive objective function maximizes the wind farm’s net-present-value
profitability by incorporating installation (i.e. civil and electrical), maintenance, and
operations costs in addition to wake effects and power capacity as illustrated by Gonzalez
et al. [53]. Furthermore, design constraints such as land-use availability, noise generation,
turbine proximity, and other infrastructure constraints generally accompany objective
functions. In this thesis, we focus on developing fast approximate optimization algorithms
by taking into account wake effects and power capacity for small and large wind farms.
4.2 Literature Review
Early research used evolutionary algorithms (EA) to maximize power capacity and
minimize infrastructure costs in small wind farms. In 1994, Mosetti et al. [16] formulated
the WFLO problem as a small discretized wind farm by modelling the wake interactions
using Jensen’s wake model and using a simplified turbine cost model. Genetic algorithm
(GA) was used to find sub-optimal layouts with completely random initial configuration for
varying complexity of wind regimes. By fine tuning GA parameters such as crossover
type, diversity percentage, population size and generation count, Grady et al. [17]
obtained better layouts with higher power capacity and lower installation costs. Huang et
al. [18] further enhanced earlier approaches by applying a distributed genetic algorithm
to decompose a large wind farm terrain search space into local search spaces to
maximize power capacity. Thus, a distributed approach yields better solution quality and
is computationally faster than traditional genetic algorithms. By incorporating problem
specific heuristics and fine tuning optimization parameters, EAs can find good solutions
to non-linear, non-convex, and non-differentiable functions that are difficult to solve using
other methods. Nevertheless, EAs don’t offer optimality guarantees and can be
computationally expensive.
Particle swarm optimization (PSO) is applied in [54], [55] by Chowdhury et al. to
varying WFLO objective functions to handle discrete and continuous variables with non-
convex objective functions. Similarly, Wan et al. [56] apply Gaussian PSO while also
31
incorporating a local search strategy based on differential evolution to enhance the
optimization results. Due to the varying objective functions and varying convergence
criteria, it is unclear whether PSO offers any computational time complexity advantage
over GA methods.
By discretizing the wind farm terrain into grid cells and representing each cell as a
mathematical decision variable, mathematical programming models can be developed for
the WFLO problem. Individual decision variables take on a binary state and determine
whether at every location a turbine is placed or not placed. The relationship between the
decision variables (i.e. wake interactions, infrastructure costs, etc.) can be encoded within
the objective function and hard constraints can be developed for budget, spacing,
proximity, terrain usage, noise, and other infrastructure constraints. Integer programs can
be solved exactly using well-designed algorithms [19] that are generally implemented in
an off-the-shelf commercial solver such as CPLEX (IBM Corp., Armonk, NY), and have
several theoretical guarantees. However, in many cases, models need to be simplified to
make the problem tractable and convex.
Donovan [20], [21] and Archer et al. [22] used a mixed integer problem (MIP) model
to minimize the wake interactions between the layout turbines. Donovan exploits the
inherent graph based structure of the problem to model the problem as a vertex packing
problem, and further explores varying branching strategies within the branch and bound
algorithm to reduce optimization time. Turner et al. [2] formulated a binary quadratic
integer program (QIP) by minimizing the wind speed deficit at each turbine as derived
from Jensen’s wake model. In [2], Adams-Sherali zero-one quadratic program
linearization technique [57] is applied to convert the QIP to a MIP. It is shown that MIP
solutions in [2] outperforms EA for complex wind regimes; nevertheless, it can take up to
24 hours to obtain good solutions for standard small wind farms. Thus, there is a need for
finding an algorithm that quickly produces good solutions for mathematical program
formulations of WFLO.
In this thesis, we pose the binary QIP formulation from [2] as an undirected graph
known as a Markov Random Field (MRF) where the pairwise wake interactions and global
32
constraints are encoded within the graph’s edges. Probabilistic inference known as
maximum-a-posteriori (MAP) is conducted on the resulting MRFs’ joint log-likelihood
function to determine the optimal turbine placements. The overall goal of this method is
to approximate turbine layouts for small to large wind farms much faster than solving the
mathematical model using exact solvers to aide engineers quickly assess and plan
infrastructure design decisions.
4.3 Background
The power capacity of each turbine is proportional to the cube of the effective local wind
speeds at the turbine hub height. Similar to [16], [17], and [2], the total power capacity of
the wind farm is calculated using Eq. (4.1), where ui is the effective wind speed of the ith
turbine of a wind farm with a total of K turbines, and the coefficient is developed from
empirical observations.
Ptotal = ∑ 0.3uj3K
i=1 (4.1)
4.3.1 Wake Modelling
By assuming that momentum is conserved in the turbine wakes, the effective wind speed
can be calculated using Jensen’s wake model (Eq. (4.2)). The effective wind speed u at
the turbine hub height is directly related to the distance x from a turbine upstream.
Additionally, the effective wind speed is dependent upon the axial induction factor a,
entrainment or wake decay constant α, downstream rotor radius r1 that is the wake radius
immediately after the upstream turbine, and freestream wind velocity uo. A schematic
illustration of Jensen’s wake model for a turbine, T, is shown in Figure 4.1.
u = uo [1 −2𝑎
(1+𝛼(𝑥
𝑟1))
2] (4.2)
33
Figure 4.1: Jensen’s wake model schematic illustration
The turbine induction factor, wake decay constant, and downstream rotor radius are
calculated using Betz’s theory [58] as shown in Eq. (4.3) – (4.5) where RR is the turbine
radius, CT is the turbine thrust coefficient, z is the turbine hub height, and zo is the surface
roughness constant. The rate of expansion of the wake downstream is given by Eq. (4.6),
where rx is the radius of the wake distance x downstream of the turbine. Finally, to avoid
structural damage to the turbines due to wake interactions, the proximity between any
two turbines need to be at least more than 5 times their rotor radius.
CT = 4a(1 − a) (4.3)
r1 = Rr√1−𝑎
1−2𝑎 (4.4)
α =0.5
ln(𝑧
𝑧𝑜) (4.5)
rx = r1 + αx (4.6)
4.3.2 Wind Farm Layout Optimization Mathematical Program
In order to simplify wind farm modeling and optimization, the land being surveyed for
turbine siting can be discretized into smaller areas. More particularly, the wind farm can
be divided into cells, in which each cell can only hold a single turbine, thereby limiting the
number of possible locations that a turbine can be placed within the wind farm. Thus, the
problem space is discretized into N cells, where each cell is represented with a binary
variable xi such that i ∈ I, I = {1, … , N}. If a turbine is placed at a given location i then the
xi takes on the value of 1 and 0 otherwise. In order to calculate the effective wind speed
at any location i, wakes caused by turbines upstream need to be considered (Figure 4.2).
34
Figure 4.2: Wakes effecting Turbine 3 (T3) due to the wake region produced by Turbine 1 and 2 (T1, T2)
The effective wind speed at a given location j, j ∈ I due to multiple wakes is calculated
by assuming that the kinetic energy deficit at j is the linear sum of all the kinetic energy
deficits caused by the individual wakes produced by turbines at locations i upstream of j
(Eq. (4.7)). Thus, the effective wind speed uj of a downstream turbine j is calculated using
Eq. (4.8).
(1 −uj
uo)
2
= ∑ (1 −ui
uo)
2N
i=1,i≠j
xi (4.7)
uj = uo [1 − √∑ (1 −ui
uo)
2
xiNi=1,i≠j ] (4.8)
As illustrated by Turner et al. [2] maximizing wind farm power capacity is equivalent to
maximizing the effective wind speed at individual turbine hub heights. Eq. (4.9) maximizes
the effective wind speed by only placing turbines at optimal locations j that have an overall
smaller kinetic energy deficits and higher effective wind speeds. The freestream constant
speed uo is omitted as it does not affect the maximization.
max ∑ xjNj=1 [1 − √∑ (1 −
ui
uo)
2Ni=1,i≠j xi] (4.9)
Similarly, a quadratic integer program (QIP) is derived in [2] by illustrating that
maximizing effective wind speed is approximately equivalent to minimizing the kinetic
energy deficit (Eq. (4.10)).
35
min ∑ ∑ xj (1 −ui
uo)
2
xiNi=1.i≠j
Nj=1 (4.10)
An interaction matrix W ∈ ℝNxN can be generated by calculating the kinetic energy
deficits at every location j ∈ I caused by a turbine at all locations i ∈ I, i ≠ j prior to the
optimization (Eq. (4.11)). The sparsity of W depends upon wake parameters and wind
regime complexity.
wij = {∑ ∑ (1 −ui
uo)
2Ni=1
Nj=1
0
, i ≠ j, i = j
(4.11)
The QIP formulation can be written as Eq. (4.12a), where X = (x1, x2, … , xN) is a binary
decision vector such that X ∈ {0,1}N. Due to budget constraints, land feasibility,
government regulations, and grid capacity the number of turbines to be placed on the
wind farm is determined prior to the farm development. Thus, the turbine constraint in
Eq. (4.12a) enforces that only a set number of turbines K will be placed on the farm, where
𝐞 is a vector {1, … ,1}N. Depending on the Jensen model’s parameter values, complexity
of the wind regime, and the problem state space, QIP can be highly non-convex and non-
tractable.
minimize XTWX (4.12a)
subject to ∑ xjNj=1 = K or 𝐞T𝐗 = K (4.12b)
xj ∈ {0,1} (4.12c)
4.3.3 WFLO as a Markov Random Field
The binary QIP (Eq. (4.12a)) is posed as an undirected graph G = (𝒱, ℰ) known as a
Markov Random Field (MRF) with a set of vertices 𝒱 represented by decision vector X ∈
{0,1}N (also known as random variables) connected by edges ℰ that encode the pairwise
wake interactions from W and the turbine constraints. To encode the turbine constraint
from Eq. (4.12b), we dualize the constraint using an augmented Lagrangian method by
applying a quadratic penalty function (Eq. (4.13)), which can be simplified into Eq. (4.14).
36
f(X, β) = XTWX + β(𝐞TX − K)T(𝐞TX − K) (4.13)
f(X, β) = XTWX + β𝐗T [
00⋮
2 ⋯ 20 ⋱ ⋮⋮ ⋱ 2
0 0 ⋯ 0
] X + β(−2K + 1)𝐞TX + K2 (4.14)
Eq. (4.15) shows the unconstrained binary QIP where parameter β affect the
smoothness of the penalty contour and the K2 constant term is omitted as it does not
affect the optimization.
argminX∈ {0,1}N,β
XT(W + βE)X + (−2Kβ + β)𝐞TX (4.15)
The relationship between the random variables from Eq. (4.15) can be concisely
captured using a graphical model. Thus, the resulting undirected graph G (Eq. (4.15)) can
be formulated as a probabilistic graphical model, which is a graphical representation of
the joint probability distribution of the set of random variables [13]. Probabilistic inference
known as maximum-a-posteriori (MAP) is conducted on the resulting MRFs’ joint log-
likelihood function to determine the optimal turbine placements. The overall goal of this
method is to approximate optimal turbine layouts much faster than solving the
mathematical model using exact solvers to aide engineers quickly assess and plan aide
infrastructure design decisions.
MRFs can be decomposed as a product of factors composed of a set of random
variables Xc over the set of maximal cliques C of graph G [13], and such that the set of
maximal cliques satisfy Xc ⊂ X ∀ c ∈ C and ⋃ Xcc∈C = X. Depending on the Jensen
model’s parameter values and the wind regime, the interactions between the random
variables maybe localized and form maximal cliques. Thus, given a set of binary random
variables X ∈ {0,1}N, the joint distribution of a pairwise MRF based on the QIP model (Eq.
(4.15)) is shown in Eq. (4.16), which is proportional to the product of potential
functions θC(XC) over maximal cliques C, where Z is the partition function (Eq. (4.17)). The
partition function normalizes the joint distribution.
p(X) =1
Z∏ θC(XC)C (4.16)
37
Z = ∑ ∏ θC(XC)CX (4.17)
The positive potential functions θC(XC) encodes the problem specific relationships
between the random variables, which in this case are the pairwise wake interactions and
turbines constraints. Furthermore, the potential function is generally expressed as an
exponential function, Eq. (4.18), where E(Xc) represents the energy function. Thus, a
random variable assignment with a higher probability has a lower overall energy [15].
θC(XC) = exp(−E(Xc)) (4.18)
The product of the exponential functions results in the sum of factors of energy
functions (see Eq. (4.19)), where φs(Xs) represent all unary potentials over the model
vertices 𝒱, and φst(Xs, Xt) represent the pairwise potentials over the edges ℰ. The term
(W + β1E) and (−2Kβ + β) from Eq. (4.15) represent the coefficients of the quadratic
pairwise and linear unary potentials respectively.
E(X, φ) = ∑ φs(Xs)s∈𝒱 + ∑ φst(Xs, Xt)(s,t)∈ℰ (4.19)
Then the joint distribution (i.e. Gibbs distribution) [15] is given in Eq. (4.20).
p(X, φ) =1
Zexp(− ∑ φs(Xs)s∈𝒱 − ∑ φst(Xs, Xt)(s,t)∈ℰ ) (4.20)
Determining the configuration of random variables XM in a graphical model that yields
the maximum probability over the joint distribution (Eq. (4.20)) is known as maximum-a-
posteriori (MAP) inference problem as formulated in Eq. (4.21). This is equivalent to
minimizing the energy function (Eq. (4.22)).
XM = argmaxX
p(X, φ) (4.21)
XM = argminX∈ {0,1}N
E(X, φ) = argminX∈ {0,1}N
(∑ φs(Xs)s∈𝒱 + ∑ φst(Xs, Xt)(s,t)∈ℰ ) (4.22)
The MAP problem can be linearized by introducing auxiliary marginal variables over
the random variables associated with the vertices {μs(Xs)}s∈𝒱 such that ∑ μs(Xs)Xs= 1,
and for every edge {μst(Xs, Xt)}(s,t)∈ℰ within the graphical model such that
38
∑ μst(Xs, Xt)Xs,Xt= 1 [29]. Thus, the linear program can be defined by Eq. (4.23) over the
binary marginal polytope M(G) (Eq. (4.24)) where the dimension d of 𝛍 is equivalent to
2|𝒱| + 22|ℰ| and the vertices of M(G) are always integral. In general, MAP problem is an
NP-hard problem [15].
argmin𝛍∈M(G)
𝛗𝐓𝛍 = argmin𝛍∈M(G)
(∑ ∑ μs(Xs)φs(Xs)Xss∈𝒱 + ∑ ∑ μst(Xs, Xt)φst(Xs, Xt)Xs,Xt(s,t)∈ℰ )
(4.23)
M(G) = {𝛍 ∈ ℝd|∃p(𝐗, 𝛗) s. t μs(Xs) = ∑ p(𝐗, 𝛗)X𝒱\s
μst(Xs, Xt) = ∑ p(𝐗, 𝛗)X𝒱\s,t
} (4.24)
The IP in Eq. (4.23) can be solved using cutting plane, branch-and-bound, and branch-
and-cut algorithms by solving a partially relaxed IP and iteratively applying appropriate
integer constraints to ensure feasibility and satisfactory optimal or sub-optimal solutions.
However, depending on the number of variables and state-space, general purpose
solvers can take a very long time to solve this problem. For example, the branch-and-cut
method, when applied to linearized QIP from Eq. (4.12c) for complex wind regimes, can
take up to 24 hours to solve discretized problem space with 100 potential turbine locations
[2].
4.3.4 Message-Passing Algorithms
It is hypothesized that approximate solutions can be generated quickly for large MRFs by
class of algorithms known as message-passing algorithms that works by exploiting the
decomposable factors within the graphical model. Message-passing algorithms
approximate MAP assignments of relaxed IPs by iteratively passing beliefs locally along
the edges of the graphical model in a distributed, decentralized, and asynchronous
manner.
In 1988, Pearl [6] developed the max-product belief propagation (BP), which is
guaranteed to converge to the optimal MAP configuration for graphical models with only
trees. BP is not guaranteed to converge or find optimal configurations for graphical
models with cycles. Nevertheless, BP has still produced good empirical results as shown
39
in the fields of computer vision (e.g. [9] and [10]) and computational biology (e.g. [12])
with problem structures similar to Eq. (4.15).
Multi-directional wind regimes create cycles within the MRF; thus, it is important to
study variants of BP that properly handle graphs with cycles to optimally estimate the
MAP assignment. Two such variants of BP that efficiently compute MAP configuration for
MRFs with cycles are Max Product Linear Programming algorithm (MPLP) and Tree-
Reweighted Message Passing (TRW).
MPLP [8] uses block coordinate descent in the dual of the LP Relaxation of the MAP
problem to compute beliefs between nodes. MPLP is guaranteed to converge and
computes the exact solution to the relaxed LP problem for binary variables. Similarly, in
[29] Wainwright et al. developed the TRW algorithm by decomposing an arbitrary
graphical model with cycles into a convex combination of tree-structured distributions to
calculate the optimal upper bound. The authors show that an optimal MAP configuration
and a tight lower bound can be obtained if and only if the combination of tree distributions
share a common optimum when the integer constraints are relaxed. TRW works by
maximizing the lower bound of the free energy objective. However, Kolmogorov [7]
showed that TRW does not guarantee to increase this lower bound, and in certain
situations may actually decrease the lower bound value during the iterative algorithm.
Furthermore, TRW is not guaranteed to converge. In [7], Kolmogorov developed the
sequential tree-reweighted message passing algorithm (TRW-S) that guarantees that the
lower bound will never decrease. Due to its convergence properties, in this work, we use
TRW-S to minimize the objective function in Eq. (4.23) over the binary marginal polytope
to determine the approximate turbine layout configurations.
Both MPLP and TRW-S rely on the assumption that the resulting linear program is
tight when the integer constraints are relaxed by forming a local polytope L(G), where
M(G) ⊂ L(G) and min𝛍∈L(G)
𝛗𝐓𝛍 ≤ min𝛍∈M(G)
𝛗𝐓𝛍. The relaxed constraints are hardly tight, which
requires us to investigate methodologies that can be used to dynamically generate
constraints within the message passing paradigm to make the solution space results tight.
40
Similar to cutting-plane algorithms, in [36] and [37] Sontag et al. demonstrated that
tighter relaxations can be acquired by iteratively enforcing edge consistency over a small
subset of triplet clusters c ⊂ C to generate the polytope P. Similar to first lifting of the
Sherali-Adams hierarchy, clusters are chosen at every iteration that improve the dual LP
bound to create a sequence of polytopes P0 ⊂ P1 ⊂ P2 … ⊂ M(G) in which the relaxations
are continually tighter and approach the marginal polytope [14]. Thus, by iteratively
approximating the MIP solution using TRW-S and generating tighter polytopes, it is
expected that good turbine placements can be acquired in a relatively shorter amount of
time compared to other exact solver techniques.
It is hypothesized that by applying message passing algorithms, more specifically
TRW-S, to the relaxed WFLO MIP model, while enforcing key constraints using cluster
based approach, an approximate and sub-optimal turbine layouts can be generated faster
than traditional exact approaches. In this research, we conduct a thorough computational
study to measure the effectiveness of TRW-S in comparison with branch-and-cut
algorithm to generate turbine layouts under varying wind regime complexity and problem
dimensionality. Please note that in the following sections, to preserve simplicity, we refer
to TRW-S as message passing algorithm (MP).
4.4 Experimental Results and Discussion
We calculate the effectiveness of message-passing algorithm (MP) over two benchmark
wind resources – WR-1 and WR-36. Wind regime WR-1 has uniform wind speed blowing
from a single direction, and WR-36 is a complex wind regime with varying wind speeds
blowing from 36 uniformly distributed directions with varying probabilities. The wind rose
of WR-1 and WR-36 are illustrated in Figure 4.3 and Figure 4.4 respectively. Wake
modelling, wind farm, and simulation parameters are listed in Table 4.1; the parameter
values and wind regime distributions are chosen to establish a fair comparison with
respect to results from [16], [17], and [2]. Additionally, we compare results obtained by
solving the linearized MIP using exact state-of-the-art solver CPLEX 12.1 (IBM Corp.,
Armonk, NY) by applying branch-and-cut algorithm, and solving the unconstrained
objective function (Eq. (4.23)) over the relaxed marginal polytope (Eq. (4.24)) using the
41
developed approximate solver based on MP method on varying discrete resolutions (100,
400, and 2,500 square cells), while keeping the square farm area to a constant at 4km2.
Figure 4.3: Wind rose of unidirectional wind regime
Figure 4.4: Wind rose of multidirectional complex wind regime
42
Parameter Value
Turbine rotor radius (𝐑𝐑) 20 m Turbine hub height (𝐳) 60 m Thrust coefficient (𝐂𝐓) 0.88 Surface roughness constant (𝐳𝐨) 0.3 Axial induction factor (𝒂) 0.3268 Entrainment/wake-decay constant (𝜶) 0.1
Farm size 4km2 (2,000m x 2,000m)
Workstation Intel® Xeon® Processor 16 cores @ E5-2450 2.10GHz 256 GB Memory
Exact solver CPLEX 12.1 (IBM Corp., Armonk, NY) Table 4.1: Table of parameters
4.4.1 Case 1 – 100 cells
Discretizing the 4km2 wind farm area into 100 cells yields a square cell size
of 200m x 200m, which automatically satisfies the turbine proximity constraint. Wind farm
layout power capacity results for varying number of turbines for WR-1 and WR-36 are
shown in Table 4.2 and Table 4.3 respectively. The corresponding values in the columns
are calculated using branch-and-cut algorithm and MP methods. Initially, the branch-and-
cut algorithm is conducted using CPLEX using 1 thread with a cut-off of 1 hour (E1);
however, if relative optimality is not guaranteed as indicated by the relative gap, further
branch-and-cut iterations are conducted based on 4 threads with a cut-off of 1 hour (E2)
and 12 threads with a cut-off of 24 hours (E3). An asterisk (*) expresses that an optimal
solution was found, while a dash (-) indicates a 100% gap. MP simulations are run using
source-code in [59] by applying a total of 5,000 triplet clusters with a cut-off of 1 hour on
a single thread. Triplet clusters used to tighten the relaxed marginal polytope (Eq. (4.24))
are generated using modified source-code in [60] developed from [37].
43
MIP – 1 Thread – 1 hour
cut-off (E1) MP
K Time (s)
Gap (%)
Power (W)
Time (s)
Power (W)
10 1 * 5,184 3,303 5,184
20 21 * 10,157 3,414 9,974
30 2 * 14,410 3,824 13,972
40 1 * 17,833 3,953 17,122
50 1 * 20,256 3,119 19,951
60 1 * 21,927 3,944 21,850
70 1 * 22,929 3,570 22,795
80 1 * 23,761 3,269 23,706
90 1 * 24,466 3,093 24,443
100 1 * 25,121 50 25,121
Table 4.2: Power capacity results under WR-1 with 100 cells
MIP – 1 Thread – 1 hour
cut-off (E1) MIP – 4 Threads – 1 hour
cut-off (E2) MIP – 12 Threads – 24
hour cut-off (E3) MP
K Time (s)
Gap (%)
Power (W)
Time (s)
Gap (%)
Power (W)
Time (s) Gap (%)
Power (W)
Time (s)
Power (W)
10 3,570 11 9,302 5,455 0 9,302 8,654 0 9,302 3,046 9,096
20 3,590 83 17,862 11,516 75 17,862 583,138 41 17,920 3,304 17,447
30 3,591 89 25,834 11,787 79 25,891 788,843 57 25,900 3,523 25,334
40 3,597 50 33,546 13,257 43 33,548 847,190 33 33,576 3,604 32,575
50 3,598 15 40,900 12,390 11 40,900 670,572 1 40,937 3,399 40,270
60 3,590 47 47,726 13,251 5 47,763 201,977 1 47,809 3,700 47,050
70 3,576 4 54,125 12,856 3 54,196 657,841 1 54,197 3,369 53,741
80 3,590 2 60,341 13,132 2 60,347 237,185 1 60,347 3,180 59,934
90 296 * 66,249 219 * 66,249 227 * 66,249 3,141 66,068
100 1 * 71,923 1 * 71,923 1 * 71,923 3,311 71,923
Table 4.3: Power capacity results under WR-36 with 100 cells
All turbine placements for WR-1 reach optimality using E1 in a few seconds due to the
low sparsity of the interaction matrix and the relatively small state space. Under WR-36,
the highly dense interaction matrix causes turbine placements to have a larger gap when
applying E1, thus, needing application of E2 and E3 to generate better bounds and cuts,
requiring up to several days in CPU time to compute optimal solutions. However, a large
gap still occurs while placing 20 – 40 turbines due to the relatively large complexity of the
turbine constraint polytope as characterized by the difficulty to generate effective cuts and
better bounds that well approximate the integer polytope.
As shown in Table 4.2 – 4.3 many layouts generated using MP capture a lower power
capacity compared to E1 and E1,2,3 for WR-1 and WR-36 respectively. The percentage
difference in the power capacities between MP and various branch-and-cut instances are
44
illustrated in Figure 4.5 for both wind regimes. Under WR-1, MP generates layouts with a
power capacity that are 0 – 4% less than the optimal values captured by layouts from E1,
with the largest difference occurring between layouts with demanding turbine constraint
polytopes (i.e. 20 – 50 turbines).
To better compare MP with E1, E2, and E3 for WR-36, we compute the percentage
difference between the three pairs and plot the range of values as indicated by the
minimum and maximum bars for a given number of turbines. The minimum of the range
occurs between E1 and MP, while the maximum occurs between E3 and MP because E3
has a higher cut-off and uses more computing resources to generate similar or better
layouts compared to E1. Power capacity values generated using MP are less than 0 – 3%
compared to E1, E2, and E3 for placement of all turbine values, while varying less than
0.5% between E1, E2, and E3 for any given number of turbines.
Figure 4.5: Percent difference in power capacity values between CPLEX and MP for varying turbines under WR-1 and WR-36
Even though MP generates layouts that are sub-optimal, they are consistently within
5% of the power capacity generated using state-of-the-art branch-and-cut algorithm for a
wide variety of cases with varying gaps and cut-off periods. We attribute sub-optimality to
the generation of lack of effective triplet clusters, choice of penalty constant, and local
45
numerical instability caused by locally similar wake decay values. In latter sections, we
test whether these inefficiencies of MP perform worse in comparison to branch-and-cut
when generation of cuts and bounds is further challenged when the problem’s state space
is increased by increasing discrete resolution.
Table 4.4 – 4.5 compare power capacity results from MP with EAs in [16] and [17],
and mathematical programming in [2]. MP outperforms Mosetti et al. [16] for placing 26
and 15 turbines under WR-1 and WR-36 respectively; contrarily, MP performs worse than
Grady et al. [17] for placing 30 and 39 turbines under WR-1 and WR-36 respectively.
Even though methods employed by both [16] and [17] are similar, it is postulated that
Grady et al. [17] outperforms MP due to fine-tuned GA parameters and longer run time.
Interestingly, results from branch-and-cut either outperform or match Turner et al. [2] even
when both methods have similar mathematical formulation; we speculate that this may
be due to failure to account for an appropriate angle of wake spread by Turner et al. [2].
K Model Power (W)
26
Mosetti et al. [16] 12,474
Turner et al. [2] 12,686
MILP (E1) 12,709
MP 12,486
30
Grady et al. [17] 14,410
Turner et al. [2] 14,410
MILP (E1) 14,410
MP 13,972
Table 4.4: Results comparison under WR-1 with 100 cells
K Model Power (W)
15
Mosetti et al. [16] 13,374
Turner et al. [2] 13,671
MIP (E3) 13,679
MP 13,395
39
Grady et al. [17] 32,377
Turner et al. [2] 31,947
MIP (E3) 32,818
MP 32,142
Table 4.5: Results comparison under WR-36 with 100 cells
Figure 4.6 – Figure 4.9 show the various layouts for the results presented in Table 4.4 –
4.5. In general, we observe that layouts generated using mathematical programming in
46
Turner et al. [2] and MIP (E1,2,3) yield uniformly distributed layouts, while EAs in [16] and
[17], and MP suffer from local non-uniformity.
Optimizing over several other parameters (e.g. land-use constraints, noise generation,
infrastructure costs) significantly adds to the computational complexity of the problem.
Therefore, design engineers optimize over few key variables such as wind stochasticity,
and then heuristically modify determined turbine placements to adhere to further set of
constraints and objectives. Consequently, the benefits of uniformity and non-uniformity is
highly dependent upon project specifics. For instance, while uniform layouts generated
using branch-and-cut are wake optimal, they may increase electrical cabling costs due to
the increased distance between turbines. Similarly, non-uniform layouts that are wake
sub-optimal with closer turbine proximity may have lower maintenance costs, yet increase
the concentration of noise level in certain parts of the wind farm.
Mosetti et al. Turner et al. MIP (E1) MP
Figure 4.6: Comparison of layouts for WR-1 scenario with 26 Turbines
Grady et al. Turner et al. MIP (E1) MP
Figure 4.7: Comparison of layouts for WR-1 scenario with 30 turbines
Mosetti et al. Turner et al. MIP (E3) MP
Figure 4.8: Comparison of layouts for WR-36 scenario with 15 turbines
47
Grady et al. Turner et al. MIP (E3) MP
Figure 4.9: Comparison of layouts for WR-36 scenario with 39 turbines
4.4.2 Case 2 – 400 cells
Resolution is increased by four-fold from 100 cells to 400 cells, while keeping the wind
farm area to a constant 4km2 and decreasing the square cell size to 100m x 100m.
Turbine proximity constraint is met without an addition of further constraints due to the
adequate distance between centroids of the neighbouring cells. Higher resolutions further
increases the computational complexity of finding good solutions due to an exponential
increase in the state space of the integer program. To the best of our knowledge, no other
studies were found in the literature within the realm of mathematical programming and
heuristic methods for discrete formulations that assess the effects of increasing resolution
to power capacities, layout design, and run-time for varying number of turbines.
Table 4.6 – 4.7 show the wind farm layout power capacity results for varying number
of turbines for WR-1 and WR-36 respectively. For both wind regimes we apply E1, E2,
and E3. As running branch-and-cut under using E3 requires significant computational and
memory resources, in some cases, in sufficient memory (INM) due to large problem size
causes the program to terminate without generating a layout.
Generation of triplet clusters for 400 cells is a computationally expensive task and can
take up to 3 – 4 hours. The graphical model structure based on the marginal polytope
generated using the unconstrained objective function (Eq. (4.23)) hardly changes while
optimizing for varying number of turbines using MP; thus, we pre-calculate 5,000 triplet
clusters based on 10 turbines to save time on layout generation for additional turbines.
Thus, all results presented on Table 4.6 – 4.7 using MP are based on pre-generated
clusters for individual wind regimes.
Additionally, due to the unconstrained nature of the formulation, MP simulations
sometimes produce layouts that contain one more or less turbine than needed. This
48
aberration is fixed by greedily adding or subtracting a turbine from the layout that results
in the highest power capacity.
Under WR-1, the layout power capacity increases for each set of added turbines until
it peaks for a placement of 220 turbines for both methods. Power capacity decreases at
the addition of more turbines as the densely packed turbines couple together to increase
the wake interactions between turbines and significantly decrease the effective wind
speed in the narrow horizontal wake columns. Power capacity does not peak under WR-
36 as the stronger winds allows more energy to be harnessed than the energy lost due
to wake interactions. However, under both wind regimes addition of each turbine over 20
turbines results in a lower power capacity yield per turbine compared to power capacity
yield for a lower number of turbines.
Figure 4.10 shows the percentage difference in power capacity results obtained from
the three instances of branch and cut, and MP for varying number of turbines under both
wind regimes. A positive percentage demonstrates that MP produces a higher power
capacity, while a negative percentage a lower power capacity compared to a given
instances of branch and cut and number of turbines. Under WR-1, similar to the
formulation with 100 cells, MP consistently captures layouts with a 0 – 5% lower power
capacity than E1, E2, and E3 and with under 1% variation between the three instances.
Furthermore, the largest difference occurs in formulations with 30 – 140 turbines due to
their difficult turbine constraint polytope.
Under WR-36, placing 50 – 240 turbines using MP produces a higher power capacity
than E1 and E2. Applying E3 yields tighter bounds and cuts, hence, reducing the gap
within this turbine range, which results in E3’s power capacity results that are better than
MP. The performance of MP is favorably comparable to instances where there exists a
large optimality gap in results computed using branch-and-cut. Furthermore, MP
consistently performs well as an approximate solution for instances, commonly E3, in
which branch-and-cut produces results that are optimal or have a small gap.
49
Figure 4.10: Percent difference in power capacity values between CPLEX and MP for varying turbines under WR-1 and WR-36
MIP – 1 Thread – 1 hour
cut-off (E1) MIP – 4 Threads – 1 hour
cut-off (E2) MIP – 12 Threads – 24
hour cut-off (E3) MP
K Time (s)
Gap (%)
Power (W)
Time (s)
Gap (%)
Power (W)
Time (s) Gap (%)
Power (W)
Time (s)
Power (W)
10 1 * 5,184 1 * 5,184 1 * 5,184 646 5,184
20 1 * 10,368 1 * 10,368 1 * 10,368 1,002 10,368
30 3,595 - 15,339 10,671 - 15,341 586,310 - 15,341 1,024 15,259
40 3,596 - 20,101 10,646 - 20,054 564,897 - 20,106 1,029 19,245
50 3,596 - 24,417 10,518 - 24,429 559,054 - 24,503 1,052 23,804
60 3,597 - 28,325 10,531 - 28,398 597,192 - 28,454 1,060 27,697
70 3,596 - 31,858 9,843 - 31,944 560,929 - 32,065 1,073 30,768
80 3,597 - 35,472 10,330 - 35,605 620,235 - 35,697 1,069 33,729
90 3,597 - 37,960 10,633 - 38,124 650,517 - 38,188 1,063 36,665
100 3,598 - 40,377 8,845 - 40,439 641,507 - 40,727 1,062 39,178
120 3,598 - 44,667 8,951 - 44,954 INM - - 1,085 42,852
140 3,593 19 47,484 12,385 12 47,486 INM - - 1,050 45,960
160 3,594 3 49,357 12,809 1 49,402 INM - - 1,058 47,806
180 3,594 2 50,472 12,949 1 50,428 26,962 1 50,428 1,065 48,962
200 148 1 51,282 642 1 51,288 685 1 51,308 1,089 50,118
220 378 1 52,025 2,252 1 51,250 5,192 1 51,250 1,078 51,292
240 3,595 1 50,906 3,039 1 51,116 4,033 1 50,882 1,068 50,927
260 3,597 1 50, 082 12,975 1 50, 165 23,153 1 50,207 1,063 50,150
280 3,581 1 49, 261 12,221 1 49, 310 INM - - 1,009 49,240
300 3,582 1 48,320 12,618 1 48,325 INM - - 1,033 48,326
320 3,588 1 47,277 10,946 1 47,248 INM - - 1,050 47,169
340 3,590 1 46,151 11,661 1 46,151 INM - - 983 46,156
360 3,569 1 45,013 11,339 1 45,013 618,898 1 45,013 980 45,086
380 2 1 43,820 7 1 43,820 7 1 43,820 972 43,941
400 1 * 42,660 1 * 42,660 1 * 42,660 4,506 42,660
Table 4.6: Power capacity results under WR-1 with 400 cells
50
MIP – 1 Thread – 1 hour
cut-off (E1) MIP – 4 Threads – 1 hour
cut-off (E2) MIP – 12 Threads – 24
hour cut-off (E3) MP
K Time (s)
Gap (%)
Power (W)
Time (s)
Gap (%)
Power (W)
Time (s) Gap (%)
Power (W)
Time (s)
Power (W)
10 3,435 - 9,282 10,279 - 9,300 814,609 90 9,317 2,017 9,105
20 3,598 - 17,774 10,649 - 17,906 804,187 98 17,950 2,050 16,632
30 3,590 - 24,941 10,810 - 25,220 808,427 - 26,038 2,064 25,131
40 3,598 - 32,214 10,618 - 33,040 807,081 - 33,604 2,070 32,179
50 3,598 - 38,916 10,785 - 39,201 826,492 - 40,860 2,050 39,879
60 3,598 - 46,005 11,081 - 45,682 827,122 - 47,669 2,135 46,250
70 3,598 - 51,627 10,922 - 51,838 850,897 - 54,439 2,153 54,089
80 3,594 - 58,015 10,711 - 58,157 836,268 - 61,227 2,157 59,200
90 3,596 - 64,057 10,264 - 63,905 756,351 - 65,274 2,140 65,792
100 3,596 - 69,734 10,378 - 69,730 861,872 - 72,490 2,137 71,482
120 3,596 - 80,656 11,855 - 80,618 887,169 - 86,061 2,092 83,324
140 3,594 - 90,046 11,171 - 90,384 536,166 98 92,860 2,078 95,042
160 3,597 - 101,116 7,189 - 101,116 409,573 90 104,895 2,089 105,101
180 3,593 - 91,298 7,182 - 109,116 763,095 86 114,923 2,079 114,980
200 3,591 - 119,716 6,864 97 119,716 772,600 84 124,243 2,170 123,691
220 3,587 - 109,237 10,102 79 126,958 825,105 67 132,782 2,150 131,678
240 3,591 62 135,515 10,376 61 137,218 789,276 49 141,312 2,083 139,900
260 3,582 48 143,280 10,740 41 148,203 821,311 36 148,627 2,094 147,390
280 3,586 36 150,598 9,237 27 155,605 INM - - 2,120 154,533
300 3,594 18 162,118 9,234 18 162,317 INM - - 2,038 160,593
320 3,593 11 168,355 8,537 11 168,419 INM - - 2,021 166,908
340 3,597 6 174,269 8,203 6 174,359 INM - - 2,018 173,108
360 3,588 2 179,698 6,919 2 179,703 INM - - 2,034 178,908
380 3,587 1 184,740 7,630 1 184,747 INM - - 2,016 184,414
400 1 0 189,395 1 0 189,395 1 0 189,395 4,763 189,395
Table 4.7: Power capacity results under WR-36 with 400 cells
4.4.3 Case 3 – 2,500 cells
Integer optimization using any method becomes a magnitude more challenging when the
resolution in increased twenty-five fold to 2,500 cells, while still maintaining the wind farm
area to a constant 4km2 and decreasing the square cell size to 40m x 40m. At this
resolution, placing turbines in adjacent and nearby cells violate turbine proximity
constraint, therefore, an inequality constraint is introduced such that if the distance
between the two cells is less than 5 times the rotor radius, 100m, then there can only be
a placement of 1 turbine between the two cells. Generation of effective bounds, cuts, and
triplet clusters is challenging under this case due to the demanding memory requirement
and computational complexity. Due to the large state-space we notice that we run out of
memory while iterating to tighten the relaxed marginal polytope. Therefore, to preserve
computational feasibility we decrease the cut-off time of E3 to 4 hours, and remove
51
generation of triplet clusters and employ a simple rounding scheme to construct a feasible
integral solution for MP.
Table 4.8 – 4.9 show the wind farm layout power capacity results for varying number
of turbines for WR-1 and WR-36 respectively. The 100m proximity constraint and 40m
distance between adjacent cells limits the layout feasibility to no more than approximately
280 turbines. Results show that there exists large gap for many placements after
applying E1, E2, and E3 for both wind regimes, and E3 generally produces layouts with
better power capacity. Even though the lack of triplet cluster generation for MP is
expected to produce layouts with uncertain power capacity we observe that in certain
cases power produced by MP is better than E1, E2, and E3. It is conjectured that this occurs
due to the difficulty in constructing effective bounds and cuts using CPLEX given the
limited amount of resources available and the large problem size.
In Figure 4.11 we plot the percentage difference in power capacity results obtained
from the three instances of branch and cut, and MP for varying number of turbines under
both wind regimes. Under WR-1, branch and cut algorithms determines an optimal layout
configuration for the placement of 10 – 50 turbines, while MP is only able to determine
optimal configuration for 50 turbines. Furthermore, E1, E2, and E3 produce layouts with up
to 20% higher power capacity than MP for up to 100 turbine placements. Contrarily, MP
produces layouts with up to 20% higher power capacity than branch and cut instances for
turbine placements over 100. This occurs as branch and bound is able to generate better
bounds and cuts for the turbine constraint polytope with a smaller state space due to a
lower demand on computational complexity. Furthermore, the performance
between E1, E2, and E3 varies between 0 – 5%.
Under WR-36, both algorithms produce layouts with similar power capacity for
placement of over 50 turbines. Nevertheless, MP performance is approximately better
than branch and bound for less than 50 turbines. The large gap even after applying E3 for
both wind regimes highlights the need to better assess the problem and generate further
local constraints that assist in producing problem specific bounds and cuts that are
effective in solving in the problem.
52
Figure 4.11: Percent difference in power capacity values between CPLEX and MP for varying turbines under WR-1 and WR-36
MIP – 1 Thread – 1 hour
cut-off (E1) MIP – 4 Threads – 1 hour
cut-off (E2) MIP – 12 Threads – 1
hour cut-off (E3) MP
K Time (s)
Gap (%)
Power (W)
Time (s)
Gap (%)
Power (W)
Time (s) Gap (%)
Power (W)
Time (s)
Power (W)
10 89 * 5,184 92 * 5,184 33 * 5,184 42 5,184
20 63 * 10,368 98 * 10,368 33 * 10,368 41 10,037
30 95 * 15,552 85 * 15,552 33 * 15,552 41 15,312
40 70 * 20,736 95 * 20,736 32 * 20,736 42 17,516
50 3,626 - 25,861 4,009 - 25,768 33 * 25,920 42 25,920
60 3,586 - 30,201 9,275 - 30,283 13,918 - 30,229 41 29,683
70 3,618 - 32,478 8,362 - 33,945 16,888 - 34,757 41 30,505
80 3,598 - 39,207 9,939 - 39,032 20,280 - 39,211 41 33,743
90 3,571 - 43,329 10,559 - 43,506 22,514 - 43,673 42 42,803
100 3,571 - 46,403 8,832 - 41,309 18,191 - 45,104 41 46,361
120 3,541 - 46,049 10,122 - 45,693 22,280 - 47,131 41 48,961
140 3,600 - 49,832 9,444 - 51,257 24,681 - 52,310 42 58,671
160 3,523 - 55,119 10,719 - 55,214 26,602 - 54,811 42 62,988
180 3,531 - 59,957 8,857 - 59,490 29,474 - 60,660 58 72,157
200 3,616 - 65,108 9,621 - 63,060 29,990 - 63,677 59 73,449
220 3,613 - 66,504 11,111 - 66,143 11,364 - 67,262 58 83,843
240 3,618 - 64,616 7,298 - 68,431 13,461 - 68,431 41 79,727
260 3,618 - 70,790 6,761 - 70,790 7,927 - 70,790 41 86,356
280 3,622 - 72,470 7,486 - 72,433 6,055 - 72,145 42 85,247
Table 4.8: Power capacity results under WR-1 with 2,500 cells
53
MIP – 1 Thread – 1 hour
cut-off (E1) MIP – 4 Threads – 1 hour
cut-off (E2) MIP – 12 Threads – 1
hour cut-off (E3) MP
K Time (s)
Gap (%)
Power (W)
Time (s)
Gap (%)
Power (W)
Time (s) Gap (%)
Power (W)
Time (s)
Power (W)
10 3,600 - 8,572 10,366 - 8,572 6,831 - 8,874 121 9,184
20 3,600 - 16,974 4,609 - 16,178 INM - - 122 17,562
30 3,600 - 25,075 8,538 - 24,177 INM - - 149 25,450
40 3,600 - 31,931 6,027 - 31,931 INM - - 172 32,964
50 3,600 - 39,261 10,139 - 39,021 INM - - 171 36,983
60 3,600 - 47,158 5,529 - 46,859 16,043 - 46,859 171 47,766
70 3,600 - 53,845 5,573 - 53,845 18,107 - 53,845 172 54,628
80 3,600 - 60,501 6,095 - 60,501 16,422 - 60,501 171 60,953
90 3,600 - 67,078 4,356 - 67,139 17,817 - 67,139 171 67,920
100 3,600 - 72,698 3,972 - 72,915 INM - - 171 72,429
120 3,600 - 84,469 4,183 - 84,134 16,914 - 84,134 172 85,141
140 3,600 - 95,124 3,507 - 95,188 19,700 - 95,188 124 95,520
160 3,600 - 103,911 3,989 - 105,431 16,834 - 105,431 122 104,691
180 3,600 - 113,692 4,205 - 115,251 16,154 - 115,251 121 112,329
200 3,602 - 122,406 10,279 - 124,471 17,688 - 124,471 122 123,728
220 3,607 - 132,770 9,540 - 133,254 17,411 - 133,254 122 129,204
240 3,602 - 141,592 10,960 - 141,592 18,290 - 141,592 122 141,200
260 3,614 - 149,573 11,225 - 149,573 19,454 - 149,573 121 149,426
280 3,617 - 157,275 10,590 - 157,687 18,973 - 157,687 121 156,616
Table 4.9: Power capacity results under WR-36 with 2,500 cells
Figure 4.12 shows wind farm layouts for placing 200 turbines generated under WR-1
using MIP (E1) and MP. Similar to the visibly uniform and symmetrical layouts (Figure 4.6
– Figure 4.9) generated by branch-and-cut under 100 cells, layouts generated by MIP
under 2,500 cells are also uniformly distributed and symmetrical in comparison to layouts
generated by MP as shown in Figure 4.12. Nevertheless, in Figure 4.12 layout generated
by MP yields a higher power capacity and is more structured compared to the MIP layout.
MIP (E1) MP
Figure 4.12: Comparison of layouts for WR-1 scenario with 200 turbines
54
4.4.4 Discreteness Analysis
In earlier analysis, we’ve observed that computational complexity of the integer program
increases as the wind farm resolution is increased. While choosing a lower resolution may
offer computational feasibility, the resulting turbine configurations may be highly sub-
optimal due to stricter limitations on the placement of the turbines, compared to
continuous formulations or discrete formulations with a higher resolution. Therefore,
determining the problem resolution while maintaining a handle on tractability poses a
challenge for the WFLO problem.
Figure 4.13 illustrates the power capacity for various number of turbines for three
cases (100, 400, and 2,500 cells) produced using branch-and-cut and MP algorithms
under WR-1 and WR-36 wind regimes. Please note that while most plotted power
capacity values for branch-and-cut were generated using E3, values generated using E2
were used when using E3 resulted in insufficient memory. Both algorithms, MP and
branch-and-cut, produce a higher power capacity for a higher resolution compared to a
lower resolution under WR-1. Contrarily, both algorithms produce similar power capacity
regardless of the resolution under WR-36. This illustrates the need to carefully assess the
effects of problem parameters on the effectiveness of the resolution prior to optimization.
Scalability of branch-and-cut and MP poses a challenge due to the difficulty in
generating effective cuts and triplets respectively. Nevertheless, for many turbine
placements MP’s performance is equivalent or better than E1, E2, and E3 for resolutions
with 400 and 2,500 cells under both wind regimes; interestingly, these occurrences also
produce layouts that are sub-optimal under both methods. Hence, MP provides a more
competitive and scalable approach to generating sub-optimal layouts in comparison with
exhaustive branch-and-cut while using requiring fewer computational resources.
Furthermore, sub-optimal layouts generated using MP can be augmented by feeding the
layouts to fine-tuned heuristic and evolutionary methods. Therefore, MP provides an
interesting alternative perspective to generating fast and approximate layouts to the
challenging WFLO problem.
55
Figure 4.13: Power capacity generated using E3 and MP for 100, 400, and 2,500 square cells under WR-1 and WR-36 wind regimes
4.5 Conclusions
In this thesis, we model the binary QIP formulation of the WFLO problem as an undirected
graphical model by incorporating pairwise wake interactions and global turbine budget
constraint within the graph’s edges. The linearized QIP formulation is NP-hard, which
requires an exhaustive and computationally expensive branch-and-cut algorithm to
determine optimal layouts. In many cases, branch-and-cut produces sub-optimal layouts
with a large relative gap for high discrete resolution and challenging turbine budget
constraints. We’ve demonstrated that by determining the most probable assignment of
the graph’s random variables based upon local, distributed, and dynamic probabilistic
inference using message passing algorithm we can decode good sub-optimal layouts for
a given number of turbine placements. This thesis conducts a thorough computational
study to determine the effectiveness of message passing algorithm in comparison with
branch-and-cut algorithm in generating turbine layouts under varying wind regime
complexity and problem resolution.
56
Branch-and-cut is effective at finding optimal layouts for a low resolution problem (100
cells) under both wind regimes, while in comparison, MP produces sub-optimal layouts
that are consistently within 5% of the power capacity. As both methods have difficulty
generating cuts to tighten the relaxed polytope at higher resolutions, the generated
layouts are sub-optimal and rely on locally optimal rounding schemes. At these higher
resolutions with 400 and 2,500 cells, the performance of MP is better or equivalent in
comparison to branch-and-cut to many turbine placements under WR-1 and WR-36.
Additionally, we show that MP can produce better approximate layouts in a shorter period
of time while using significantly less computing resources. Overall, MP offers a
competitive and scalable alternative to computationally expensive branch-and-cut
algorithm, especially when design engineers are seeking to generate approximate and
sub-optimal layouts quickly. We foresee that a population of approximate layouts
generated from MP can be further optimized for other project variables (e.g. land-use
constraints, noise generation, infrastructure costs) using fine-tuned stochastic and
evolutionary search methods.
Graphical models provide a very succinct way to capture the problem objectives’
structural variable dependencies. There are still considerable limitations for employing
probabilistic inference methods to approximate graphically modelled integer programs.
For instance, although current state-of-the-art message passing algorithms (i.e. TRW-S
and MPLP) generate good approximate solutions in practice, they don’t provide a
certificate of optimality. Furthermore, feasibility issues arise as multi-variable constraints
are incorporated using augmented Lagrangian method. While generating triplets tighten
the relaxed polytope, a method to generate and incorporate problem specific cuts within
the graphical model is required for message passing algorithms to become a practical
alternative approximate solver for problems with multiple constraints and objectives.
While we’ve demonstrated that WFLO using approximate probabilistic inference offers
a compelling alternative optimization technique, there are further challenging facets that
need to be addressed. More specifically, approximate algorithms that better handle
problem intractability and non-convexity need to be developed to solve comprehensive
57
multi-objective WFLO incorporating wake effects, noise level, land-use availability,
infrastructure costs, and other geographical constraints.
58
Chapter 5
Concluding Remarks
5.1 Conclusions
The principle goal of this thesis has been to explore techniques to solve the wind farm
layout optimization (WFLO) problem. We restricted the scope of the problem to optimal
placement of a limited number of wind turbines on a flat terrain in response to wake effects
generated by upstream wind turbines due to stochastic wind conditions. This restriction
allows us to model the WFLO problem as a binary discrete mathematical program using
a simple analytical wake model (Jensen’s model [3]), and motivate our study on
optimization methodologies.
In Chapter 3, we showed that the derived WFLO problem is structurally similar to the
Quadratic Knapsack Problem (QKP). Subsequently, in Chapter 4 we illustrate that while
several exact solving procedures based on linearization and generating better bounds
exist to solve quadratic integer programs, these methods don’t scale well to large-scale
optimization problems. By observing that a QKP’s objective function can be represented
as an undirected graph, we study message passing algorithms, a class of probabilistic
inference methods, that exploit the graph-theoretic structure through graph factorization
and decomposition to determine the maximum-a-posteriori (MAP) assignment of the
graph’s random variables. By devising a method to incorporate the QKP’s global linear
inequality constraint (budget constraint) into the graphical model, we show that a variant
of belief propagation (TRW-S) can be used to generate good results when applied to in-
approximable NP-Hard benchmark QKPs and the WFLO problem.
5.2 Future work
We discuss some future considerations, recommendations, and challenges of our work
within the context of WFLO, message passing inference, and graphical models.
59
5.2.1 Wind Farm Layout Optimization
Other Objectives This thesis restricts the WFLO problem to a single objective function
by only considering the optimal placement of wind turbines on a flat terrain by minimizing
their wake effects on each other. There are several other variables and constraints that
significantly affect the development and planning of wind farms that need to be
considered as part of the problem objective. We list a few notable variables that can be
considered as part of the objective function:
Noise – Wind turbines are a known cause of noise pollution, especially in cases where
turbines are placed in close proximity. Several industrial guidelines exist that constraint
noise levels in residential and industrial communities such as the ISO standard [61].
As noise level at a given location is a function of its proximity to wind turbines, noise
propagation of wind turbines must be suitably modelled and included when determining
the placement of turbines.
Infrastructure Costs – The variable cost of a wind turbine installation can be broken
down into the cost of electrical cabling required to obtain grid connectivity, maintenance
infrastructure, and the type of excavation required. These costs vary depending on the
terrain type and location. For example, although placing turbines far apart reduces their
wake interactions it may result in higher infrastructure costs due to the added distance
between them. A multi-objective function considering these costs needs to be modelled
to determine project feasibility and net-present value profitability.
Land Use Availability – Rarely is the case that the entire geographical area under
consideration of wind farm installation can be used for wind turbine installations. Land-
use restrictions also need to be added into the problem as constraints.
It is important to note that the addition of further constraints and objectives makes the
optimization function highly non-linear, non-convex, and intractable. Thus, finding the
optimal solution becomes computationally challenging. Thus, the approximate inference
algorithms we study here serve as a stepping stone towards developing advanced
60
approximate and hybrid evolutionary algorithms to generate good solutions to the multi-
objective WFLO problem.
Wake Modelling and Terrain Type As described in our earlier recommendation, the
WFLO problem is a multi-objective function with difficult constraints. While we’ve used a
simple analytical wake model (Jensen’s wake model) to model wake effects, more
accurate analytical wake models need to be developed that work well with complex
terrains and that can be pre-calculated to take advantage of discrete optimization models.
In general, complex wake models that accurately describe wake effects are
computationally exhaustive and memory intensive. Thus, this further adds to the time
complexity of the optimization problem.
Robustness As with any real-world systems, all measured cost vectors suffer from
uncertainty, which compounded over several factors and variables can have a significant
impact on the solution state space optimality, feasibility, and project reliability. Hence,
uncertainty within the problem space needs to be added, and an optimization algorithm
capable of performing robust optimization needs to be used appropriately. While robust
optimization algorithms are available in the context of exact discrete and continuous
optimization, they have not been widely studied in the context of message passing
inference. However, the probabilistic nature of graphical models make them an ideal
candidate to incorporate and study variable uncertainty.
5.2.2 Message passing algorithms for inference and optimization
Capacity Constraint We showed in Chapter 3 that a global linear inequality constraint
(capacity constraint) can be incorporated successfully into an undirected graphical model
using penalty functions derived from an augmented Lagrangian approach. Nevertheless,
this approach requires added iterations for convergence, adds sub-optimal stationary
points, and it can suffer from numerical ill-conditioning. Research in the direction of adding
capacity constraints within graphical models using auxiliary variables and higher-order
graph potentials needs to be explored.
61
Tight Relaxation of Integral Capacity Constraint As belief propagation and its variants
operate on the relaxed marginal polytope (relaxed objective function), we apply methods
in [36] and [37] that add triplet clusters to enforce local consistency based on cutting-
plane methods and Sherali-Adams hierarchy to iteratively tighten the relaxed marginal
polytope. While this approach has been shown to work in a limited capacity when applied
to the various benchmark QKPs and the WFLO problem to reduce their integrality gap,
we recommend generating and incorporating cuts (constraints) specifically to tighten
global capacity constraint to effectively further tighten the relaxed problem polytope.
Therefore, research in the direction of generating tight cover, extended cover, and lifted
cover inequalities [62] that tighten the relaxed integral capacity constraint needs to be
conducted. Furthermore, alternative approaches to incorporating several local inequality
constraints within the graphical model needs to be explored.
62
Bibliography
[1] International Energy Agency (IEA), "Energy and Climate Change," International
Energy Agency, France, 2015.
[2] S. D. O. Turner, D. A. Romero, P. Y. Zhang and T. C. Y. Chan, "A new
mathematical programming approach to optimize wind," Renewable Energy, vol.
63, pp. 674-680, 2014.
[3] N. O. Jensen, "A note on wind generator interaction," Riso National Laboratory,
Roskilde, 1983.
[4] D. Pisinger, "The quadratic knapsack problem—a survey," Discrete applied
mathematics, vol. 155, no. 5, pp. 623-648, 2007.
[5] D. J. Rader Jr and G. J. Woeginger, "The quadratic 0–1 knapsack problem with
series–parallel support," Operations Research Letters, vol. 30, no. 3, pp. 159-166,
2002.
[6] J. Pearl, Probabilistic reasoning in intelligent systems: networks of plausible
inference, Morgan Kaufmann, 2014.
[7] V. Kolmogorov, "Convergent tree-reweighted message passing for energy
minimization," Pattern Analysis and Machine Intelligence, IEEE Transactions, vol.
28.10, pp. 1568-1583, 2006.
[8] A. Globerson and T. Jaakkola, "Fixing Max-Product: Convergent Message Passing
Algorithms for MAP LP-Relaxations," in Advances in Neural Information
Processing Systems, Vancouver, 2007.
[9] J. Sun, N.-N. Zheng and H.-Y. Shum, "Stereo matching using belief propagation,"
Pattern Analysis and Machine Intelligence, IEEE Transactions, vol. 25, no. 7, pp.
787-800, 2003.
[10] P. F. Felzenszwalb and D. P. Huttenlocher, "Efficient belief propagation for early
vision," International journal of computer vision, vol. 70, no. 1, pp. 41-54, 2006.
[11] N. Lazic, Message Passing Algorithms for Facility Location Problems, University of
Toronto, 2011.
63
[12] C. Yanover and Y. Weiss, "Approximate inference and protein-folding," in
Advances in neural information processing systems, 2002.
[13] C. M. Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
[14] D. A. Sontag, "Approximate inference in graphical models using LP relaxations
(Doctoral dissertation)," Massachusetts Institute of Technology, 2010.
[15] D. F. N. Koller, Probabilistic Graphical Models, Cambridge: The MIT Press, 2009.
[16] G. P. C. D. B. Mosetti, "Optimization of wind turbine positioning in large windfarms
by means of genetic algorithm," Journal of Wind Engineering and Industrial
Aerodynamics, vol. 51, pp. 105-116, 1994.
[17] S. H. M. A. M. Grady, "Placement of wind turbines using genetic algorithms,"
Renewable Energy, vol. 30, pp. 259-270, 2005.
[18] H. Huang, "Distributed genetic algorithm for optimization of wind farm annual
profits," in Intelligent Systems Applications to Power Systems, IEEE, 2007.
[19] D. W. R. Bertsimas, Optimization over integers, Belmont: Dynamic Ideas, 2005.
[20] S. Donovan, "Wind farm optimization," in Proceedings of the 40th annual ORSNZ
conference, 2005.
[21] S. Donovan, "An improved mixed integer programming model for wind farm layout
optimisation," in Proceedings of the 41st annual conference of the Operations
Research Society, 2006.
[22] R. Archer, "Wind turbine interference in a wind farm layout optimization mixed
integer linear programming model," Wind Engineering, vol. 35, pp. 165-175, 2011.
[23] "GH WindFarmer Theory Manual," Garrad Hassan and Partners Ltd, 2009.
[24] P. Y. Zhang, D. A. Romero, C. J. Beck and C. H. Amon, "Solving wind farm layout
optimization with mixed integer programs and constraint programs," The
Association of European Operational Research Societies, vol. 2, pp. 195-219,
2014.
[25] M. Weigt and H. Zhou, "Message passing for vertex covers," Physical Review E,
vol. 74, no. 4, 2006.
[26] N. Lazic, F. J. Brendan and P. Aarabi, "Solving the uncapacitated facility location
problem using message passing algorithms," in International Conference on
Artificial Intelligence and Statistics, 2010.
64
[27] J. Yarkony and C. Fowlkes, "Covering trees and lower-bounds on quadratic
assignment," in Computer Vision and Pattern Recognition (CVPR), IEEE
Conference, 2010.
[28] C. C. Moallemi and B. Van Roy, "Resource allocation via message passing,"
INFORMS journal on Computing, vol. 23, no. 2, pp. 205-219, 2011.
[29] M. J. Wainwright, T. S. Jaakkola and A. S. Willsky, "MAP estimation via agreement
on trees: message-passing and linear programming," Information Theory, IEEE
Transactions, vol. 11, pp. 3697-3717, 2005.
[30] S. Sanghavi, D. Malioutov and A. S. Willsky, "Linear programming analysis of
loopy belief propagation for weighted matching," in Advances in neural information
processing systems, 2008.
[31] A. Z. S. Kumar, "Message-passing algorithms for quadratic programming
formulations of MAP estimation," in arXiv preprint, 2012.
[32] P. L. J. Ravikumar, "Quadratic programming relaxations for metric labeling and
markov random field map estimation," in Proceedings of the 23rd international
conference on Machine learning, 2006.
[33] D. Bertsimas and J. . N. Tsitsiklis, ntroduction to linear optimization, Belmont:
Athena Scientific, 1997.
[34] V. W. M. Kolmogorov, "On the optimality of tree-reweighted max-product message-
passing," arXiv preprint, 2012.
[35] C. Yanover, T. Meltzer and Y. Weiss, "Linear Programming Relaxations and Belief
Propagation--An Empirical Study," The Journal of Machine Learning Research,
vol. 7, pp. 1887-1907, 2006.
[36] D. M. T. G. A. W. Y. J. T. Sontag, "Tightening LP Relaxations for MAP using
Message-Passing," in Conference in Uncertainty in Artificial Intelligence, 2008.
[37] D. C. D. K. L. Y. Sontag, "Efficiently Searching for Frustrated Cycles in MAP
Inference," in Uncertainty in Artificial Intelligence, 2012.
[38] R. E. Bellman and S. E. Dreyfus, Applied dynamic programming, 1962.
[39] A. Caprara, D. Pisinger and P. Toth, "Exact solution of the quadratic knapsack
problem," INFORMS Journal on Computing, vol. 11, no. 2, pp. 125-137, 1999.
[40] C. Witzgall, "Mathematical methods of site selection for Electronic Message
Systems (EMS)," NASA STI/Recon Technical Report, vol. 76, p. 18321, 1975.
65
[41] C. E. Ferreira, A. Martin, C. C. de Souza, R. Weismantel and L. A. Wolsey,
"Formulations and valid inequalities for the node capacitated graph partitioning
problem," Mathematical Programming, vol. 74, no. 3, pp. 247-266, 1996.
[42] D. J. Laughhunn, "Quadratic binary programming with application to capital-
budgeting problems," Operations Research, vol. 18, no. 3, pp. 454-461, 1970.
[43] A. Billionnet, A. Faye and É. Soutif, "A new upper bound for the 0-1 quadratic
knapsack problem," European journal of operational research, vol. 112, no. 3, pp.
664-672, 1999.
[44] P. Michelon and L. Veilleux, "Lagrangean methods for the 0–1 quadratic knapsack
problem," European Journal of Operational Research, vol. 92, no. 2, pp. 326-341,
1996.
[45] C. Helmberg, F. Rendl and R. Weismantel, "A semidefinite programming approach
to the quadratic knapsack problem," Journal of combinatorial optimization, vol. 4,
no. 2, pp. 197-215, 2000.
[46] G. Gallo, P. l. Hammer and B. Simeone, "Quadratic knapsack problems,"
Combinatorial Optimization, pp. 132-149, 1980.
[47] A. Billionnet and F. Calmels, "Linear programming for the 0–1 quadratic knapsack
problem," European Journal of Operational Research, vol. 92, no. 2, pp. 310-325,
1996.
[48] B. A. Julstrom, "Greedy, genetic, and greedy genetic algorithms for the quadratic
knapsack problem," in Proceedings of the 7th annual conference on Genetic and
evolutionary computation, New York, 2005.
[49] D. P. Bertsekas, Constrained optimization and Lagrange multiplier methods,
Academic press, 2014.
[50] A. Billionnet and É. Soutif, "An exact method based on Lagrangian decomposition
for the 0–1 quadratic knapsack problem," European Journal of operational
research, vol. 157, no. 3, pp. 565-575, 2004.
[51] A. Jimenez, A. Crespo, E. Migoya and J. Garcia, "Advances in large-eddy
simulation of a wind turbine wake," Journal of Physics: Conference Series, vol. 75,
no. 1, p. 012041, 2007.
[52] P. Beaucage, N. Robinson, M. Brower and C. Alonge, "Overview of six commercial
and research wake models for large offshore wind farms," in Proceedings of the
European Wind Energy Associate (EWEA).
66
[53] J. S. González, A. G. G. Rodríguez, J. C. Mora, M. B. Payán and J. R. Santos,
"Overall design optimization of wind farms," Renewable Energy, vol. 36, pp. 1973 -
1982, 2011.
[54] S. Chowdhury, J. Zhang, A. Messac and L. Castillo, "Unrestricted wind farm layout
optimization (UWFLO): Investigating key factors influencing the maximum power
generation," Renewable Energy, vol. 38, pp. 16 - 30, 2012.
[55] S. Chowdhury, J. Zhang, A. Messac and L. Castillo, "Optimizing the arrangement
and the selection of turbines for wind farms subject to varying wind conditions,"
Renewable Energy, vol. 52, pp. 273 - 282, 2013.
[56] C. Wan, J. Wang, G. Yang, H. Gu and X. Zhang, "Wind farm micro-siting by
Gaussian particle swarm optimization with local search strategy," Renewable
Energy, vol. 48, pp. 276 - 286, 2012.
[57] W. P. Adams and H. D. Sherali, "A Tight Linearization and an Algorithm for Zero-
One Quadratic Programming Problems," Management Science, vol. 32, no. 10, pp.
1274 - 1290, 1986.
[58] M. O. Hansen, Aerodynamics of Wind Turbines, London: Earthscan, 2008.
[59] [Online]. Available: http://pub.ist.ac.at/~vnk/software/SRMP-v1.01.zip.
[60] [Online]. Available: http://cs.nyu.edu/~dsontag/code/mplp_ver2.tgz.
[61] International Organization for Standardization, "Acoustics -- Attenuation of sound
during propagation outdoors -- Part 2: General method of calculation (ISO 9613-
2:1996)," International Organization for Standardization, 1996.
[62] Z. Gu, G. L. Nemhauser and M. W. Savelsbergh, "Lifted cover inequalities for 0-1
integer programs: Computation," INFORMS Journal on Computing, vol. 10, no. 4,
pp. 427-437, 1998.