heuristic algorithms giovanni righini - unimi.it › righini › didattica › algoritmieuristici...

Combinatorial optimization

Combinatorial optimization problemsHeuristic Algorithms

Giovanni Righini

University of Milan

Department of Computer Science (Crema)


Optimization

In general an optimization problem can be formulated as:

minimize z(x)

subject to x ∈ X

where

• x is a vector of variables;

• z(x) is the objective function;

• X is the feasible region, i.e. the set of solutions satisfying theconstraints.

A solution is an assignment of values to the variables.



In general a combinatorial optimization problem can be formulated as:

minimize z(x)

subject to x ∈ {0, 1}|E |

The feasible region X is defined as a subset of the set of all possiblesubsets of a given ground set (!).

Let’s say it again with an example...


Combinatorial optimization: example

Ground set E : the set of edges of a given graph G.

All possible subsets of E are the 2|E | subsets of edges.

Only a subset X of them are, for instance, spanning trees.

The condition x ∈ X describes the feasible region of any probleminvolving the search of an optimal spanning tree.

Although the ground set E is rather small, the number of its subsets isexponential in its cardinality (2|E |) and hence the number of solutionscan be very large.

Even restricting the search to feasible solutions, the cardinality of Xcan be combinatorial: it grows as a combinatorial number when |E |grows.


The combinatorial problems structure

The variables, the constraints and the objective function of acombinatorial optimization problem define its “combinatorialstructure”. This is a semi-informal way to indicate the maincharacteristics of the problem, that affect the effectiveness of differentsolution procedures.

The analysis of the combinatorial structure of any given problemgives useful indications on the most suitable algorithm to solve it.

Moreover it can uncover similarities between seemingly differentproblems.


Problems on weighted sets: the knapsack problem (KP)

From a ground set of items, select a subset such that the value of theselected items is maximum and their weight does not exceed a givencapacity. We are given

• a ground set N of items• a weight function a : N → N

• a capacity b ∈ N

• a value function c : N → N

We can associate a binary variable x with each element of the groundset: the solution space is {0, 1}n, where n = |N|. In this way everysolution corresponds to a subset and to its binary characteristicvector x .The feasible region X contains the subsets with total weight not largerthan b:

X = {x ∈ {0, 1}n :∑

j∈N

ajxj ≤ b}

The objective is the maximization of the total value:

max z(x) =∑

j∈N

cjxj .


Example

D

B

AF

C

E

Knapsack

N A B C D E Fc 7 2 4 5 4 1a 5 3 2 3 1 1

b = 8

x ′ = (0, 0, 1, 1, 1, 0, 0) ∈ X x ′′ = (1, 0, 1, 1, 0, 0, 0) 6∈ Xz(x ′) = 13 z(x ′′) = 16

DC

E

Knapsack

D

A

C

Knapsack


Problems on subsets with a metric: the Max Diversity Problem(MDP)

Given a ground set of items, we want to select a subset of givencardinality, maximizing a measure of the pairwise distances betweenthe selected items.We are given

• a ground set N,• a distance function d : N × N → N,• a positive integer number k ∈ {1, . . . , |N|}.

A possible choice of the variables is analogous to the previous case:x is the binary characteristic vector of the selected subset.

The feasible region X contains all the subsets of cardinality k :

X = {x ∈ {0, 1}n :∑

j∈N

xj = k}.

The objective is to maximize the sum of the pairwise distances:

max z(x) =∑

i,j∈N

dijxixj .

It is a quadratic function.


Example

A

B

F

D

C

E

G

x ′ = (0, 0, 1, 1, 1, 0, 0) ∈ X x ′′ = (1, 0, 1, 0, 0, 0, 1) ∈ Xz(x ′) = 24 z(x ′′) = 46

A

B

F

D

C

E

G

A

B

F

D

C

E

G


Additive and non-additive objective functions

In general the objective function associates rational or integer valueswith (feasible) subsets of the ground set.

z : X → N

Computing its value can be more or less difficult.

• The KP has an additive (linear) objective function: its value is thesum of the values of another value function c whose domain isthe ground set:

c : N → N

• The MDP has a non-additive (quadratic) objective-function.

Both of them are easy to compute, but the additive objective functionof the KP is easier to update when an item is inserted or deleted fromthe solution. It is enough to

• add cj for each inserted item j ∈ N;

• subtract cj for each deleted item j ∈ N.

For the non-additive objective function of the MDP this is not true.


Set partitioning problems: the Bin Packing Problem (BPP)

A set of weighted items must be partitioned into the minimum numberof subsets, so that the total weight of each subset is within a givencapacity.We are given:

• a set N of items,• a set M of bins,• a weight function a : N → N,• a capacity b of the bins.

The ground set of the problem contains all the item-bin pairs.

E = N × M

A solution is represented by nm binary variables with two indicesi ∈ N and j ∈ M.The feasible region contains the partitions of the items, complyingwith the capacity constraints:

X = {x ∈ {0, 1}nm :∑

j∈M

xij = 1 ∀i ∈ N,∑

i∈N

aixij ≤ b ∀j ∈ M}.


Set partitioning problems: the Bin Packing Problem (BPP)

The objective is to minimize the number of bins used.

min z(x) = |{j ∈ M :∑

i∈N

xij > 0}|.

To get rid of the function “cardinality of”, we need additional binaryvariables: the characteristic vector y of the bin subset.We obtain:

X = {(x , y) ∈ {0, 1}nm+m :∑

j∈M

xij = 1 ∀i ∈ N,∑

i∈N

ai xij ≤ byj ∀j ∈ M}.

min z(y) =∑

j∈M

yj .


Example

S′ ={

(A, 1) , (B, 1) , (C, 2) , (D, 2) , (E , 2) , (F , 3) ,(G, 4) , (H, 5) , (I, 5)

}

∈ X

y ′ = (1, 1, 1, 1, 1)z(y ′) = 5

S′′ ={

(A, 1) , (B, 1) , (C, 2) , (D, 2) , (E , 2) , (F , 3) ,(G, 4) , (H, 1) , (I, 4)

}

6∈ X

y ′′ = (1, 1, 1, 1, 0)z(y ′′) = 4


Set partitioning problems: the Parallel Machine SchedulingProblem (PMSP)

A set of indivisible jobs of given duration must be assigned to a set ofmachines, minimizing the overall completion time. We are given:

• a set N of jobs,• a set M of machines,• a processing time function p : N → N.

The ground set contains all job-machine pairs. We can use the samevariable choice as for the BPP. The feasible region X contains the

partitions of N into subsets.

X = {x ∈ {0, 1}nm :∑

j∈M

xij = 1 ∀i ∈ N.}

The objective function is the minimization of the maximum working

time among all machines.

min z(x) = maxj∈M

∑

i∈N

pixij .


Example

N = {L1, L2, L3, L4, L5, L6}

M = {M1,M2,M3}

Job L1 L2 L3 L4 L5 L6p 80 40 20 30 15 80

L1

L2

L5

L3 L4

M1

M2

95

M3 L6

S′ ={

(L1,M1) , (L2,M2) , (L3,M2) ,(L4,M2) , (L5,M1)

}

∈ X

z(x ′) = 95

L1 L2

L5L3 L4

M1

M2

120

M3 L6

S′′ ={

(L1,M1) , (L2,M1) , (L3,M2) ,(L4,M2) , (L5,M2)

}

∈ X

z(x ′′) = 120


Sensitive and insensitive objective functions

The objective functions of the BPP and the PMSP

• are not additive,

• are not easy to compute.

Small changes in the solution x may have different impact on theobjective function value:

• variation equal to the duration of the changed job (e.g. L5 onM1);

• no variation (e.g., L5 on M3);

• intermediate variation (e.g., L2 on M2).

This is because the effect of the change depends both

• on the modified elements,

• on the non-modified elements.

In both problems the objective function is “flat”: many differentfeasible solutions have the same value.


Problems on matrices: the Set Covering Problem (SCP)

Given a binary matrix and a vector of costs associated with thecolumns, select a minimum cost subset of columns covering all therows.We are given:

• a binary matrix a ∈ Bm,n with a set R or m rows and a set C of n

columns,• a cost function c : C → N.

A column j ∈ C covers a row i ∈ R if and only if aij = 1. The ground

set is the set of columns C. The feasible region contains the subsets

of columns that cover all the rows.

X = {x ∈ {0, 1}n :∑

j∈C

aijxj ≥ 1 ∀i ∈ R}.

The objective is to minimize the total cost of the selected columns:

min z(x) =∑

j∈C

cjxj .


Example

c 4 6 10 14 5 6

0 1 1 1 1 00 0 1 1 0 0

a 1 1 0 0 0 10 0 0 1 1 11 1 1 0 1 0

0 1 1 1 1 0 20 0 1 1 0 0 1

a 1 1 0 0 0 1 10 0 0 1 1 1 11 1 1 0 1 0 3

x ′ = (1, 0, 1, 0, 1, 0) ∈ X

z(x ′) = 19

0 1 1 1 1 0 10 0 1 1 0 0 0

a 1 1 0 0 0 1 20 0 0 1 1 1 21 1 1 0 1 0 2

x ′′ = (1, 0, 0, 0, 1, 1) 6∈ X

z(x ′′) = 15


The feasibility test

In a heuristic algorithm the following sub-problem may often occur:Given a solution x , is it feasible or not? x ∈ X? This is a decision

problem. The feasibility test may require

• an instant check on a single number(e.g. the total volume in the KP, the cardinality of the subset inthe MDP)

• a quick scan of some attributes of the solution (e.g. exactly onemachine for each job PMSP)

• the computation and check of many different values(e.g. the volume in each bin in the BPP).

The time required may be different according to the feasibility testbeing done

• on a generic solution x ;

• on a solution x ′ obtained by a slight modification of a feasiblesolution x .


Problems on matrices: the Set Packing Problem

Given a binary matrix and a weight vector associated with thecolumns, select a maximum weight subset of columns with noconflicts.We are given:

• a binary matrix a ∈ Bm,n with a set R of m rows and a set C of n

columns;• a weight function w : C → N.

Two columns j ′, j ′′ ∈ C are in conflict if and only if there is at least arow i ∈ R such that aij′ = aij′′ = 1. The ground set is the set of

columns C. The feasible region contains the subsets of columns with

no conflicts.

X = {x ∈ {0, 1}n :∑

j∈N

aijxj ≤ 1 ∀i ∈ R}.

The objective is to maximize the total weight of the selected columns.

max z(x) =∑

j∈N

wjxj


Example

w 4 6 10 14 5 6

0 1 0 0 1 00 0 1 1 0 0

a 1 0 0 0 0 10 0 0 1 1 11 1 1 0 0 0

0 1 0 0 1 0 10 0 1 1 0 0 1

a 1 0 0 0 0 1 00 0 0 1 1 1 11 1 1 0 0 0 1

x ′ = (0, 1, 0, 1, 0, 0) ∈ X

z(x ′) = 20

0 1 0 0 1 0 20 0 1 1 0 0 0

a 1 0 0 0 0 1 10 0 0 1 1 1 21 1 1 0 0 0 1

x ′′ = (1, 0, 0, 0, 1, 1) 6∈ X

z(x ′′) = 15


Problems on matrices: the Set Partitioning Problem (SPP)

Given a binary matrix and a cost vector associated with the columns,select a minimum subset of columns covering all the rows and with noconflicts.

• a binary matrix a ∈ Bm,n with a set R of m rows and a set C of n

columns;

• a cost function c : C → N.

The ground set is the set of columns C. The feasible region contains

the subsets of columns covering all the rows with no conflicts.

X = {x ∈ {0, 1}n :∑

j∈N

aijxj = 1 ∀i ∈ R}.

The objective is to minimize the total cost of the selected columns.

min z(x) =∑

j∈N

cjxj .


Example

c 4 6 10 14 5 6

0 1 0 0 1 00 0 1 1 0 0

a 1 0 0 0 0 10 0 0 1 1 01 1 1 0 0 0

0 1 0 0 1 0 10 0 1 1 0 0 1

a 1 0 0 0 0 1 10 0 0 1 1 0 11 1 1 0 0 0 1

x ′ = (0, 1, 0, 1, 0, 1) ∈ X

z(x ′) = 20

0 1 0 0 1 0 20 0 1 1 0 0 0

a 1 0 0 0 0 1 10 0 0 1 1 0 11 1 1 0 0 0 1

x ′′ = (1, 0, 0, 0, 1, 1) /∈ X

z(x ′′) = 15


The search for feasible solutions

In a heuristic algorithm the following sub-problem may occur: Find a

feasible solution x ∈ X . This is a search problem. For some problems

the solution is trivial.

• in the KP the empty subset is feasible;

• in the MDP, any subset of k elements is feasible;

• in the SCP, the whole column set is feasible (or no feasiblesolution exists);

• in other problem it is enough to respect easy consistencyconstraints.

But in some cases the search for a feasible solution is difficult.

• in the BPP it is not obvious how many bins are needed (an upperbound is the number of items);

• in the SPP no polynomial-time algorithm is known to provide afeasible solution.


Graph optimization problems. the Vertex Cover Problem (VCP)

Given a graph G = (V ,E), select a minimum cardinality vertex subsetsuch that every edge is incident to it.We are given a graph G(V ,E), where:

• V is the set of vertices, of cardinality n;

• E is the set of edges.

The ground set is V (we have a binary variable for each vertex).

The feasible region contains all vertex subsets covering the edges:

X = {x ∈ {0, 1}n : xi + xj ≥ 1∀[i, j] ∈ E}

The objective is to minimize the number of selected vertices:

min z(x) =∑

i∈V

xi


Example

A B C

D E F G

H

A B C

D E F G

H

x ′ = (0, 1, 0, 1, 1, 1, 1, 0) ∈ X

z(x ′) = 5

A B C

D E F G

H

x ′′ = (1, 0, 1, 0, 0, 0, 0, 1) /∈ X

z(x ′′) = 3


Graph optimization problems: the Max Clique Problem (MCP)

Given a graph with weights associated with the vertices, select amaximum weight vertex subset such that all its elements are adjacentto each other.We are given:

• a graph G = (V ,E) with n vertices;• a weight function w : V → N.

The ground set is the vertex set V .

The feasible region contains the cliques (complete vertex subsets, i.e.vertex subsets containing all possible edges):

X = {x ∈ {0, 1}n : xi + xj ≤ 1∀[i, j] 6∈ E}.

The objective is to maximize the weight of the selected subset:

max z(x) =∑

i∈V

wixi .


Example

A B C

D E F G

H

Uniform weight: wi = 1 for each i ∈ V

A B C

D E F G

H

x ′ = (0, 1, 1, 0, 0, 1, 1) ∈ X

z(x ′) = 4

A B C

D E F G

H

x ′′ = (1, 0, 0, 1, 1, 0, 0) ∈ X

z(x ′′) = 3


Graph optimization problems:the Max Independent Set Problem (MISP)

Given a graph with weights associated with the vertices, select amaximum weight vertex subset such that all its elements are notadjacent to each other.We are given:

• a graph G = (V ,E) with n vertices;• a weight function w : V → N.

The ground set is the vertex set V .

The feasible region contains the independent sets (empty vertexsubsets, i.e. vertex subsets containing no edges):

X = {x ∈ {0, 1}n : xi + xj ≤ 1∀[i, j] ∈ E}.

The objective is to maximize the weight of the selected subset:

max z(x) =∑

i∈V

wixi .


Example

A B C

DE F

G

H

A B C

DE F

G

H

x ′ = (0, 1, 1, 0, 0, 1, 1) ∈ X

z(x ′) = 4

A B C

DE F

G

H

x ′′ = (1, 0, 0, 1, 1, 0, 0) ∈ X

z(x ′′) = 3


Relationships between problems (1)

Every instance of the MCP is equivalent to an instance of the MISPdefined on the complementary graph.

A B C

D E F G

H

A B C

DE F

G

H

A B C

DE F

G

H

A B C

D E F G

H



The VCP and the SCP are also linked:every VCP instance can be translated into a SCP instance:

• each edge of the graph in the VCP corresponds to a row of thematrix in the SCP;

• each vertex of the graph in the VCP corresponds to a column ofthe matrix in the SCP;

• if and only if an edge e is incident to a vertex v , then aev = 1 (twoentries are equal to 1 in each row).

The optimal solution of the SCP corresponds to the optimal solutionof the VCP.


A B C

D E F G

H

A B C D E F G H(A,D) 1 0 0 1 0 0 0 0(A,E) 1 0 0 0 1 0 0 0(B,C) 0 1 1 0 0 0 0 0(B,F ) 0 1 0 0 0 1 0 0(B,G) 0 1 0 0 0 0 1 0(C,F ) 0 0 1 0 0 1 0 0(C,G) 0 0 1 0 0 0 1 0(D, E) 0 0 0 1 1 0 0 0(D, H) 0 0 0 1 0 0 0 1(F ,G) 0 0 0 0 0 1 1 0(F ,H) 0 0 0 0 0 1 0 1

The converse is not possible in general.



The BPP and the PMSP are also equivalent, but the correspondenceis more complex:

• jobs in the PMSP correspond to items in the BPP;• machines in the PMSP correspond to bins in the BPP, but

• in the BPP the capacity is given and the number of bins isminimized;

• in the PMSP the number of machines is given and the completiontime is minimized.

To find the minimum number of bins of the BPP

1. set a tentative value k ;

2. define the corresponding instance of the PMSP (with kmachines);

3. find the minimum completion time t ;• if t is larger than the bin capacity of the BPP, increase k and repeat;• if t is not larger than the bin capacity of the BPP, decrease k and

repeat.


L1

L2

L5

L3 L4

M1

M2

95

M3 L6

L1

L2

L5

L3

L4

M1

M2

80

M3 L6

M4

The inverse procedure is also possible. The two problems are

equivalent but the transformation implies that one of them be solvedrepeatedly.


Graph optimization problems:the (Asymmetric) Traveling Salesman Problem (ATSP)

Given a digraph with costs on the arcs, find a minimum costHamiltonian circuit.We are given:

• a digraph G = (N,A);• a cost function c : A → N

The ground set is the arc set A (we use a binary variable per eacharc).

The feasible region contains the Hamiltonian circuitsHow can we describe such a feasible region?

How can we modify a feasible solution into another feasible solution?Is it always possible to find a Hamiltonian circuit?

The objective is to minimize the total cost of the selected arcs:

min z(x) =∑

(i,j)∈A

cijxij


Example

6 7 8

2 3 4 5

1

6 7 8

2 3 4 5

1

C′ ={

(1, 4) , (4, 5) , (5, 8) , (8, 7) ,(7, 6) , (6, 2) , (2, 3) , (3, 1)

}

∈ X

z(x ′) = 102

6 7 8

2 3 4 5

1

C′′ ={

(4, 5) , (5, 8) , (8, 7) , (7, 4) ,(1, 2) , (2, 3) , (3, 6) , (6, 1)

}

/∈ X

z(x ′′) = 106


Graph optimization problems:the (Asymmetric) Vehicle Routing Problem (AVRP)

Given a digraph with costs on the arcs, a depot node, a demand foreach node and a capacity, find a minimum cost subset of circuits suchthat they cover all the nodes, they all include the depot and the totaldemand in each of them does not exceed the capacity. We are given:

• a digraph G = (N,A),

• a depot node d ∈ N,

• a cost function c : A → N,

• a demand function w : N → N,

• a capacity W ∈ N.


The ground set E can be

• the arc set A;

• the set of possible pairs (node,circuit).

The feasible region could contain

• all subsets of arcs satisfying the constraints (The feasibility testwould require to visit a graph.)

• all partitions of the non-depot nodes into subsets of limitedweight (demand) that can be visited along a circuit also includingthe depot (Difficult sub-problem (ATSP)!)

The objective is to minimize the total cost of the selected arcs:

min z(x) =∑

(i,j)∈A

cjxij

This is the expression of z with A as a ground set.


Example

6 7 8

2 3 4 5

d = 1

The solutions could be represented as:

6 7 8

2 3 4 5

d = 1

• subsets of arcsS =

{

(d , 2) , (2, 3) , (3, 6) , (6, d) , (d , 4) ,(4, 5) , (5, 8) , (8, 7) , (7, d)

}

∈ X

• partitions of nodesS =

{

(2, 1) , (3, 1) , (6, 1) ,(4, 2) , (5, 2) , (7, 2) , (8, 2)

}

∈ X

In both cases z(x) = 133.


Combining alternative formulations

The AVRP (as many other combinatorial optimization problems)exhibits an important feature: different definitions of the ground setare preferable for different purposes.

• When the ground set is the arc set• it is easy to evaluate the objective function;• it is difficult to test the feasibility;

• When the ground set is made by the (node,circuit) pairs• it is easy to test the feasibility;• it is difficult to evaluate the objective.

The same holds when we update/modify a solution.

Which formulation should we adopt?

• The one that makes the most frequent operations more efficient.

• Both of them, with the additional task of keeping them consistent.

heuristic algorithms giovanni righini - unimi.it › righini › didattica › algoritmieuristici...

Documents