running time analyses of benson type algorithms with an...

Running time analyses of Benson type algorithmswith an application to multiobjectivecombinatorial optimization problems

Fritz Bokler, Petra Mutzel

Chair of Algorithm Engineering, TU Dortmund, Germany

September 12, 2014

1 of 38

Outline

1 Model of Computation

2 Multiobjective Linear Optimization

3 Benson’s Algorithm

4 Multiobjective Combinatorial Optimization

5 Experiments

2 of 38

Enumeration Complexity: Motivation

MOCO Intractability

Long known: Size of Pareto-frontier might be exponential

No polynomial time algorithm

Important: In the worst case!

And in practice?

Smoothed Analysis: E (|XE |) ∈ O(n2dΦd) [Brunsch, Roglin]Φ is a perturbation parameter

Algorithm wanted: Fast if YN is small, not too slow if YN islarge

3 of 38

Enumeration Complexity: Introduction

Definition (Enumeration Problem)

Consist of

Set of instances I ⊆ Σ∗

Mapping C : I 7→ 2Σ∗

x ∈ I , y ∈ C(x) : |y | ∈ O(poly(|x |))

Goal: Enumerate C(x) for instance x ∈ I .

Observation

|C(x)| ∈ Ω(2|x |) possible

Polynomial Total Time (PTT), a.k.a.: Output PolynomialTime

Enumerate C(x) in time poly(|x |, |C(x)|) 4 of 38

Enumeration Complexity: Delay

Delay [Johnson, Yannakakis, Papadimitriou 1988]

The k-th delay is the time between outputting the k-th and(k + 1)-th element.

0-th delay: Time before outputting the first element,

|C (x)|-th delay: Time after outputting the last element untiltermination

Polynomial Time Delay (PTD)

Enumerate C(x) such that every delay is at most poly(|x |)

Incremental Polynomial Time (IncP)

Enumerate C(x) such that the k-th delay is at most poly(|x |, k)

5 of 38

Enumeration Complexity: MOO Negative Results

Multiobjective Linear Programming

By [Khachiyan, Boros, Borys, Elbassioni, Gurvich 08]:P 6= NP⇒ cannot enumerate all Pareto-optimal basic feasiblesolutions of MOLP in PTT

Multiobjective Shortest Path

P 6= NP⇒ MO-SP /∈ PTT even for d = 2 and on outerplanargraphs

And the decision variant?

There exist MOCO P, such that PDec ∈ NPC, but P ∈ IncP

6 of 38

Outline

5 Experiments

7 of 38

Multiobjective Linear Program

Definition (MOLP)

Matrices A ∈ Rm×n, C ∈ Rd×n and b ∈ Rm

pmin Cx

Ax ≥ b

Possible Goals: Finding Pareto-optimal basic feasible solutions,finding a representation of the Pareto-surface

pminCx | Ax ≥ b

8 of 38

Multiobjective Linear Program

Definition (MOLP)

Matrices A ∈ Qm×n, C ∈ Qd×n and b ∈ Qm

pmin Cx

Ax ≥ b

Possible Goals: Finding Pareto-optimal basic feasible solutions,finding a representation of the Pareto-surface

pminCx | Ax ≥ b

8 of 38

Is it really so hard?

Reminder

By [Khachiyan, Boros, Borys, Elbassioni, Gurvich 08]:P 6= NP⇒ cannot enumerate all Pareto-optimal basic feasiblesolutions of MOLP in PTT

More important in practice

Solutions x1, x2 with Cx1 = Cx2 are indistinguishable

Wanted: One solution for each nondominated extreme pointof P := Cx | Ax ≥ b+ Rd

Our Question

Can we enumerate vertP in polynomial total time?

9 of 38

Previous Work

Exact MOLP Solver

Multiobjective Simplex, e.g., [Evans, Steuer 1973, Zionts,Wallenius 1980, Steuer 1985, Armand 1993, ...]

Objective space methods, e.g., [Dauer, Liu 1990, Dauer, Saleh1990, Dauer 1993, Benson 1998, ...]

Rarely: Theoretical running time guarantees

Theoretical Running Time Guarantees

Okamoto and Uno 2007

Output: All efficient bfsAlgorithm based on reverse search [Avis, Fukuda 92]Polynomial delay for nondegenerate MOLPs

10 of 38

Contribution

MOLP: Benson’s Algorithm and Dual Variant

For fixed number of objectives: Prove PTT for Benson’s andthe Dual Algorithm

Removing exponential overhead on both algorithms, provingIncP for the improved Dual Algorithm

Running time in # vertices and # facets of P

Application for dual algorithm: Enumerating extreme pointsof MOCO Problems

Experiments to show practicability on multiobjectiveassignment and spanning tree problems

First experiments for d > 4

11 of 38

Contribution

11 of 38

Contribution

11 of 38

Contribution

11 of 38

Contribution

11 of 38

Contribution

11 of 38

Contribution

First experiments for d > 411 of 38

Outline

5 Experiments

12 of 38

Benson’s Algorithm

Very brief History

Proposed by [Benson 1998]

Further improvements, e.g., [Burton and Ozlen 2010, Hamel,Lohne, Rudloff 2013, Csirmaz 2013]

Major recap with geometric duality theory and a dual variant[Ehrgott, Lohne, Shao 2012]

Starts with polyhedron containing PIn each iteration: Finds supporting hyperplane to PNecessary assumption: Exists y ∈ Rd : P ⊆ Rd

≥ + y

13 of 38

Very brief History

Proposed by [Benson 1998]

Further improvements, e.g., [Burton and Ozlen 2010, Hamel,Lohne, Rudloff 2013, Csirmaz 2013]

Major recap with geometric duality theory and a dual variant[Ehrgott, Lohne, Shao 2012]

Starts with polyhedron containing PIn each iteration: Finds supporting hyperplane to PNecessary assumption: Exists y ∈ Rd : P ⊆ Rd

≥ + y

13 of 38

Example

14 of 38

Example

14 of 38

Example

14 of 38

Example

14 of 38

Example

14 of 38

Example

14 of 38

Example

14 of 38

Example

14 of 38

Example

14 of 38

Example

14 of 38

Example

14 of 38

Example

14 of 38

Example

14 of 38

Example

14 of 38

Running Time: Subproblems

Subproblems

LP Oracles: Finding ideal point, finding point at boundary,finding new hyperplane

A special case of vertex enumeration

Vertex Enumeration

Have to find new extreme points of intermediate polyhedra

Employing one Double Description (DD) step[Motzkin, Raiffa, Thompson, Thrall 53, Fukuda, Prodon 96, ..]

P is of dimension d

Running time for one iteration in O(dHV 3)

Using upper bound theorem: dHO(d)

15 of 38

LP Oracles

Example: P2(y)

minz ∈ R | x ∈ Rn,Ax ≥ b,Cx − 1z ≤ y

Solving with the Ellipsoid Method—Major Problem: How large cany become in the process of the algorithm?

Lemma (Encoding Length of Intermediate Extreme Points)

For each intermediate extreme point y : |yi | ∈ O(poly(n, L))

Theorem (Running Time: Benson’s Algorithm)

Let ve : # vertices of P, d fixed, running time:O(ve

Θ(d) poly(n,m, L) + veΘ(d2))

16 of 38

LP Oracles

Example: P2(y)

16 of 38

LP Oracles

Example: P2(y)

16 of 38

Finding Facets

Drawback

Example in [Ehrgott, Lohne, Shao 2012]: Supportinghyperplanes might support in face of dimension < d − 1

Exponentially many redundant inequalities

Redundant DD steps

Redundant intermediate extreme points

Nonredundancy: Speed up adjacency checks

Nice to have: Running time in number of extreme points andfacets of P

Can we find facets only?

17 of 38

Finding Facets

Drawback

Redundant DD steps

17 of 38

Finding Facets

Drawback

Redundant DD steps

17 of 38

Improved Algorithm: Finding Facets

Theorem (Finding Facets)

By solving a lexicographic variant of hyperplane finding LP, weensure that for a point y at the boundary of P the hyperplanesupports P in a facet containing y .

Theorem (Running Time w.r.t. vertices and facets)

Let ve : # vertices, vf : # facets of P, d fixedImproved version’s running time:O((ve + vf ) poly(n,m, L) + vf

Θ(d))

18 of 38

Improved Algorithm: Finding Facets

By solving a lexicographic variant of hyperplane finding LP, weensure that for a point y at the boundary of P the hyperplanesupports P in a facet containing y .

Theorem (Running Time w.r.t. vertices and facets)

Let ve : # vertices, vf : # facets of P, d fixedImproved version’s running time:O((ve + vf ) poly(n,m, L) + vf

Θ(d))

18 of 38

A Dual Variant of Benson’s Algorithm

History

Geometric duality in MOLP from [Heyde and Lohne 2008]

Algorithm proposed in [Ehrgott, Lohne, Shao 2007/2012]

Dual Variant of Benson’s Algorithm

Pretty much the same strategy

Different polyhedron: D,“dual” to P

Theorem (Running Time)

Let ve : # vertices of P, d fixedO(ve

19 of 38

History

19 of 38

History

19 of 38

Encoding Lengths

LP Oracle

P1(`) : min`TCx | x ∈ Rn,Ax ≥ b

Again: How large can ` become?

Lemma (Encoding Length of Intermediate dual ExtremePoints)

For each intermediate extreme point v : |vi | ∈ O(poly(n, L))

20 of 38

Improved Algorithm

By solving a lexicographic variant of the single objective LP, weensure that for an optimal point y of D the hyperplane supports Din a facet containing y .

Theorem (Running Time w.r.t. vertices and facets of P)

Let ve : # vertices, vf : # facets of P, d fixedImproved version’s running time:O((ve + vf ) poly(n,m, L) + ve

Θ(d))

Theorem (Running Time w.r.t. vertices P)

Let ve # vertices of P, d fixedImproved version’s running time: O(ve

Θ(d) poly(n,m, L) + veΘ(d))

21 of 38

Improved Algorithm

By solving a lexicographic variant of the single objective LP, weensure that for an optimal point y of D the hyperplane supports Din a facet containing y .

Theorem (Running Time w.r.t. vertices and facets of P)

Let ve : # vertices, vf : # facets of P, d fixedImproved version’s running time:O((ve + vf ) poly(n,m, L) + ve

Θ(d))

Theorem (Running Time w.r.t. vertices P)

Let ve # vertices of P, d fixedImproved version’s running time: O(ve

Θ(d) poly(n,m, L) + veΘ(d))

21 of 38

And considering the delay?

Original Algorithm

Number of iterations: number of faces/facets

Might take many iterations until new extreme point is found

Theorem (Delay of the Dual Algorithm)

Let’s fix d , k is number of extreme points so far,using the improved algorithm we get a delay of:

O(kΘ(d) poly(n,m, L))

22 of 38

Outline

5 Experiments

23 of 38

Multiobjective Combinatorial Optimization

Definition (Instance of a MOCO Problem)

An instance of a MOCO problem consists of

a finite base set A,

a set of feasible solutions S ⊆ 2A and

a cost function c : A 7→ Rd .

Implicit cost function for x ∈ S : c(x) :=∑

ω∈x c(ω)

Definition (MOCO Problem)

For each non-dominated y ∈ Y := c(S): find a solution x ∈ Ssuch that c(x) = y

24 of 38

Multiobjective Combinatorial Optimization

Definition (Instance of a MOCO Problem)

An instance of a MOCO problem consists of

a finite base set A,

a set of feasible solutions S ⊆ 2A and

a cost function c : A 7→ Rd .

Implicit cost function for x ∈ S : c(x) :=∑

ω∈x c(ω)

Definition (MOCO Problem)

For each non-dominated y ∈ Y := c(S): find a solution x ∈ Ssuch that c(x) = y

24 of 38

Finding Extreme Points of the Pareto-Frontier

Previous work for d > 2

Przybylski, Gandibleux, Ehrgott 2010 (PGE10)

Huge improvements for d = 3Experiments for d = 3No performance guarantees

Ozpeynirci and Koksalan 2010 (OK10)

Experiments for d ∈ 3, 4No performance guarantees

Polynomial delay for finding supported spanning treesNo experiments

25 of 38

Finding Extreme Points of the Pareto-Frontier

Previous work for d > 2

Przybylski, Gandibleux, Ehrgott 2010 (PGE10)

Huge improvements for d = 3Experiments for d = 3No performance guarantees

Ozpeynirci and Koksalan 2010 (OK10)

Experiments for d ∈ 3, 4No performance guarantees

Polynomial delay for finding supported spanning treesNo experiments

25 of 38

Why Dual of Benson’s Algorithm?

Relation MOCO and MOLP

For instance of MOCO exists MOLP s.t. YX = vertPP is always bounded from below

We do not have to construct the MOLP explicitly

We only have to solve oracle problems:

of the single objective version (P1)of a lexicographic version (lex-P)

of the problem

Lexicographic Optimization

Usually not much harder than single objective optimization

e.g., Matroids, Shortest Path, Assignments, ...

26 of 38

Why Dual of Benson’s Algorithm?

Relation MOCO and MOLP

For instance of MOCO exists MOLP s.t. YX = vertPP is always bounded from below

We do not have to construct the MOLP explicitly

We only have to solve oracle problems:

of the single objective version (P1)of a lexicographic version (lex-P)

of the problem

Lexicographic Optimization

Usually not much harder than single objective optimization

e.g., Matroids, Shortest Path, Assignments, ...

26 of 38

Results

Theorem (Dual Algorithm for MOCO)

For MOCO O, fixed d and P1 ∈ P: YX of O can be enumerated inpolynomial total time

Theorem (Improved Dual Algorithm for MOCO)

For MOCO O, fixed d and lex-P1 ∈ P: YX of O can beenumerated in incremental polynomial time

27 of 38

Outline

5 Experiments

28 of 38

Experiment Overview

Implemented combinatorial dual algorithm in C++

DD Method:

For d ∈ 3, 4: DD algorithm with adjacency informationFor d > 4: CDD library by K. Fukuda

29 of 38

MO Assignment Problem Experiments

General

Benchmark problem used in many experiments before

Complete bipartite graph, U ∪ W = V , |U| = |W |We draw weights i.i.d. from 0, . . . , 20For each number of nodes: 10 independent instances

For d = 3: Instances from PGE10

lex-P Solver

Using Hungarian algorithm to find lexicographic minimalassignments

30 of 38

MO Assignment Problem Experiments

General

Benchmark problem used in many experiments before

Complete bipartite graph, U ∪ W = V , |U| = |W |We draw weights i.i.d. from 0, . . . , 20For each number of nodes: 10 independent instances

For d = 3: Instances from PGE10

lex-P Solver

Using Hungarian algorithm to find lexicographic minimalassignments

30 of 38

Comparison - Outline

CPU: P4 EE 3.73 GHz,Mem: 4 GB

Does not filternon-extreme points

Numerical problems forlarger n

CPU: Pentium M 1.6 Ghz,Mem: 256MB

Not same instances,similarly generated

Our implementation

CPU: Core i7-3770 3.4 GHz, Mem: 16 GB

Did not find numerical problems

31 of 38

Comparison: d = 3

20 40 60 80 100

Number of Nodes

PGE10OK10Dual Benson

Summary

PGE10 faster

OK10 for n = 60: 418.84s

Does OK10 suffer fromCPU only?

Reimplementation needed

32 of 38

Comparison: d = 3

20 40 60 80 100

Number of Nodes

Summary

PGE10 faster

OK10 for n = 60: 418.84s

32 of 38

Comparison: d = 3

20 40 60 80 100

Number of Nodes

Summary

PGE10 faster

OK10 for n = 60: 418.84s

32 of 38

Comparison: d = 3

20 40 60 80 100

Number of Nodes

Summary

PGE10 faster

OK10 for n = 60: 418.84s

32 of 38

Assignment Instances, d = 4

20 30 40 50 60 70 80 90

Number of Nodes

0 5000 10000 15000

Number of Extreme Points

n = 20: Mean running times: OK10: 18.58s, Dual Benson: 0.12s33 of 38

Assignment Instances, d = 5

16 20 24 28 32 36

Number of Nodes

0 1000 2000 3000 4000

34 of 38

MO Spanning Tree Problem Experiments

Input Graphs

Using graphs similar to [Johnson, Minkoff, Phillips 2000]

MST on points in the plane, parameter to adjust density

Instances here: mean density of 2

First objective function: Euclidean distance

Other objective functions: i.i.d. from 1, . . . , 10030 instances per number of nodes

lex-P Solver

Using Kruskal’s algorithm with lexicographic sorting, UNION FINDdata structure, path compression

35 of 38

MO Spanning Tree Problem Experiments

Input Graphs

Using graphs similar to [Johnson, Minkoff, Phillips 2000]

MST on points in the plane, parameter to adjust density

Instances here: mean density of 2

First objective function: Euclidean distance

Other objective functions: i.i.d. from 1, . . . , 10030 instances per number of nodes

lex-P Solver

Using Kruskal’s algorithm with lexicographic sorting, UNION FINDdata structure, path compression

35 of 38

Spanning Tree Instances, d = 3

30 50 70 90 110 130 150

Number of Nodes

0 2000 4000 6000 8000

36 of 38

Spanning Tree Instances, d = 5

8 10 12 14 16 18 20 22 24

Number of Nodes

0 2000 4000 6000 8000

37 of 38

Summary

General

Enumerating (parts of) the Pareto-frontier is an enumerationproblem

Fixed d : Benson’s Algorithm and Dual Variant run in PTT

Improvements through finding of facets, Dual Variant in IncP,Speedup when using suggested improvements

Fixed d : If P1 ∈ P (lex-P1 ∈ P): Finding of YX in PTT(IncP)

Dual algorithm is practically capable of finding YX for MOCOs38 of 38

Sorting Gap

200 400 600 800

Spanning Tree Instances, d=3, a=.75

Number of Edges

1 of 2

Hungarian Algorithm for Lex-AP

C ∈ Rd×n

lexmin Cx

i ∈ [n] :∑j∈[n]

xij = 1

j ∈ [n] :∑i∈[n]

xij = 1

xij ≥ 0

lexmax2n∑i=0

(i , j) ∈ V ×W : Ui + Uj ≤lex Cij

U ∈ Rd×2n

2 of 2

running time analyses of benson type algorithms with an...

Documents

interviewing mana 5341 dr. george benson benson@uta.edu

benson boilers

benson parish neighbourhood plan - southoxon.gov.uk...

internal and external recruiting mana 4328 dr. george benson...

benson proofs

analyses of the resources system selection algorithms for...

analyses of the resources system selection algorithms for

intent to reissue air emission permit to benson power...

page 1 of 92ndpembin/heritage_89.pdfbennett, joe 182...

benson work

stephen benson

clustering algorithms to make sense of microarray data: ...

wall benson

benson presentation

selection decisions mana 5341 dr. george benson...

theory of evolutionary algorithms · evolutionary...

benson boiler

application of data mining algorithms in atmospheric...

abstract algorithms and complexity...

algorithm and experiment design with heuristiclab · –...