Multiplicative Boundsfor Metric Labeling

M. Pawan KumarÉcole Centrale Paris

École des Ponts ParisTechINRIA Saclay, Île-de-France

Joint work with Phil Torr, Daphne Koller

Energy Minimization

Variables V = { V1, V2, …, Vn}

Energy Minimization

Variables V = { V1, V2, …, Vn}

Energy Minimization

Va Vb

Labels L = { l1, l2, …, lh}Variables V = { V1, V2, …, Vn}

Labeling f: { 1, 2, …, n} {1, 2, …, h}

E(f) = Σa θa(f(a)) + Σ(a,b) θab(f(a),f(b))minf

θa(f(a))

θb(f(b))θab(f(a),f(b))

Energy Minimization

Va Vb

E(f)minf

NP hard

= Σa θa(f(a)) + Σ(a,b) θab(f(a),f(b))

Metric Labeling

Va Vb

E(f)minf = Σa θa(f(a)) + Σ(a,b) θab(f(a),f(b))

Metric Labeling

Va Vb

sab is non-negative d(.) is a metric distance function

Low-level vision applications NP hard

E(f)minf = Σa θa(f(a)) + Σ(a,b) sabd(f(a),f(b))

Minka. Expectation Propagation for Approximate Bayesian Inference, UAI, 2001 Murphy et al. Loopy Belief Propagation: An Empirical Study, UAI, 1999 Winn et al. Variational Message Passing, JMLR, 2005 Yedidia et al. Generalized Belief Propagation, NIPS, 2001 Besag. On the Statistical Analysis of Dirty Pictures, JRSS, 1986 Boykov et al. Fast Approximate Energy Minimization via Graph Cuts, PAMI, 2001 Komodakis et al. Fast, Approximately Optimal Solutions for Single and Dynamic MRFs, CVPR, 2007 Lempitsky et al. Fusion Moves for Markov Random Field Optimization, PAMI, 2010 Chekuri et al. Approximation Algorithms for Metric Labeling, SODA, 2001 Goemans et al. Improved Approximate Algorithms for Maximum-Cut, JACM, 1995 Muramatsu et al. A New SOCP Relaxation for Max-Cut, JORJ, 2003 Ravikumar et al. QP Relaxations for Metric Labeling, ICML, 2006 Alahari et al. Dynamic Hybrid Algorithms for MAP Inference, PAMI 2010 Kohli et al. On Partial Optimality in Multilabel MRFs, ICML, 2008 Rother et al. Optimizing Binary MRFs via Extended Roof Duality, CVPR, 2007

.

.

.

Approximate Algorithms

Multiplicative Boundsf*: Optimal Labeling f: Estimated Labeling

Σa θa(f(a)) + Σ(a,b) sabd(f(a),f(b))

Σa θa(f*(a)) + Σ(a,b) sabd(f*(a),f*(b))

≥

Multiplicative Boundsf*: Optimal Labeling f: Estimated Labeling

≤M

Σa θa(f(a)) + Σ(a,b) sabd(f(a),f(b))

Σa θa(f*(a)) + Σ(a,b) sabd(f*(a),f*(b))

Outline• Convex Relaxations

• Move-Making Algorithms

• Comparison

• Move-Making for Metric Labeling

• Move-Making for Truncated Convex Models

Integer Linear Program

Number of facets grows exponentially in problem size

Minimize a linear function over a set of feasible solutions

Linear Programming Relaxation

Schlesinger, 1976; Chekuri et al., 2001; Wainwright et al., 2003

Conic Programming Relaxation

Muramatsu and Suzuki, 2003; Ravikumar and Lafferty, 2006

Convex Relaxations

TimeLP

1976

SOCP

2003

QP

2006

Tigh

tnes

s

Expected

Analyzed

Kumar, Kolmogorov and Torr, NIPS 2007

Outline• Convex Relaxations

• Move-Making Algorithms

• Comparison

• Move-Making for Metric Labeling

• Move-Making for Truncated Convex Models

Move-Making AlgorithmsSpace of All Labelings

f

Expansion Algorithm

Initialize labeling f = f0 (say f0(a) = 1, for all Va)

For α = 1, 2, … , h

End

fα = argminf’ E(f’)

s.t. f’(a) {f(a)} U {lα}

Update f = fα

Boykov, Veksler and Zabih, 2001

Repeat until

convergence

Expansion AlgorithmVariables take label lα or retain current label

Slide courtesy Pushmeet Kohli

Expansion Algorithm

SkyHouse

TreeGround

Initialize with TreeStatus: Expand GroundExpand HouseExpand Sky

Slide courtesy Pushmeet Kohli

Variables take label lα or retain current label

Outline• Convex Relaxations

• Move-Making Algorithms

• Comparison

• Move-Making for Metric Labeling

• Move-Making for Truncated Convex Models

Multiplicative Bounds

Expansion LP

Potts 2M 2

dmax = maxi≠ kd(i,k) dmin = mini≠ kd(i,k)

M = dmax dmin

Multiplicative Bounds

Expansion LP

Potts 2 2

Metric 2M O(log h)

h = number of putative labels

Multiplicative Bounds

Expansion LP

Potts 2 2

Metric 2M O(log h)

TruncatedLinear

2M 2 + √2

TruncatedQuadratic

2M O(√M)

Outline• Convex Relaxations

• Move-Making Algorithms

• Comparison

• Move-Making for Metric Labeling

• Move-Making for Truncated Convex Models

Kumar and Koller, UAI 2009

Expansion Algorithm

Initialize labeling f = f0 (say f0(a) = 1, for all Va)

For α = 1, 2, … , h

End

fα = argminf’ E(f’)

s.t. f’(a) {f(a)} U {α}

Update f = fα

Repeat until

convergence

Boykov, Veksler and Zabih, 2001

Modified Expansion Algorithm

Initialize labeling f = f0 (say f0(a) = 1, for all Va)

For α = 1, 2, … , h

End

fα = argminf’ E(f’)

s.t. f’(a) {f(a)} U {α}

Update f = fα

Repeat until

convergence

Modified Expansion Algorithm

Initialize labeling f = f0 (say f0(a) = 1, for all Va)

For α = 1, 2, … , h

End

fα = argminf’ E(f’)

s.t. f’(a) {f(a)} U {Mα(a)}

Update f = fα

Repeat until

convergence

Any label Mα(a) instead of the same label α

Modified Expansion Algorithm

Multiplicative Bound = 2

dmax = maxi,k d(i,k)

dmax dmin

i = Mα(a), k = Mβ(b), α ≠ β

dmin = mini,k d(i,k) i = Mα(a), k = Mβ(b), α ≠ β

Outline• Convex Relaxations

• Move-Making Algorithms

• Comparison

• Move-Making for Metric Labeling• Visualizing Metrics• Uniform Metric• Hierarchically Separated Tree (HST) Metrics• General Metrics

• Move-Making for Truncated Convex Models

Visualizing Metrics

l5

l1l2

l4l3

w1w2

w3

w4

w5

w6

w7 w9w8

d( i , j ) : shortest path defined by the graph

Visualizing Metrics

l5

l1l2

l4l3

1 15

1

1

11

1 32

d( i , j ) : shortest path defined by the graph

Visualizing Metrics

l5

l1l2

l4l3

1 15

1

1

11

1 32

d( i , j ) : shortest path defined by the graph

d(1,4) = 3

Visualizing Metrics

l5

l1l2

l4l3

1 15

1

1

11

1 32

d( i , j ) : shortest path defined by the graph

d(1,2) = 5

Outline• Convex Relaxations

• Move-Making Algorithms

• Comparison

• Move-Making for Metric Labeling• Visualizing Metrics• Uniform Metric• Hierarchically Separated Tree (HST) Metrics• General Metrics

• Move-Making for Truncated Convex Models

Uniform Metric

l1 l2 l3

w w w

Modified Expansion Algorithm

Initialize labeling f = f0 (say f0(a) = 1, for all Va)

For α = 1, 2, … , h

End

fα = argminf’ E(f’)

s.t. f’(a) {f(a)} U {Mα(a)}

Update f = fα

Repeat until

convergence

Any label Mα(a) instead of the same label α

Uniform Metric

l1 l2 l3

w w w

Mα(a) = lα for all random variables Va

Multiplicative Bound = 2

dmax = maxi,k d(i,k)

dmax dmin

i = Mα(a), k = Mβ(b), α ≠ β

dmin = mini,k d(i,k) i = Mα(a), k = Mβ(b), α ≠ β

Uniform Metric

l1 l2 l3

w w w

Mα(a) = lα for all random variables Va

Multiplicative Bound = 2

dmax = 2w

dmax dmin

dmin = mini,k d(i,k) i = Mα(a), k = Mβ(b), α ≠ β

Uniform Metric

l1 l2 l3

w w w

Mα(a) = lα for all random variables Va

Multiplicative Bound = 2

dmax = 2w

dmax dmin

dmin = 2w

Uniform Metric

l1 l2 l3

w w w

Mα(a) = lα for all random variables Va

Multiplicative Bound = 2

dmax = 2w

dmin = 2w

Same bound as LP

Outline• Convex Relaxations

• Move-Making Algorithms

• Comparison

• Move-Making for Metric Labeling• Visualizing Metrics• Uniform Metric• Hierarchically Separated Tree (HST) Metrics• General Metrics

• Move-Making for Truncated Convex Models

HST Metric

w1

l1 l2 l3 l4 l5 l6 l7 l8 l9

w1w1

w2 w2 w3 w3

w4

w5 w5 w6 w6 w7 w7 w8 w8

Graph is a Tree. Labels are leaves

HST Metric

w1

l1 l2 l3 l4 l5 l6 l7 l8 l9

w1w1

w2 w2 w3 w3

w4

w5 w5 w6 w6 w7 w7 w8 w8

Edge lengths for all children are the same

HST Metric

w1

l1 l2 l3 l4 l5 l6 l7 l8 l9

w1w1

w2 w2 w3 w3

w4

w5 w5 w6 w6 w7 w7 w8 w8

Edge lengths decrease from root to leaf by factor r ≥ 2

HST Metric

w1

l1 l2 l3 l4 l5 l6 l7 l8 l9

w1w1

w2 w2 w3 w3

w4

w5 w5 w6 w6 w7 w7 w8 w8

w2 ≤ w1/r w3 ≤ w1/r w4 ≤ w1/r

HST Metric

w1

l1 l2 l3 l4 l5 l6 l7 l8 l9

w1w1

w2 w2 w3 w3

w4

w5 w5 w6 w6 w7 w7 w8 w8

w5 ≤ w2/r w6 ≤ w2/r w7 ≤ w3/r w8 ≤ w3/r

Metric Labeling for 3-level HST

l1 l2 l3 l4 l5 l6 l7 l8 l9

w1 w1 w1

w2 w2 w2 w3 w3 w3 w4 w4 w4

Metric Labeling for 3-level HST

l1 l2 l3 l4 l5 l6 l7 l8 l9

w1 w1 w1

w2 w2 w2 w3 w3 w3 w4 w4 w4

Mα(a) = lα

Metric Labeling for 3-level HST

l1 l2 l3 l4 l5 l6 l7 l8 l9

w1 w1 w1

w2 w2 w2 w3 w3 w3 w4 w4 w4

f1

Multiplicative bound of 2 for (a,b) where f*(a), f*(b) {1,2,3}

Metric Labeling for 3-level HST

l1 l2 l3 l4 l5 l6 l7 l8 l9

w1 w1 w1

w2 w2 w2 w3 w3 w3 w4 w4 w4

f1

Multiplicative bound of 2 for (a,b) where f*(a), f*(b) {4,5,6}

f2

Metric Labeling for 3-level HST

l1 l2 l3 l4 l5 l6 l7 l8 l9

w1 w1 w1

w2 w2 w2 w3 w3 w3 w4 w4 w4

f1

Multiplicative bound of 2 for (a,b) where f*(a), f*(b) {7,8,9}

f2 f3

Metric Labeling for 3-level HST

l1 l2 l3 l4 l5 l6 l7 l8 l9

w1 w1 w1

w2 w2 w2 w3 w3 w3 w4 w4 w4

f1 f2 f3

M1(a) = f1(a) M2(a) = f2(a) M3(a) = f3(a)

Metric Labeling for 3-level HST

l1 l2 l3 l4 l5 l6 l7 l8 l9

w1 w1 w1

w2 w2 w2 w3 w3 w3 w4 w4 w4

f1 f2 f3

Multiplicative bound of at most 2(1+1/r)

Metric Labeling for 4-level HST

Multiplicative bound of at most 2(1+1/r + 1/r2)

w1

l1 l2 l3 l4 l5 l6 l7 l8 l9

w1w1

w2 w2 w3 w3

w4

w5 w5 w6 w6 w7 w7 w8 w8

Metric Labeling for L-level HST

Multiplicative bound = 2(1+1/r + … + 1/rL-2)

< 2r/(r-1)

Constant bound for all HST metrics

Outline• Convex Relaxations

• Move-Making Algorithms

• Comparison

• Move-Making for Metric Labeling• Visualizing Metrics• Uniform Metric• Hierarchically Separated Tree (HST) Metrics• General Metrics

• Move-Making for Truncated Convex Models

General Metrics

Any metric can be approximated using a mixture ofO(h log h) HST metrics with distortion O(log h).

Fakcharoenphol, Rao and Talwar, 2003

ΣT ρ(T)dT(i,j)

ΣT ρ(T) = 1

d(i,j) ≤ ≤ O(log h) d(i,j)

Metric Labeling for General Metrics

Approximate d using ΣT ρ(T)dT

Solve metric labeling for each HST dT

Pick the labeling with the lowest energy

Multiplicative bound = 2r/(r-1)*O(log h) = O(log h)

Same bound as LP

Outline• Convex Relaxations

• Move-Making Algorithms

• Comparison

• Move-Making for Metric Labeling

• Move-Making for Truncated Convex Models

Kumar and Torr, NIPS 2008

Truncated Convex Models

Va Vb

sab is non-negative d(.) is a convex function

E(f)minf = Σa θa(f(a)) + Σ(a,b) sab min{d(f(a),f(b)), M}

Truncated Convex Models

sab is non-negative d(.) is a convex function

Low-level vision applications NP hard

E(f)minf = Σa θa(f(a)) + Σ(a,b) sab min{d(f(a),f(b)), M}

Truncated Linear Truncated Quadratic

Modified Expansion Algorithm

Initialize labeling f = f0 (say f0(a) = 1, for all Va)

For α = 1, 2, … , h

End

fα = argminf’ E(f’)

s.t. f’(a) {f(a)} U {Mα(a)}

Update f = fα

Repeat until

convergence

Any label Mα(a) instead of the same label lα

Modified Expansion Algorithm

Initialize labeling f = f0 (say f0(a) = 1, for all Va)

For α = 1, 2, … , h

End

fα = argminf’ E(f’)

s.t. f’(a) {f(a)} U {Mα(a)}

Update f = fα

Repeat until

convergence

Any interval of labels Mα(a) instead of a single label

Modified Expansion Algorithm

Initialize labeling f = f0 (say f0(a) = 1, for all Va)

For α = 1, 2, … , h

End

fα = argminf’ E(f’)

s.t. f’(a) {f(a)} U {α, α+1, …, α+L}

Update f = fα

Repeat until

convergence

Any interval of labels Mα(a) instead of a single label

Length of the Interval

Submodular problem

Exact optimization in polynomial time

Small interval implies bigger ratio of dmax/dmin

Length of the Interval

Non-submodular problem

Length of the Interval

Submodular problem

Optimization of upper bound in polynomial time

Big interval implies smaller ratio of dmax/dmin

Length of the IntervalTruncated Linear

Bound = 2 + max 2M , L

L M

L = √2M Bound = 2 + √2

Truncated Quadratic

Bound = O(√M)L = √M

Conclusion

MoveMaking

LP

Potts 2 2

Metric O(log h) O(log h)

TruncatedLinear

2 + √2 2 + √2

TruncatedQuadratic

O(√M) O(√M)

Future Work

Move-making and convex relaxations connection

Better measures than multiplicative bounds

Designing hybrid algorithms

Questions?

http://cvc.centrale-ponts.fr/personnel/pawan