p 3 & beyond solving energies with higher order cliques pushmeet kohli pawan kumar philip h. s....

P3 & BeyondSolving Energies with Higher Order Cliques

Pushmeet Kohli Pawan Kumar Philip H. S. Torr

Oxford Brookes University

CVPR 2007

Energy Functions

Observed Variables

Hidden Variables

MAP Inference

Energy Functions

MAP Inference

Energy Minimization

Energy Functions

• Pairwise Energy Functions

Unary Pairwise

Energy Functions


Unary Pairwise

Efficient Algorithms for Minimization• Message Passing (BP, TRW)• Move Making (Expansion/Swap)

Energy Functions


Unary Pairwise

Efficient Algorithms for Minimization• Message Passing (BP, TRW)• Move Making (Expansion/Swap)

Restricted Expressive Power!

Energy Functions

• Higher Order Energy Functions

Unary Pairwise Higher order

More expressive than pairwise

FOE: Field of Experts(Roth & Black CVPR05)

Energy Functions



• Computationally expensive to minimize!• Exponential Complexity O(LN)

• L = Number of Labels• N = Size of Clique

Minimizing Higher Order Energies

Efficient BP in Higher Order MRFs(Lan, Roth, Huttenlocher & Black, ECCV 06)

• 2x2 clique potentials for Image Denoising• Searched a restricted state space• 16 minutes per iteration

Pairwise MRF Higher order MRFNoisy Image

Energy Functions



• Our Method• Move making algorithm

• Can handle cliques of thousand of variables

• Extremely Efficient ( works in seconds)

Talk Outline

• Move making Algorithms

• Solvable Higher Order Potentials

• Moves for the PN Potts Model

• Application: Texture Segmentation

Move Making Algorithms

Solution Space

En

erg

y

Move Making Algorithms

Search Neighbourhood

Current Solution

Optimal Move

Solution Space

En

erg

y

Computing the Optimal Move


Current Solution

Optimal Move

x

E(x)

xc Transformation function

T

EmMove Energy(t)

Em(t) = E(T(xc, t))

T(xc, t) = xn = xc + t



Current Solution

Optimal Move

E(x)


T

EmMove Energy(t)

x

Em(t) = E(T(xc, t))

T(xc, t) = xn = xc + t



Current Solution

Optimal Move

E(x)

xc

Em(t) = E(T(xc, t))

Transformation function

T

EmMove Energy

T(xc, t) = xn = xc + t minimize

t*Optimal Move

(t)

x



Current Solution

Optimal Move

E(x)


T

EmMove Energy(t)

x

Key Characteristic: Search Neighbourhood

Bigger the better!

Moves using Graph Cuts

Expansion and Swap Move Algorithm

[Boykov, Veksler, Zabih]• Exponential Move Search Space (Good )• Move encoded by binary vector t• Move Energy

Moves using Graph Cuts

Expansion and Swap Move Algorithm

[Boykov, Veksler, Zabih]• Exponential Move Search Space (Good )• Move encoded by binary vector t• Move Energy

• Optimal move t* in polynomial time

Submodular

Expansion Move

Expansion Transformation• Variables take label or retain current label

• Optimal move can be computed for pairwise potentials which are metric.

[Boykov, Veksler, Zabih]

Expansion Move

Sky

House

Tree

GroundInitialize with Tree

Status:

Expand GroundExpand HouseExpand Sky


Swap Move

• Optimal move can be computed for pairwise potentials which are semi-metric.

- Swap Transformation

• Variables labeled can swap their labels


Swap Move

Sky

House

Tree

GroundSwap Sky, House


Moves for Higher Order Potentials

• Question you should be asking:

Can my higher order potential be solved using α-expansions?



• Show that move energy is submodular for all xc




• Show that move energy is submodular for all xc


Not an easy thing to do!

• Form of the Higher Order Potentials


Clique Inconsistency function:

Pairwise potential:

xi

xj

xk

xm xl

cSum Form

Max Form

Theoretical Results: Swap

• Move energy is always submodular if

non-decreasing concave.

See paper for proofs

Theoretical Results: Expansion

• Move energy is always submodular if

increasing linear

See paper for proofs

PN Potts Model

PN Potts Model

c

PN Potts Model

c Cost : red

PN Potts Model

c Cost : max

Optimal moves for PN Potts

• Computing the optimal swap move

c

Label 3

Label 4

Label 1Label 2



c

Label 3

Label 4

Case 1

Not all variables assigned label 1 or 2

Label 1Label 2



c

Label 3

Label 4

Case 1

Not all variables assigned label 1 or 2

Move Energy is independent of tc and can be ignored.

Label 1Label 2



c

Label 1Label 2 Label 3

Label 4

Case 2

All variables assigned label 1 or 2



c

Label 3

Label 4

Case 2

All variables assigned label 1 or 2

Can be minimized by solving a st-mincut problem

Label 1Label 2

Solving the Move Energy

Add a constant

This transformation does not effect the solution



Source

Sink

v1 v2 vn

Ms

Mt

ti = 0 vi Source Set

tj = 1 vj Sink Set



Case 1: all xi = (vi Source)

Cost:

Source

Sink

v1 v2 vn

Ms

Mt



v1 v2 vn

Ms

MtCost:

Source

Sink

Case 2: all xi = (vi Sink)



Cost:

v1 v2 vn

Ms

Mt

Source

Sink

Case 3: all xi = (vi Source, Sink)


• The expansion move energy

• Similar graph construction.

See paper for details

Experimental Results

Unary(Colour)

Pairwise(Smoothness)

Higher Order(Texture)

Original Image

• Texture Segmentation



Unary(Colour)



Colour HistogramUnary Cost: Tree



Unary(Colour)



Edge Sensitive Smoothness Cost



Unary(Colour)



Expansion Solution

Higher Order Texture Potentials

Patch Dictionary (Tree)

5x5 patches

Higher Order Texture Potentials

Patch Dictionary (Tree)

G (c,ps): L1 distance between patch ps and

pixel set



Unary(Colour)



Original Pairwise Higher order


Original Swap (3.2 sec)

Expansion (2.5 sec)

Pairwise Higher Order

Swap (4.2 sec)

Expansion (3.0 sec)


Original

Pairwise Higher Order

Swap (4.7 sec)

Expansion (3.7sec)

Swap (5.0 sec)

Expansion (4.4 sec)

Conclusions & Future Work

• Efficient minimization of certain higher order energies

• Can handle very large cliques

• Allows more expressive functions

• Explore other interesting family of potential functions

Thanks

• Questions?

p 3 & beyond solving energies with higher order cliques pushmeet kohli pawan kumar philip h. s....

Documents

energy optimal

x c t

energy tx c

zabih slide

tt x slide

expansionswap slide

seconds slide

energy tt e