Energy Minimization with Label Costsand Model Fitting

presented by Yuri Boykov

co-authors:Andrew Delong Anton Osokin Hossam Isack

The University of

OntarioOverview Standard models in vision (focus on discrete case)

• MRF/CRF, weak-membrane, discontinuity-preserving...• Information-based: MDL (Zhu&Yuille’96) , AIC/BIC (Li’07)

Label costs and their optimization • LP-relaxations, heuristics, α-expansion++

Model Fitting• dealing with infinite number of labels ( PEARL )

Applications• unsupervised image segmentation• geometric model fitting (lines, circles, planes, homographies, ...)• rigid motion estimation• extensions…

The University of

Ontario

Reconstruction in Vision: (a basic example)

L

observed noisy image I image labeling L(restored intensities)

How to compute L from I ?

I

L = { L1, L2 , ... , Ln }I = { I1, I2 , ... , In }

The University of

Ontario

Energy minimization(discrete approach)

MRF framework• weak membrane model (Geman&Geman’84, Blake&Zisserman’83,87)

pL qL

ZZLp 2:

Nqp

qpp

pp LLVILE),(

2 ),()()(L ,V

T Tdiscontinuity preserving potentials

Blake&Zisserman’83,87

spatial regularizationdata fidelity

The University of

OntarioOptimization Convex regularization

• gradient descent works• exact polynomial algorithms

TV regularization• a bit harder (non-differentiable)• global minima algorithms (Ishikawa, Hochbaum, Nikolova et al.)

Robust regularization• NP-hard, many local minima• good approximations (message passing, a-expansion)

,V

,V

,V

Nqp

qpp

pp LLVILE),(

2 ),()()(L

The University of

Ontario

Potts model(piece-wise constant labeling)

Robust regularization• NP-hard, many local minima• provably good approximations (a-expansion)

,V

)(),( TwV

maxflow/mincut combinatorial algorithms

Nqp

qpp

pp LLVILE),(

2 ),()()(L

The University of

Ontario

Left eye imageRight eye image

Potts model(piece-wise constant labeling)

Robust regularization• NP-hard, many local minima• provably good approximations (a-expansion)

,V

)(),( TwV

depth layers

maxflow/mincut combinatorial algorithms

Nqp

qpp

pp LLVLDE),(

),()()(L

The University of

Ontario

Potts model(piece-wise constant labeling)

Robust regularization• NP-hard, many local minima• provably good approximations (a-expansion)

,V

)(),( TwV

0

C

1

maxflow/mincut combinatorial algorithms

Nqp

qpp

pp LLVLDE),(

),()()(L

The University of

OntarioAdding label costs

Lippert [PAMI 89]• MDL framework, annealing

Zhu and Yuille [PAMI 96]• continuous formulation (gradient des cent )

H. Li [CVPR 2007]• AIC/BIC framework, only 1st and 3rd terms• LP relaxation (no guarantees approximation)

Our new work [CVPR 2010] , extended a-expansion • all 3 terms, 3rd term is represented as some high-order clique), optimality bound• very fast heuristics for 1st & 3rd term (facility location problem, 60-es)

L

LLN)q,p(

qpp

pp )(h)L,L(V)L(D)(E LL

- set of labelsallowed at each

point p

otherwise

LLp pL ,0

:,1)(L

The University of

OntarioThe rest of the talk…

Why label costs?

The University of

OntarioModel fitting

pL

||Lp||minargL̂

}b,a{L

p

2xy )bapp(||Lp||

y=ax+b

SSD

The University of

Ontariomany outliersquadratic errors fail

use more robust error measures, e.g.

gives “MEDIAN” line|bapp|||Lp|| xy

- more expensive computations

(non-differentiable)- still fails if

outliers exceed 50%

RANSAC

The University of

Ontariomany outliers

RANSAC

1. sample randomlytwo points, get a line

The University of

Ontariomany outliers

10 inliers

RANSAC

1. sample randomlytwo points, get a line2. count inliers for

threshold T

The University of

Ontariomany outliers

30 inliers

1. sample randomlytwo points, get a line2. count inliers for

threshold T

3. repeat N times and select

model with most inliers

RANSAC

The University of

OntarioMultiple models and many outliers

Why not RANSAC

again?

The University of

OntarioMultiple models and many outliers

In general, maximization of inliersdoes not work for

outliers + multiple models

Why not RANSAC

again?

Higher noise

The University of

OntarioEnergy-based approach

p

||Lp||)L(E

energy-based interpretation of RANSAC criteria forsingle model fitting:

- find optimal label Lfor one very specific

error measure

TdistifTdistif

dist,1,0

||||

The University of

OntarioEnergy-based approach

Npq

qpp

p LLTwLpE )(||||)(L

If multiple models

- assign different models (labels Lp) to every point

p

- find optimal labelingL = { L1, L2 , ... , Ln }

Need regularization!

The University of

OntarioEnergy-based approach

Npq

qpp

p LLTwLpE )(||||)(L

If multiple models

- assign different models (labels Lp) to every point

p

- find optimal labelingL = { L1, L2 , ... , Ln }

The University of

OntarioEnergy-based approach

If multiple models

- assign different models (labels Lp) to every point

p

- find optimal labelingL = { L1, L2 , ... , Ln }

L

LLp

p )(h||Lp||)(E LL

otherwise

LLp pL ,0

:,1)(L

- set of labelsallowed at each

point p

The University of

OntarioEnergy-based approach

If multiple models

- assign different models (labels Lp) to every point

p

- find optimal labelingL = { L1, L2 , ... , Ln }

L

LLN)q,p(

qpp

p )(h)LL(Tw||Lp||)(E LL

Practical problem: number of potential labels (models) is huge, how are we going to use a-expansion?

The University of

OntarioPEARL

data points

ProposeExpandAndReestimateLabels

The University of

OntarioPEARL

data points + randomly sampled models

sample datato generatea finite set

of initial labels

ProposeExpandAndReestimateLabels

The University of

OntarioPEARL

models and inliers (labeling L)

a-expansion:minimize E(L)

segmentationfor fixed

set of labels

Npq

qpp

p )LL(Tw||Lp||)(E L

ProposeExpandAndReestimateLabels

The University of

OntarioPEARL

ProposeExpandAndReestimateLabels

models and inliers (labeling L)

reestimating labels in

for given inliers

minimizes first term

of energy E(L)

Npq

qpp

p )LL(Tw||Lp||)(E L

The University of

OntarioPEARL

ProposeExpandAndReestimateLabels

models and inliers (labeling L)

Npq

qpp

p )LL(Tw||Lp||)(E L

a-expansion:minimize E(L)

segmentationfor fixed

set of labels

The University of

OntarioPEARL

iterate until convergence

Npq

qpp

p )LL(Tw||Lp||)(E L

after 5 iterations

ProposeExpandAndReestimateLabels

The University of

Ontario

PEARL can significantlyimprove initial models

single line fittingwith 80% outliers

number of initial samplesde

viat

ion

(fr

om g

roun

d tr

uth

)

The University of

Ontario

Comparison formulti-model fitting

original data points

Low noise

The University of

Ontario

Comparison formulti-model fitting

some generalization of RANSAC

Low noise

The University of

Ontario

Comparison formulti-model fitting

PEARL

Low noise

The University of

Ontario

Comparison formulti-model fitting

original data points

High noise

The University of

Ontario

Comparison formulti-model fitting

Some generalization of RANSAC (Multi-RANSAC, Zuliani et al. ICIP’05)

High noise

The University of

Ontario

Comparison formulti-model fitting

Other generalization of RANSAC (J-linkage, Toldo & Fusiello, ECCV’08)

High noise

The University of

Ontario

Comparison formulti-model fitting

Finding modes in Hough-space, e.g. via mean-shift(also maximizes the number of inliers)

Hough transform

High noise

The University of

Ontario

Comparison formulti-model fitting

PEARL

High noise

The University of

OntarioWhat other kinds of models?

The University of

OntarioFitting circles

Here spatial regularization does not work well

regularization with label costs only

The University of

OntarioFitting planes (homographies)

Original image (one of 2 views)

The University of

OntarioFitting planes (homographies)

(a) Label costs only

The University of

OntarioFitting planes (homographies)

(b) Spatial regularity only

The University of

OntarioFitting planes (homographies)

(c) Spatial regularity + label costs

The University of

Ontario

(unsupervised) Image Segmentation

Original image

The University of

Ontario

(unsupervised) Image Segmentation

(a) Label costs only [Li, CVPR 2007]

The University of

Ontario

(unsupervised) Image Segmentation

(b) Spatial regularity only [Zabih&Kolmogorov CVPR 04]

The University of

Ontario

(unsupervised) Image Segmentation

(c) Spatial regularity + label costs

Zhu and Yuille 96used continuous

variational formulation(gradient discent)

The University of

Ontario

(unsupervised) Image Segmentation

(c) Spatial regularity + label costs

The University of

Ontario

(unsupervised) Image Segmentation

Spatial regularity + label costs

The University of

Ontario

(unsupervised) Image Segmentation

Spatial regularity + label costs

The University of

Ontario

(unsupervised) Image Segmentation

Spatial regularity + label costs

The University of

Ontario

(unsupervised) Image Segmentation

Spatial regularity + label costs

The University of

Ontario

(rigid)Motion Estimation

Original image

3 motions

[Rene Vidal]

The University of

Ontario

(rigid)Motion Estimation

(a) Label costs only

3 motions

The University of

Ontario

(rigid)Motion Estimation

(b) Spatial regularity only

7 motions

The University of

Ontario

(rigid)Motion Estimation

(c) Spatial regularity + label costs

3 motions

The University of

Ontario

(rigid)Motion Estimation

The University of

Ontario

(rigid)Motion Estimation

The University of

Ontario

(rigid)Motion Estimation

The University of

OntarioPlane fitting

The University of

OntarioPlane fitting

Note very small steps between each floor

The University of

Ontario

Affine model fitting(from a rectified stereo pair)

photoconsistency + smoothness

dense model assignments to pixels

Birchfield & Tomasi’99(fit initial models to output of other stereo

algorithm ++ α-expansion + reestimation)

geometric errors + smoothness+ label cost

sparse model assignments to features

PEARL(sample data + α-expansion + reestimation)

The University of

Ontario

Duh...use right geometric error measure!!!

“disparity” errors d1 and d2 (bad idea!)“quatient”-based errors d (standard)

The University of

Ontario

Affine model fitting(from a rectified stereo pair)

photoconsistency + smoothness

dense model assignments to pixels

Birchfield & Tomasi’99(fit initial models to output of other stereo

algorithm ++ α-expansion + reestimation)

geometric errors + smoothness+ label cost

sparse model assignments to features

PEARL(sample data + α-expansion + reestimation)

The University of

Ontario

Photoconsistency vs. Geometric Alignment

photoconsistency optimization (Birchfield & Tomasi’99)

densestereo

The University of

Ontario

Photoconsistency vs. Geometric Alignment

geometric error minimization via PEARL

sparse data

sparsestereo