fast global stereo matching via energy pyramid minimization

ISPRS – PCV 2014

Fast global matching via energy pyramid (disparity esAmaAon) Zurich, 9/5/2014 Bruno Conejo, Phd student ([email protected]) with S. Leprince, F. Ayoub & JP. Avouac (GPS, Caltech)

B.Conejo -‐ Fast global Matching via Energy Pyramid 2

Disparity is inversely propor?onal to depth!

Epipolar geometry stereo-‐imaging setup Introduc?on:

Reference Image Target Image

Disparity map

Given a stereo-‐pair of images (Ir ,It) how to retrieve the most probable disparity map d*?

Regulariza?on: priors on disparity

Matching: encourages similarity

In term of probability, we need to es?mate the Maximum A Posteriori (MAP) of:


Modeling: a bayesian approach

Gibbs measure relates probability density func?on to energy:

Energy of configura?on x

Normalizing constant

Reference Image: Ir Target Image: It

From the Gibbs measure we relates probabili?es to the energies (EM , ER , E):

Matching: Similarity criteria (L1, L2, ZNCC, ...)

Regulariza?on: Piecewise constant prior:

Modulated by radiometric discon?nuity:


Modeling: con?nuous Condi?onal Random Field (CRF)

First order Condi?onal Random Field (CRF):

p q

Associated graph

Reference Image

Set of nodes Set of edges

We need to globally op?mize a con?nuous CRF over all possible disparity maps (D):

However, this is a non-‐convex problem: varia?onal approaches can not work!


Modeling: non-‐convexity

Many local minima!

Solu%on: Restrict d to take value in a finite discrete set, i.e., the “search space” encoded by a label space. This leads to globally op?mize a first order discrete CRF (s?ll NP-‐Hard) : -‐  Message passing (quadra?c w.r.t search space): Loopy BP, TRW-‐S, DD-‐MRF, … -‐  Making move (linear w.r.t search space) : α-‐exp, β-‐swap, Fast-‐PD, …

B.Conejo -‐ Fast global Matching via Energy Pyramid

Discrete op?miza?on

6

Pairwise term: Encodes prior (regulariza?on) Unary term: Encodes similarity (matching) Label space: Encodes for each node all poten?al disparity to evaluate

Mul?-‐scale approaches

We work with large images (30,000 by 30,000) and we have large disparity range (-‐300,300). A direct approach is inefficient (even impossible) and unnecessary! Locally the disparity range is “small”. We can use a mul?-‐scale approach: -‐  Coarsest scales: “large” dispari?es with low

spa?al frequencies (natural topography).

-‐  Finest scales: “small” dispari?es with high spa?al frequencies (man made objects).

Two mul?-‐scale schemes are possible: -‐  Image pyramid (classic, GM-‐IP algorithm). -‐  Energy pyramid (ours, GM-‐EP algorithm).


Reference Image

Associated disparity


Mul?-‐scale: GM-‐IP algorithm (Image Pyramid)

Build & Opt. CRF

Build & Opt. CRF

Ir It Algorithm 1)  Build pyramid of image for each image by itera?ve downsampling 2)  Compute and op?mize CRF at coarsest scale 3)  Define new search space around current solu?on 4)  Repeat (2-‐3) un?l finest scale


Mul?-‐scale: GM-‐EP algorithm (Energy Pyramid)

Op?mize CRF

Op?mize CRF

Energy of CRF

◊©◊©◊

◊©◊©◊

Algorithm: 1)  Compute CRF at finest scale 2)  Build energy pyramid by

itera?ve downsampling 3)  Op?mize CRF at coarsest

scale 4)  Define new search space

around current solu?on 5)  Repeat (3-‐4) un?l finest scale


Mul?-‐scale: CRF sparsifica?on

Label before op?m. Label amer op?m. Label space to explore Removed label range


Mul?-‐scale: GM-‐IP(Image Pyramid) vs GM-‐EP (Energy Pyramid)

The image pyramid yields a smoothed representa?on of the energy and destroys local minimums, especially at coarse scale:

Different minima! Energy pyramid Image pyramid


Stereo-‐imaging in urban context: (Reference Image)


Stereo-‐imaging in urban context: (Target Image)


Stereo-‐imaging in urban context: Area of interest in reference image

Baseline = no pyramid (direct op@miza@on)

Quan?ta?ve comparison between algorithms:

Unary terms: ZNCC with 5x5 windows 4 scales, 1 itera?on per scale.


Coarse scale: GM-‐EP (Energy Pyramid) vs GM-‐IP(Image Pyramid)


Finest scale: GM-‐EP (Energy Pyramid) vs GM-‐IP(Image Pyramid)


GM-‐EP (Energy Pyramid) vs MicMac (Image Pyramid)

Key points: •  A versa?le matching model efficiently op?mized with state of the art discrete

op?miza?on technique.

•  Energy pyramid yields a beper representa?on of the energy. Future work: •  Modeling:

•  Impact of images’ noise, •  Symmetry w.r.t. the images, •  Occlusions, •  CRF parameters (unary terms, weights of CRF, distance func?on).

•  Op?miza?on •  Auto defini?on of the search-‐space, •  Mul?grid instead of mul?scale, •  Paralleliza?on for shared and distributed memory architectures.


Conclusion & Future work:


Stereo-‐imaging in urban context: (Target Image)


Finest scale: GM-‐EP (Energy Pyramid) vs MicMac (Image Pyramid)

Micmac GM-‐EP (Energy Pyramid)


Finest scale: GM-‐EP (Energy Pyramid) vs MicMac (Image Pyramid)


Mul?-‐scale: GM-‐IP algorithm (Image Pyramid)


Mul?-‐scale: GM-‐EP algorithm (Energy Pyramid)

fast global stereo matching via energy pyramid minimization

Science

build energy pyramid

disparity matching

crf energy of crf algorithm

conejo fast global matching

energy pyramid discrete

build pyramid of image

disparity b

image pyramid classic