layered segmentation and optical flow estimation over...

Layered Segmentation and Optical Flow Estimation Over Time

Deqing Sun1, Erik B. Sudderth1, and Michael J. Black1,2

1Department of Computer Science, Brown University, Providence, RI, USA, 2Max Planck Institute for Intelligent Systems, 72076 Tubingen, Germany

Acknowledgements: DS and MJB were supported in part by the NSF Collaborative Research in Computational Neuroscience Program (IIS-0904875) and a gift from Intel Corporation.

𝑰𝑡+2 𝑰𝑡−1 𝑰𝑡+1 𝑰𝑡

𝒈𝑡1

𝒈𝑡2

𝒖𝑡1, 𝒗𝑡1

𝒖𝑡2, 𝒗𝑡2

𝒖𝑡3, 𝒗𝑡3 𝒌𝑡∗

Layer 1

support

Layer 2

support

Segmentation

Composite flow

Layer 1 flow

Layer 2 flow

Layer 3

(background) flow

MIT Layer Segmentation Benchmark

Ground truth Flow field

nLayers

Layers++ [5]

Layer segmentation Flow field

Classic+NL [4]

Toy Example Multiple Frames Help

Error Flow field Layer segmentation

End-point error: 0.345

End-point error: 0.311

t+2 t+1 t t-1

Using

frames t

and t+1

Using all 4

frames

Darker: larger

Middlebury Optical Flow Benchmark

Public table (June 2012)

Classic+NL [4] nLayers MDP-Flow2 [8] Ground truth

Black: occlusion

Layer segmentation

[1] Grundmann, Kwatra, Han, & Essa. Efficient hierarchical graph-based video segmentation. CVPR, 2010.

[2] Liu, Freeman, Adelson, & Weiss. Human-assisted Motion Annotation. CVPR, 2008.

[3] Sudderth & Jordan, Shared Segmentation of Natural Scenes Using Dependent Pitman-Yor Processes. NIPS, 2009.

[4] Sun, Roth, & Black, Secrets of Optical Flow Estimation and Their Principles. CVPR, 2010.

[5] Sun, Sudderth, & Black, Layered Image Motion with Explicit Occlusions, Temporal Consistency and Depth Ordering. NIPS,

2010.

[6] Wang & Adelson, Representing Moving Images with Layers. IEEE TIP 1994.

[7] Weiss, Smoothness in Layers: Motion Segmentation Using Nonparametric Mixture Estimation. CVPR, 1997.

[8] Xu, Jia, & Matsushita. Motion Detail Preserving Optical Flow Estimation. IEEE TPAMI to appear.

References

Occluded

Occlusion Reasoning

Robust penalty (generalized Charbonnier)

Constant penalty (uniform distribution)

Visible

Discrete-Continuous Optimization

After discrete

optimization

After continuous

Optimization [5]

After affine K-means

clustering

Optical flow [4]

𝑰𝑡 𝑰𝑡+1

𝒖𝑡𝑘

𝒗𝑡𝑘

𝒌𝑡+1∗ 𝒌𝑡∗

𝐾

𝐾 𝐾

𝒈𝑡+1,𝑘 𝒈𝑡𝑘

𝐾 − 1

Motion

Support function

Segmentation

Extended Alpha Expansion Segmentation move simultaneously changes several support functions to change segmentation

𝐛𝑡 1 − 𝐛𝑡

𝐠𝑡1proposal

𝐠𝑡1old

𝐠𝑡2proposal

𝐠𝑡2old

𝐠𝑡1

𝐠𝑡2

Near

Far

HGVS [1]

Rand Index (RI): 0.766

RI: 0.689

RI: 0.499

nLayers

RI: 0.979

RI: 0.766

RI: 0.881

Generative Layered Model

More complex moves can simultaneously change segmentation and flow. The flow field is chosen from a

candidate set of flow fields, which are adaptively refined by continuous optimization.

Flow field Layer

segmentation

Color key