S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

1

Computing Exact Discrete Minimal Surfaces: Extending and Solving the Shortest Path Problem in 3D with

Application to Segmentation

Leo Grady Department of Imaging and Visualization

Siemens Corporate Research

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

2

Outline

• Introduction

• Extending the shortest path problem to 3D

• Method for computation

• Results

• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

3

A shortest path is easy to find in 3D, but it is no longer a boundary

Introduction – Shortest path in 3D

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

4

Introduction – Minimal boundary segmentation

Minimal segmentation specified two waysCutting edges/specifying normals

Graph CutsMax-flow/Min-cut

Specifying boundary

Intelligent scissors/Live wireDijkstra’s algorithm

44 image 44 weighted graph

abstraction

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

5

Specifying boundary Shortest path on dual graph =

Sometimes called “cracks” or “edgels”

Introduction – Minimal boundary segmentation

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

6

444 volume Primal - Cuts

Max-flow/Min-cut

Dual - Surfaces

???

Introduction – Minimal boundary segmentation

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

7

Q: Why bother if you have max-flow/min-cut?A: Different inputs, different outputs

Max-flow/min-cutInput: Source (inside) and terminal (outside)

Output: Closed contour/surface

Shortest path/min surfaceInput: A piece of the boundary

Output: Open contour/surface

Different algorithmic complexity

Both algorithms persist in 2D, but max-flow/min-cut only option in 3D

Introduction – Minimal boundary segmentation

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

8

Outline

• Introduction

• Extending the shortest path problem to 3D

• Method for computation

• Results

• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

9

Extending the Shortest Path ProblemInput: Two points (0D boundary)Output: Minimum 1D path having that boundary

How to extend problem to 3D?

Path is shortest relative to weighting (metric)

0-dimensional boundaryminimal 1-dimensional object

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

10

Extending the Shortest Path ProblemInput: Closed contour (1D boundary)Output: Minimum 2D surface having that boundary

minimal 2-dimensional object

1-dimensional boundary

Surface is minimal relative to weighting (metric)

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

11

Extending the Shortest Path Problem2D intelligent scissors/live wire ubiquitous segmentation method

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

12

Red – Initial (boundary) contoursYellow – Computed minimal surface contours

Extending the Shortest Path Problem

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

13

Outline

• Introduction

• Extending the shortest path problem to 3D

• Method for computation

• Results

• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

14

Method

How to compute?

Use boundary operator to enforce constraint

: 1D 0D : 2D 1D

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

15

MethodMinimize path/surface

2D

- Binary vector indicating presence/absence of edge in path

- Weights of each edge in the graph

- Vector indicating nodes in boundary

- Node-edge incidence matrix

3D

- Weights of each face in the complex

- Binary vector indicating presence/absence of face in surface

Use boundary operator as constraint

Subject to: - Vector indicating edges in boundary

Since boundary operator is linear, representable by a matrixSubject to:

- Edge-face incidence matrix

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

16

Continuum interpretation - I

Use generalized Stokes Theorem:

: 1D 0D : 2D 1D

Fundamental Theorem of calculus:

“Standard” Stokes Theorem

Careful! Instead of bivectors, formulated in “primal” space

Method - Digression

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

17

Continuum interpretation - II

Use boundary as constraint

Method - Digression

Subject to:

RHS consists of a unit closed contour

3D

- Vector field taking nonzeros on the normals of the surface

- Ambient vector field (e.g., derived from image gradients)

2D

- Vector field taking nonzeros along minimal path

RHS consists of two delta functions at endpoints

- Ambient vector field (e.g., derived from image gradients)

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

18

Method

Minimal surface problem

Subject to:

Integer programming problem – Bad!

However:Sometimes we can apply linear programming toan integer programming problem and guaranteean integer solution.

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

19

MethodMinimal surface problem

Subject to:

If constraints are feasible, then

For some integer . Therefore,

For the matrix spanning the nullspace of

So, we can rewrite the problem in terms of

Joint work with Vladimir Kolmogorov

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

20

MethodOriginal minimal surface problem

Subject to:

New minimal surface problem

Subject to:

Guaranteed to give an integer solution for an integer iff is totally unimodular. A matrix is totally unimodular if every square submatrix has a determinant equal to one of the set

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

21

MethodNew minimal surface problem

Subject to:

Big question:What is ?

The volume-face boundary operator

: 3D 2D Followed by the : 2D 1D, gives zero

May also be stated as:“The boundary of the boundary is zero”

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

22

Method

Volume-face incidencein dual lattice

is node-edge incidencein primal lattice

Dual Primal

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

23

Method

New minimal surface problem

Subject to:

Since all node-edge incidence matrices are totally unimodular is guaranteed to be integer, and therefore

is guaranteed to be integer.

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

24

Method

Solvable using generic linear programming code!

Conclusion:Minimal surface problem

Subject to:

(and so is the other formulation)

Subject to:

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

25

Outline

• Introduction

• Extending the shortest path problem to 3D

• Method for computation

• Results

• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

26

Results - Correctness

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

27

Red – Initial (boundary) contoursYellow – Computed minimal surface contours

Results – 3D Segmentation

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

28

Outline

• Introduction

• Extending the shortest path problem to 3D

• Method for computation

• Results

• Conclusion

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

29

Conclusion

1. Natural extension of shortest path given two pointsis minimal surface given a closed contour

2. Minimal surface problem solvable with generic linearprogramming code

3. There are, in fact, two integral linear programming problems that could be solved to achieve the solution

S

I E

M E

N S

C O

R P

O R

A T

E

R

E S

E A

R C

H

30

More Information

My webpage:http://cns.bu.edu/~lgrady

MATLAB toolbox for graph theoretic image processing at:http://eslab.bu.edu/software/graphanalysis/

Writings and code

Combinatorial minimal surface MATLAB code:http://cns.bu.edu/~lgrady/minimal_surface_matlab_code.zip

AcknowledgementsMarie-Pierre Jolly – Posing the problem

Gareth Funka-Lea – Support and enthusiasm for the work

Yuri Boykov – Enthusiasm and encouragement of the topic

Chenyang Xu – Extensive comments on the paper

Vladimir Kolmogorov – Technical analysis of LP problem