Depth and Motion Discontinuities

Stan BirchfieldPh.D. oral defense

Stanford UniversityJanuary 1999

• PHOTOMETRIC

• GEOMETRIC

Discontinuities

intensity, color, texture, ...

Depth Motion

camerasurface

camerasurface

• Fundamental to image understanding

• Object boundaries(well-defined)

• Assumption of continuity

• Occlusion

Importance of Discontinuities

camera 1 camera 2

object

Motivation

segmentation

classification

tracking

camera control

video retrieval

detection

IMAGES LOW-LEVEL FILTERS HIGH-LEVEL TASKS

Evidence for Discontinuities

• LOCAL– T-junctions

[Parida et al. 1997, Ruzon & Tomasi 1999]

– Occlusion [Toh & Forrest 1990, Wixson 1993]

– Multi-modal depth or motion[Spoerri & Ullman 1987, Little & Gillett 1990]

– Changes in intensity, color, texture, …[Canny 1986, Wang & Binford 1994]

• GLOBAL– “Continuity of discontinuities”

[Marr & Poggio 1979]

– Changes in depth or motion field[Thompson et al. 1982, Birchfield & Tomasi 1998]

Complications from Sampling

• Amount of discontinuity

• Curvature

?

Finding and Using Discontinuities

Depth discontinuities from stereo

Maximum flow for stereo and motion

Motion discontinuities from a sequence

Using discontinuities to track heads

Depth Discontinuities from a Stereo Pair

E E d(x ,x - ) u(l )data smoothness L Lx iiL

Use dynamicprogrammingto minimize:

inte

nsi

ty

L

R

dis

par

ity

lamp

wall

pixel

dissimilarity discontinuitypenalty

epipolarconstraint

Discontinuity Penalty

u(l )i

pen

alty

occlusion length

u(3) = 3u(1)

u(3) < 3u(1)

BADGOOD

l i

L

R

L

R

GOOD BAD

disp

arit

y

pixel

inte

nsi

ty

Stereo Algorithm

• Matches pixels directly with convex penalty function(no preprocessing or windows)

• Handles large untextured regions

• Solves sampling problem

• Handles slanted surfaces

• Is fast

Handling Untextured Regions

Our constraint: Depth discontinuities lie to a particular side of intensity edge Discontinuities in left (right) scanline lie to the left (right) of intensity edge

Assumption: Depth discontinuities are accompanied by intensity edge

Naïve constraint: Depth discontinuities lie near intensity edge

Intensity edges(x-derivative with

low threshold)

With naïveconstraint

With ourconstraint

Discontinuity Lies to the Left of an Edge

near object

far object

leftcamera

rightcamera

Obvious matches

Physicalsetup

Matches violateassumption

Matches areconsistent withassumption

edge

edge

depth discontinuity

depth discontinuity

0

20

40

60

80

100

120

140

inte

nsi

ty (

gra

y l

ev

els

)

The Problem of Image Sampling

0

20

40

60

80

100

120

140

inte

nsi

ty (

gra

y l

ev

els

)

0

2

4

6

8

10

12

14

16

pixel

dis

sim

ilar

ity

(gra

y le

vels

)

Absolute difference

0

2

4

6

8

10

12

14

16

pixel

dis

sim

ilar

ity

(gra

y le

vels

)

Our measure

Left scanline Right scanline

A Dissimilarity Measure That is Insensitive to Image Sampling

d(xL,xR)

xL xR

d(xL,xR) = min{d(xL,xR) ,d(xR,xL)}Our dissimilarity measure:

Analysis of Dissimilarity Measure

Concave/convex regions: measure is guaranteed to work

Inflection points: measure works in practice when lens is defocused to remove aliasing

(inflection points linear {convex, concave})

intensityfunction

020406080

100120140160180200

1 2 3 4 5 6 7 8 9 10

diss

imil

arit

yAnalysis of Dissimilarity Measure

our measure

absolute difference

maximumaverage

Tc = 1fc

cutoff frequency

As a function of lens defocus

0102030405060708090

100

9 10 11 12

actual disparity (pixels)

diss

imil

arit

y

At a hypothesized disparity of 10 pixels

aliasing

{ defocus

Speeding Computation

Dis

par

ity

map

occlusion

depthdiscontinuity

Pruning bad nodes

Standard

Ours

maximum disparity

tim

e

Fastpostprocessing

TWO STEPS:1. Reliable overruns unreliable2. Background overruns foreground

RIGHT

LE

FT

Handling Slanted Surfaces

Independent scanlines:

no coherence

Postprocessing:forbid propagationif disparity changes

by just one level

disp

arit

y

column

Results

More Results

[Images from JISCT data set]

Finding and Using Discontinuities

Depth discontinuities from stereo

Maximum flow for stereo and motion

Motion discontinuities from a sequence

Using discontinuities to track heads

Motion Discontinuities from a Sequence

• Similar to stereo(matching pixels to fit piecewise-smooth function)

• But different(2D search, non-rigid transformation, many images)

• How far can we get using sparse features?

Motion Video

Difficulties of Image Sequences

• Instantaneous velocityfrom sampled positions

• Accumulation of evidence

• Frame of reference, motion model

time

posi

tion

?

feature

posi

tion

time

feature

background

thresholdpo

siti

on

time

feature 2

feature 1

background

threshold

Image Strain

Motion Algorithm

• Select and track features

• Group

• Trace boundariesbetween groups

Finding and Using Discontinuities

Depth discontinuities from stereo

Maximum flow for stereo and motion

Motion discontinuities from a sequence

Using discontinuities to track heads

Why Maximum Flow?

• Our stereo algorithm (Dynamic programming)– Good results on difficult images

– Fast

– But, suboptimal(processes rows, then columns independently)

• Newer algorithms (Maximum flow)– Set up graph, find maximum flow ---

minimum cut yields disparity or motion map

– Able to look at whole image

Minimizing a 2D Cost Function

Minimize:

disparity

p,q

p,q

d(p, )2D:

GL

OB

AL

dis

par

ity

pixel?

p,q

d(p, )

1D:

E E d(p, ) u(l )data smoothness p p,q p,q

{p,q} N

Global

u(l )p,q

Discontinuity penalty:

lp,q

minimum cut = disparity surface

u(l )= lp,q p,q p,qsolves

LO

CA

L Local (GOOD)

(BAD)

Maximum Flow for Stereo and Motion

Global algorithm[Roy & Cox 1998,

Ishikawa & Geiger 1998]

Local algorithm[Boykov et al. 1998]

pen

alty

BAD

discontinuity amount

GOOD

Challenges for Maximum Flow

… and slant of table

With our intensity variation constraint

and large penalty

With small penaltyWith large penalty … and slant of surfaces

Finding and Using Discontinuities

Depth discontinuities from stereo

Maximum flow for stereo and motion

Motion discontinuities from a sequence

Using discontinuities to track heads

Problem

TILT

PAN

ZOOM

CHALLENGES: * rotation * multiple people * zoom

APPLICATIONS: * video conferencing * distance learning

Previous Methods

3D R

otat

ion

Mov

ing

peop

lein

bac

kgro

und

Skin color:

N ?

Background subtraction:

Y N

Template:

N Y

Contour:

Y N

Head Tracking Algorithm

MODEL(x,y)

Ellipse: vertical aspect ratio = 1.2state s = (x,y,)

ellipse in frame t

OUTPUT

Ni

Ni

iI

iMiIc

1

1

)(

))(),(min()(

s

sscolor module

Model

Current

Intersection

Skin Hair

LOCAL SEARCH

gradient module

N

iNg ii1

1 |)()(|)( sgns

gradientellipse normal

ellipse in

frame t

INPUT

frame t-1

Evaluation of Head Tracker

1. Tracks head in real time on standard hardware

2. Insensitive to - full 360-degree out-of-plane rotation - arbitrary camera movement (including zoom) - multiple moving people - severe but brief occlusion - hair/skin color, hair length, facial hair, glasses

Headtracker Video

Finding and Using Discontinuities

Depth discontinuities from stereo

Maximum flow for stereo and motion

Motion discontinuities from a sequence

Using discontinuities to track heads

Contributions• Precise, unified definitions and relationship• Pixel-to-pixel stereo algorithm [ICCV 1998]

– good results on difficult images• intensity variation constraint

• dissimilarity measure [PAMI 1998]• preserving slant

– fast (pruning search nodes, postprocessor)

• Motion discontinuities(sparse features, image strain, grouping algorithm)

• Maximum flow techniques(edge weights for global algorithm, smoothness, left border ; applied to motion)

• Elliptical head tracker [Asilomar 1997, CVPR 1998]

– advanced state of the art by tracking people undergoing 3D rotation in front of a dynamic background

Appendix: Extra slides

Comparison of Our Algorithm with Maxflow

Ouralgorithm

Localmaxflow

algorithm

Depth Discontinuity

• Proposition 1: A depth discontinuity is a point in the image plane whose projection ray grazes a surface in the world.

object

image plane

focal pointDepth discontinuities

Definition of Jump

• The jump of a discontinuity is the smallest >0 such that there is a >0 for which, if ||x-x0||<, then |f(x)-f(x0)| >

x

f(x)

x0

jump

Comparison with Intensity Edges

Original image Intensity edges Depth discontinuities

•More closely tied to actual object boundaries•Not distracted by

•change in albedo (reflectance)•change in lighting (e.g., shadows)

Comparison with Layers

Original image Motion discontinuities

Comparison with Segmentation

A plausiblesegmentation

Depthdiscontinuities

Original image

•Well-defined (not task-dependent)•Curves are not necessarily closed

Epipolar Constraint

Left camera Right camera

world point

center ofprojection

epipolarplane

epipolarline

Comparison of Dissimilarity Measure with Other Methods

absolutedifference

our measure

spurious matches

subpixelresolution

0 20 40 60 80 100

seconds

Computing time

Propagating Information Between Scanlines

• Calculate reliabilities & threshold Disparities: [ 4 4 4 4 4 7 7 7 2 ]T

Reliabilities: [ 5 5 5 5 5 3 3 3 1 ]T

unreliable < reliable < very reliable

• Propagate very reliable when– neighbor unreliable, OR

– disparity(neighbor) > disparity(pixel) + 2

• Repeat in x-direction

Fast (30% increase instead of 800%)

Features on Freethrow

Maximum Flow

• Finds greatest shipping rate from source to sink w/o violating capacity constraints

• Flow network G=(V,E) is directed graph with edge capacities c(u,v) > 0, source s, and sink t

• Flow f:E R

• Value of flow |f| = f(s,v)

• Find f* = arg max |f|

• Cut (S,T) is partition of V s.t. s in S and t in T

• capacity of cut c(S,T)

• Max-flow min-cut theorem:There exists a cut (S,T) s.t. |f*| = c(S,T)

Blowup of Meter

Global algorithm Local algorithm

Maximum Flow and Occlusion

[from University of Tsukuba, Japan]

Maximum Flow and Left Image Border