Frequency Analysis and Sheared Reconstruction for Rendering Motion Blur

Frequency Analysis and Sheared Reconstruction for Rendering Motion Blur

Kevin Egan

Yu-Ting Tseng

Nicolas Holzschuch

Frédo Durand

Ravi Ramamoorthi

Columbia University

Columbia University

Grenoble University

MIT CSAIL

University of California, Berkeley

OverviewOverview

Introduction

• Overview of Frequency Analysis

• Sampling and Sheared Filter

• Implementation

• Results

Motion BlurMotion Blur

• Objects move while camera shutter is open

– Image is “blurred” over time

• Expensive for special effects

• Necessary to remove “strobing” in animations

Garfield: A Tail of Two KittiesRhythm & Hues StudiosTwentieth Century-Fox Film Corporation

The IncrediblesPixar Animation StudiosWalt Disney Pictures

A Simple ApproachA Simple Approach

• For each pixel

– Sample many different moments in time

t = 1.00

t = 0.75

t = 0.50

t = 0.25

t = 0.00

t [0.0, 1.0]

A Simple ApproachA Simple Approach

• The simple approach is expensive

• Can we do better?

ObservationObservation

• Motion blur is expensive

• Motion blur removes spatial complexity

Standard MethodStandard Method

• Use axis-aligned pixel filter at each pixel

• Requires many samples

SPACE

TIM

E

space-time samples

axis-aligned filter

Our MethodOur Method

• Use a different filter shape at each pixel

• Filter sheared in space-time

• Fewer samples and faster renders

SPACE

TIM

E

space-time samples

sheared filter

Previous WorkPrevious Work

• Plenoptic Sampling

Chai et al., 2000

Sheared reconstruction filter in space-angle

• A Frequency Analysis of Light Transport

Durand et al., 2005

Analysis of transport, reflection and occlusion

• We extend both to space-time

Previous WorkPrevious Work

• Multidimensional Adaptive Sampling

Hachisuka et al., 2008

Anisotropic reconstruction based on contrast

Our method based on local freq estimates

• Spatial Anti-Aliasing for Animation Sequences with Spatio-Temporal Filtering

Shinya, 1993

Sheared filter across multiple images

We use speed bounds, derive sampling rates

OverviewOverview

• Introduction

Overview of Frequency Analysis

• Sampling and Sheared Filter

• Implementation

• Results

Space-Time Analysis GoalsSpace-Time Analysis Goals

• Uniform motion leads to sparse freq content

• Analyze shutter filter’s effect on freq content

• Design a filter customized to freq content

Basic ExampleBasic Example

• Object not moving

x

y

SPACE

f(x, y) f(x, t)

Space-time graph

TIM

E

Basic ExampleBasic Example

x

y t

x

f(x, t)

• Low velocity, t [ 0.0, 1.0 ]

f(x, y)

Basic ExampleBasic Example

x

y t

x

f(x, t)

• High velocity, t [ 0.0, 1.0 ]

f(x, y)

Shear in Space-TimeShear in Space-Time

x

y t

x

f(x, t)

• Object moving with low velocity

f(x, y)

shear

Large Shear in Space-TimeLarge Shear in Space-Time

x

y t

x

• Object moving with high velocity

f(x, y) f(x, t)

Shear in Space-TimeShear in Space-Time

• Object moving away from camera

x

y t

x

f(x, y) f(x, t)

Camera Shutter FilterCamera Shutter Filter

• Applying shutter blurs across time

x

y t

x

f(x, y) f(x, t)

Basic Example – Fourier DomainBasic Example – Fourier Domain

• Fourier spectrum, zero velocity

t

x

f(x, t) F(Ωx, Ωt)texture

bandwidth

Ωt

Ωx

Basic Example – Fourier DomainBasic Example – Fourier Domain

• Low velocity, small shear in both domains

f(x, t) F(Ωx, Ωt)

t

x

slope = -speed

Ωt

Ωx

Basic Example – Fourier DomainBasic Example – Fourier Domain

• Large shear

f(x, t) F(Ωx, Ωt)

t

x Ωt

Ωx

Basic Example – Fourier DomainBasic Example – Fourier Domain

• Non-linear motion, wedge shaped spectra

f(x, t)

Ωt

Ωx

F(Ωx, Ωt)

t

x

shutter bandlimits in

time

-min speed

-max speed

shutter applies blur across time

indirectly bandlimits in

space

Main InsightsMain Insights

• Common case = double wedge spectra

• Shutter indirectly removes spatial freqs

• Moving reflections and shadows in paper

Ωt

Ωx

double wedge spectrum moving reflection

OverviewOverview

• Introduction and Simple Example

• Overview of Frequency Analysis

Sampling and Sheared Filter

• Implementation

• Results

Sampling and Filtering GoalsSampling and Filtering Goals

• Minimal sampling rates to prevent aliasing

• Derive shape of new reconstruction filter

Sampling in Fourier DomainSampling in Fourier Domain

Ωt

Ωxt

x

• Sampling produces replicas in Fourier domain

• Sparse sampling produces dense replicas

Fourier DomainSpace-Time Domain

Standard Reconstruction FilteringStandard Reconstruction Filtering

• Standard filter, dense sampling (slow)

Ωt

no aliasing

Ωx

Fourier Domainreplicas

Standard Reconstruction FilterStandard Reconstruction Filter

• Standard filter, sparse sampling (fast)

Ωt

aliasing

Ωx

Fourier Domain

Sheared Reconstruction FilterSheared Reconstruction Filter

• Our sheared filter, sparse sampling (fast)

Ωt

Ωx

No aliasing!

Fourier Domain

Sheared Reconstruction FilterSheared Reconstruction Filter

• Compact shape in Fourier = wide space-time

t

x

Space-Time Domain

Ωt

Ωx

Fourier Domain

Main InsightsMain Insights

• Sheared filter allows for many fewer samples

• Paper derives sampling rates

– for constant velocity same cost as a static image

OverviewOverview

• Introduction

• Overview of Frequency Analysis

• Sampling and Sheared Filter

Implementation

• Results

Our MethodOur Method

1. Compute bounds for signal speeds

2. Compute filter shapes and sampling rates

3. Render samples and reconstruct image

Car ExampleCar Example

static motion blurred

Implementation: Stage 1Implementation: Stage 1

• Sparse sampling to compute velocity bounds

max speedmin speed

Implementation: Stage 2Implementation: Stage 2

• Calculate filter widths and sampling rates

filter width

max speed

min speed

Implementation: Stage 2Implementation: Stage 2

• Uniform velocities, wide filter, low samples

filter width

max speed

min speed

Implementation: Stage 2Implementation: Stage 2

• Static surface, small filter, low samples

filter width

max speed

min speed

Implementation: Stage 2Implementation: Stage 2

• Varying velocities, small filter, high samples

filter width

max speed

min speed

Implementation: Stage 2Implementation: Stage 2

• Then compute sampling densities

samples per pixel

Uniform velocities =low sample count

Implementation: Stage 2Implementation: Stage 2

• Then compute sampling densities

samples per pixel

Varying velocities =high sample count

Implementation: Stage 3Implementation: Stage 3

• Render sample locations in space-time

• Apply wide sheared filters to nearby samples

t

x

sheared filter overlaps samples in multiple pixels

pixels

Implementation: Stage 3Implementation: Stage 3

• Filters stretched along direction of motion

• Preserve frequencies orthogonal to motion

filter shapes

OverviewOverview

• Introduction

• Overview of Frequency Analysis

• Sheared Filter

• Implementation

Results

Car SceneCar Scene

Stratified Sampling4 samples per pixel

Our Method,4 samples per pixel

Car SceneCar Scene

Stratified Sampling4 samples per pixel

Our Method,4 samples per pixel

Car Scene InsetCar Scene Inset

MDAS4 samples / pix

Our Method4 samples / pix

Ground Truth

Car Scene InsetCar Scene Inset

MDAS4 samples / pix

Our Method4 samples / pix

Ground Truth

Ballerina VideoBallerina Video

Ballerina StillsBallerina Stills

Ballerina StillsBallerina Stills

Ballerina StillsBallerina Stills

Ballerina InsetBallerina Inset

Stratified16 samples / pix

4 min 2 sec

Our Method8 samples / pix3 min 57 sec

Stratified64 samples / pix14 min 25 sec

Equal Time Equal Quality

Teapot SceneTeapot SceneOur Method

8 samples / pix

motion blurred reflection

LimitationsLimitations

• Currently filter along line segments

• Initial sampling may miss motion

• Need to calculate speed and bandlimits

• Multiple texture / reflection / shadow signals

ConclusionConclusion

• Derivation of filter shape and sampling rates

• conservative min/max bounds for speed

• packing spectra as tightly as possible• Implementation of sheared filter

• No explicit representation for spectrum

• Easily added to existing rendering pipelines

• Faster render times

• Space-time Fourier theory for rendering

• Double wedge shape using min/max bounds

• Analysis of shutter filter

AcknowledgementsAcknowledgements

• Funding Agencies

• ONR, NSF, Microsoft, Sloan Foundation, INRIA, LJK

• Equipment and Licenses

• Pixar, Intel, NVIDIA

Questions?Questions?

t

x

sampling rate

Previous WorkPrevious Work

• Distributed Ray Tracing,

Cook et al., 1984

Sampling across image, time and lens

• The Reyes Image Rendering Architecture,

Cook et al., 1987

Separate shading from visibility

Implementation: Stage 3Implementation: Stage 3

t

x

Narrow filter, low sample count

filter width

Implementation: Stage 3Implementation: Stage 3

t

x

Wide filter, low sample count

filter width

Implementation: Stage 3Implementation: Stage 3

t

x

Narrow filter, high sample count

filter width