pixmotor: a pixel motion integrator ivan neulander rhythm & hues

Pixmotor:A Pixel Motion Integrator

Pixmotor:A Pixel Motion Integrator

Ivan NeulanderRhythm & Hues

What is Pixmotor?What is Pixmotor?

A 2D motion blur solution at Rhythm & Hues

• Makes motion-blurred images based on static images generated by 3D renderer

Night At The Museum

Garfield: A Tail of Two Kitties




• Not a physically correct but plausible

– Incorporates heuristics to minimize artifacts

Night At The Museum





• Not a physically correct but plausible

– Incorporates heuristics to minimize artifacts

• Fits well into our production pipeline

– Render objects separately, combine in comp

– Zero-centered shutter minimizesinterpolation distance from static frame

Night At The Museum


MotivationMotivation

• Processing Speed

– Invariant of geometric complexity: Typical 2k element completes in under 1 minute

– Many speed/quality controls for interactive testing


• Processing Speed

– Invariant of geometric complexity: Typical 2k element completes in under 1 minute

– Many speed/quality controls for interactive testing

• Art Direction

– Tweak shutter per-frame without rerendering

– Vary motion blur within a frame

shuttershutter/2 shutter x 2 shutter x 3


• In-Compositor Shading

– For static frames:

• Renderer outputs images ofnormals, positions, etc.;compositor does shading

Diffuse Reflection Shader

Normals Output





– For motion-blurred frames:

• Motion blurring data imagesin renderer is incorrect

Ground truth


Normals Output





– For motion-blurred frames:

• Motion blurring data imagesin renderer is incorrect

• Using pixmotor allows doingshading on static images Ground truth


Normals Output

Pixmotor

Pixmotor Data FlowPixmotor Data Flow

Pixmotor

Color Input

Motion Input

Depth Input

Settings

Computing Motion VectorsComputing Motion Vectors

• When shading a triangle at barycentric point (,,):Compute NDC motion vector as

PNDC(,,){ShutterClose} - PNDC(,,){ShutterOpen}

Shutter Open Shading Moment Shutter Close

Implementation:OverviewImplementation:Overview

• Discretize motion into a set of slices

– Each slice = particular moment within shutter span

– Total number of slices depends on maximum motion vector

• User sets maximum pixel separation between slices

– Compute each slice and add it to accumulation buffer

Implementation:OverviewImplementation:Overview

• Discretize motion into a set of slices

– Each slice = particular moment within shutter span

– Total number of slices depends on maximum motion vector

• User sets maximum pixel separation between slices

– Compute each slice and add it to accumulation buffer

1 slice 3 slices 7 slices 52 slices

Implementation:Main LoopImplementation:Main Loop

• For each slice:

– Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector


• For each slice:


• Resolve occlusion for overlappingpixels using depth info


• For each slice:



• Holes form due to divergent motion Use hole-filling heuristics


• For each slice:



• Holes form due to divergent motion Use hole-filling heuristics

Static Image Single-res WBno hole fill

Double-res WB,no hole fill

Double-res WB,with hole fill

Pixmotor

Single-slice interpolation

Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics

• Before Main Loop:

– Blur motion image (optionally)

• Filter kernel must ignore empty pixels





– Compute motion image gradient

• Force high gradient at image edges





– Compute motion image gradient

• Force high gradient at image edges

Blur = 0 Blur = 3 pixels Blur = 10 pixels Blur = 60 pixels

Motion Gradient

Result


• During Main Loop:

– Identify holes as empty pixels near high-gradient regions

– Fill holes with nearby solid neighbors along motion path





– Solid pixels at image edges are always presumed holes





– Solid pixels at image edges are always presumed holes

StaticImage

MotionGradient

Hole Fill: off

Hole Fill: on

[demo]


• After Main Loop:

– Option to fix matte for remaining holes:

Unpremultiply partial pixels with solid neighbors(distance along motion vector times user constant)

• Helps create a solid matte where needed for compositingBut: can harden some soft edges


• After Main Loop:

– Option to fix matte for remaining holes:

Unpremultiply partial pixels with solid neighbors(distance along motion vector times user constant)

• Helps create a solid matte where needed for compositingBut: can harden some soft edges

FixMatte: 0.0

FixMatte: 1.0

ResultsResults

• No overlap or transparency: excellent

3D Motion Blur

Pixmotor

ResultsResults

• No overlap or transparency: excellent

• With overlap or transparency: not so good

3D Motion Blur

Pixmotor

Results: Linear 3D MB (easy case)Results: Linear 3D MB (easy case)

Results:Pixmotor (easy case)Results:Pixmotor (easy case)

Results:Linear 3D MB (hard case)Results:Linear 3D MB (hard case)

Results:Pixmotor (hard case)Results:Pixmotor (hard case)

Motion-Blurred Shadows:Our ApproachMotion-Blurred Shadows:Our Approach

• Need to store partial occlusion in depth map use matte channel

Matte Depth



• Process depth, matte channels using Pixmotor

Pixmotor

Matte Depth

Motion



• Process depth, matte channels using Pixmotor

• Use unpremultiplied values for depth

• Depth map camera settings are unmodified

Pixmotor

Matte Depth

Motion

Matte Depth

Motion-Blurred Shadows:ResultsMotion-Blurred Shadows:Results

Frustrations & Future WorkFrustrations & Future Work

• Transparency, occlusion artifacts

use more fragments per pixelBut: speed will suffer

• Nonlinear motion (e.g. propeller)

Use acceleration + velocity for motion

• Memory-hungry on deep inputs

Process channels in batchesBut: counteracts multithreading

• Apply to reflections, refractions

– Compute motion vectors for these

Demo VideoDemo Video

• Real-time capture of interactive Icy session on aquad-CPU Opteron 2200

demo

Thank YouThank You

• Jonathan Robbins

• Martin Ryan

www.rhythm.com/~ivan/pdfs/sketch2007.pdfwww.rhythm.com/~ivan/pub/sketch2007.ppt

pixmotor: a pixel motion integrator ivan neulander rhythm & hues

Documents

overviewdiscretize motion

motion blurring data

d motion blur solution

rerenderingvary motion

maximum motion vectoruser

static framenight

compute ndc motion vector

d renderernot