pixmotor: a pixel motion integrator ivan neulander rhythm & hues

Pixmotor:A Pixel Motion Integrator

Pixmotor:A Pixel Motion Integrator

Ivan NeulanderRhythm & Hues

What is Pixmotor?What is Pixmotor?

A 2D motion blur solution at Rhythm & Hues

• Makes motion-blurred images based on static images generated by 3D renderer

Night At The Museum

Garfield: A Tail of Two Kitties

What is Pixmotor?What is Pixmotor?

A 2D motion blur solution at Rhythm & Hues

• Makes motion-blurred images based on static images generated by 3D renderer

• Not a physically correct but plausible

– Incorporates heuristics to minimize artifacts

Night At The Museum

Garfield: A Tail of Two Kitties

What is Pixmotor?What is Pixmotor?

A 2D motion blur solution at Rhythm & Hues

• Makes motion-blurred images based on static images generated by 3D renderer

• Not a physically correct but plausible

– Incorporates heuristics to minimize artifacts

• Fits well into our production pipeline

– Render objects separately, combine in comp

– Zero-centered shutter minimizesinterpolation distance from static frame

Night At The Museum

Garfield: A Tail of Two Kitties

What is Pixmotor?What is Pixmotor?

A 2D motion blur solution at Rhythm & Hues

• Makes motion-blurred images based on static images generated by 3D renderer

• Not a physically correct but plausible

– Incorporates heuristics to minimize artifacts

• Fits well into our production pipeline

– Render objects separately, combine in comp

– Zero-centered shutter minimizesinterpolation distance from static frame

Night At The Museum

Garfield: A Tail of Two Kitties

MotivationMotivation

• Processing Speed

– Invariant of geometric complexity: Typical 2k element completes in under 1 minute

– Many speed/quality controls for interactive testing

MotivationMotivation

• Processing Speed

– Invariant of geometric complexity: Typical 2k element completes in under 1 minute

– Many speed/quality controls for interactive testing

• Art Direction

– Tweak shutter per-frame without rerendering

– Vary motion blur within a frame

shuttershutter/2 shutter x 2 shutter x 3

MotivationMotivation

• In-Compositor Shading

– For static frames:

• Renderer outputs images ofnormals, positions, etc.;compositor does shading

Diffuse Reflection Shader

Normals Output

MotivationMotivation

• In-Compositor Shading

– For static frames:

• Renderer outputs images ofnormals, positions, etc.;compositor does shading

– For motion-blurred frames:

• Motion blurring data imagesin renderer is incorrect

Ground truth

Diffuse Reflection Shader

Normals Output

MotivationMotivation

• In-Compositor Shading

– For static frames:

• Renderer outputs images ofnormals, positions, etc.;compositor does shading

– For motion-blurred frames:

• Motion blurring data imagesin renderer is incorrect

• Using pixmotor allows doingshading on static images Ground truth

Diffuse Reflection Shader

Normals Output

Pixmotor

Pixmotor Data FlowPixmotor Data Flow

Pixmotor

Color Input

Motion Input

Depth Input

Settings

Computing Motion VectorsComputing Motion Vectors

• When shading a triangle at barycentric point (,,):Compute NDC motion vector as

PNDC(,,){ShutterClose} - PNDC(,,){ShutterOpen}

Shutter Open Shading Moment Shutter Close

Implementation:OverviewImplementation:Overview

• Discretize motion into a set of slices

– Each slice = particular moment within shutter span

– Total number of slices depends on maximum motion vector

• User sets maximum pixel separation between slices

– Compute each slice and add it to accumulation buffer

Implementation:OverviewImplementation:Overview

• Discretize motion into a set of slices

– Each slice = particular moment within shutter span

– Total number of slices depends on maximum motion vector

• User sets maximum pixel separation between slices

– Compute each slice and add it to accumulation buffer

1 slice 3 slices 7 slices 52 slices

Implementation:Main LoopImplementation:Main Loop

• For each slice:

– Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector

Implementation:Main LoopImplementation:Main Loop

• For each slice:

– Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector

• Resolve occlusion for overlappingpixels using depth info

Implementation:Main LoopImplementation:Main Loop

• For each slice:

– Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector

• Resolve occlusion for overlappingpixels using depth info

• Holes form due to divergent motion Use hole-filling heuristics

Implementation:Main LoopImplementation:Main Loop

• For each slice:

– Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector

• Resolve occlusion for overlappingpixels using depth info

• Holes form due to divergent motion Use hole-filling heuristics

Static Image Single-res WBno hole fill

Double-res WB,no hole fill

Double-res WB,with hole fill

Pixmotor

Single-slice interpolation

Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics

• Before Main Loop:

– Blur motion image (optionally)

• Filter kernel must ignore empty pixels

Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics

• Before Main Loop:

– Blur motion image (optionally)

• Filter kernel must ignore empty pixels

– Compute motion image gradient

• Force high gradient at image edges

Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics

• Before Main Loop:

– Blur motion image (optionally)

• Filter kernel must ignore empty pixels

– Compute motion image gradient

• Force high gradient at image edges

Blur = 0 Blur = 3 pixels Blur = 10 pixels Blur = 60 pixels

Motion Gradient

Result

Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics

• During Main Loop:

– Identify holes as empty pixels near high-gradient regions

– Fill holes with nearby solid neighbors along motion path

Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics

• During Main Loop:

– Identify holes as empty pixels near high-gradient regions

– Fill holes with nearby solid neighbors along motion path

– Solid pixels at image edges are always presumed holes

Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics

• During Main Loop:

– Identify holes as empty pixels near high-gradient regions

– Fill holes with nearby solid neighbors along motion path

– Solid pixels at image edges are always presumed holes

StaticImage

MotionGradient

Hole Fill: off

Hole Fill: on

[demo]

Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics

• After Main Loop:

– Option to fix matte for remaining holes:

Unpremultiply partial pixels with solid neighbors(distance along motion vector times user constant)

• Helps create a solid matte where needed for compositingBut: can harden some soft edges

Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics

• After Main Loop:

– Option to fix matte for remaining holes:

Unpremultiply partial pixels with solid neighbors(distance along motion vector times user constant)

• Helps create a solid matte where needed for compositingBut: can harden some soft edges

FixMatte: 0.0

FixMatte: 1.0

ResultsResults

• No overlap or transparency: excellent

3D Motion Blur

Pixmotor

ResultsResults

• No overlap or transparency: excellent

• With overlap or transparency: not so good

3D Motion Blur

Pixmotor

Results: Linear 3D MB (easy case)Results: Linear 3D MB (easy case)

Results:Pixmotor (easy case)Results:Pixmotor (easy case)

Results:Linear 3D MB (hard case)Results:Linear 3D MB (hard case)

Results:Pixmotor (hard case)Results:Pixmotor (hard case)

Motion-Blurred Shadows:Our ApproachMotion-Blurred Shadows:Our Approach

• Need to store partial occlusion in depth map use matte channel

Matte Depth

Motion-Blurred Shadows:Our ApproachMotion-Blurred Shadows:Our Approach

• Need to store partial occlusion in depth map use matte channel

• Process depth, matte channels using Pixmotor

Pixmotor

Matte Depth

Motion

Motion-Blurred Shadows:Our ApproachMotion-Blurred Shadows:Our Approach

• Need to store partial occlusion in depth map use matte channel

• Process depth, matte channels using Pixmotor

• Use unpremultiplied values for depth

• Depth map camera settings are unmodified

Pixmotor

Matte Depth

Motion

Matte Depth

Motion-Blurred Shadows:ResultsMotion-Blurred Shadows:Results

Motion-Blurred Shadows:ResultsMotion-Blurred Shadows:Results

Frustrations & Future WorkFrustrations & Future Work

• Transparency, occlusion artifacts

use more fragments per pixelBut: speed will suffer

• Nonlinear motion (e.g. propeller)

Use acceleration + velocity for motion

• Memory-hungry on deep inputs

Process channels in batchesBut: counteracts multithreading

• Apply to reflections, refractions

– Compute motion vectors for these

Demo VideoDemo Video

• Real-time capture of interactive Icy session on aquad-CPU Opteron 2200

demo

Thank YouThank You

• Jonathan Robbins

• Martin Ryan

www.rhythm.com/~ivan/pdfs/sketch2007.pdfwww.rhythm.com/~ivan/pub/sketch2007.ppt