pixmotor: a pixel motion integrator
DESCRIPTION
Pixmotor: A Pixel Motion Integrator. Ivan Neulander Rhythm & Hues. What is Pixmotor ?. A 2D motion blur solution at Rhythm & Hues Makes motion-blurred images based on static images generated by 3D renderer. Garfield: A Tail of Two Kitties. Night At The Museum. What is Pixmotor?. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/1.jpg)
![Page 2: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/2.jpg)
Pixmotor:A Pixel Motion Integrator
Pixmotor:A Pixel Motion Integrator
Ivan NeulanderRhythm & Hues
![Page 3: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/3.jpg)
What is Pixmotor?What is Pixmotor?
A 2D motion blur solution at Rhythm & Hues
• Makes motion-blurred images based on static images generated by 3D renderer
Night At The Museum
Garfield: A Tail of Two Kitties
![Page 4: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/4.jpg)
What is Pixmotor?What is Pixmotor?
A 2D motion blur solution at Rhythm & Hues
• Makes motion-blurred images based on static images generated by 3D renderer
• Not a physically correct but plausible
– Incorporates heuristics to minimize artifacts
Night At The Museum
Garfield: A Tail of Two Kitties
![Page 5: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/5.jpg)
What is Pixmotor?What is Pixmotor?
A 2D motion blur solution at Rhythm & Hues
• Makes motion-blurred images based on static images generated by 3D renderer
• Not a physically correct but plausible
– Incorporates heuristics to minimize artifacts
• Fits well into our production pipeline
– Render objects separately, combine in comp
– Zero-centered shutter minimizesinterpolation distance from static frame
Night At The Museum
Garfield: A Tail of Two Kitties
![Page 6: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/6.jpg)
What is Pixmotor?What is Pixmotor?
A 2D motion blur solution at Rhythm & Hues
• Makes motion-blurred images based on static images generated by 3D renderer
• Not a physically correct but plausible
– Incorporates heuristics to minimize artifacts
• Fits well into our production pipeline
– Render objects separately, combine in comp
– Zero-centered shutter minimizesinterpolation distance from static frame
Night At The Museum
Garfield: A Tail of Two Kitties
![Page 7: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/7.jpg)
MotivationMotivation
• Processing Speed
– Invariant of geometric complexity: Typical 2k element completes in under 1 minute
– Many speed/quality controls for interactive testing
![Page 8: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/8.jpg)
MotivationMotivation
• Processing Speed
– Invariant of geometric complexity: Typical 2k element completes in under 1 minute
– Many speed/quality controls for interactive testing
• Art Direction
– Tweak shutter per-frame without rerendering
– Vary motion blur within a frame
shuttershutter/2 shutter x 2 shutter x 3
![Page 9: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/9.jpg)
MotivationMotivation
• In-Compositor Shading
– For static frames:
• Renderer outputs images ofnormals, positions, etc.;compositor does shading
Diffuse Reflection Shader
Normals Output
![Page 10: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/10.jpg)
MotivationMotivation
• In-Compositor Shading
– For static frames:
• Renderer outputs images ofnormals, positions, etc.;compositor does shading
– For motion-blurred frames:
• Motion blurring data imagesin renderer is incorrect
Ground truth
Diffuse Reflection Shader
Normals Output
![Page 11: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/11.jpg)
MotivationMotivation
• In-Compositor Shading
– For static frames:
• Renderer outputs images ofnormals, positions, etc.;compositor does shading
– For motion-blurred frames:
• Motion blurring data imagesin renderer is incorrect
• Using pixmotor allows doingshading on static images Ground truth
Diffuse Reflection Shader
Normals Output
Pixmotor
![Page 12: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/12.jpg)
Pixmotor Data FlowPixmotor Data Flow
Pixmotor
Color Input
Motion Input
Depth Input
Settings
![Page 13: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/13.jpg)
Computing Motion VectorsComputing Motion Vectors
• When shading a triangle at barycentric point (,,):Compute NDC motion vector as
PNDC(,,){ShutterClose} - PNDC(,,){ShutterOpen}
Shutter Open Shading Moment Shutter Close
![Page 14: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/14.jpg)
Implementation:OverviewImplementation:Overview
• Discretize motion into a set of slices
– Each slice = particular moment within shutter span
– Total number of slices depends on maximum motion vector
• User sets maximum pixel separation between slices
– Compute each slice and add it to accumulation buffer
![Page 15: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/15.jpg)
Implementation:OverviewImplementation:Overview
• Discretize motion into a set of slices
– Each slice = particular moment within shutter span
– Total number of slices depends on maximum motion vector
• User sets maximum pixel separation between slices
– Compute each slice and add it to accumulation buffer
1 slice 3 slices 7 slices 52 slices
![Page 16: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/16.jpg)
Implementation:Main LoopImplementation:Main Loop
• For each slice:
– Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector
![Page 17: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/17.jpg)
Implementation:Main LoopImplementation:Main Loop
• For each slice:
– Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector
• Resolve occlusion for overlappingpixels using depth info
![Page 18: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/18.jpg)
Implementation:Main LoopImplementation:Main Loop
• For each slice:
– Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector
• Resolve occlusion for overlappingpixels using depth info
• Holes form due to divergent motion Use hole-filling heuristics
![Page 19: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/19.jpg)
Implementation:Main LoopImplementation:Main Loop
• For each slice:
– Copy each pixel from source image to a work buffer, shifting its coordinates based on motion vector
• Resolve occlusion for overlappingpixels using depth info
• Holes form due to divergent motion Use hole-filling heuristics
Static Image Single-res WBno hole fill
Double-res WB,no hole fill
Double-res WB,with hole fill
Pixmotor
Single-slice interpolation
![Page 20: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/20.jpg)
Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics
• Before Main Loop:
– Blur motion image (optionally)
• Filter kernel must ignore empty pixels
![Page 21: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/21.jpg)
Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics
• Before Main Loop:
– Blur motion image (optionally)
• Filter kernel must ignore empty pixels
– Compute motion image gradient
• Force high gradient at image edges
![Page 22: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/22.jpg)
Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics
• Before Main Loop:
– Blur motion image (optionally)
• Filter kernel must ignore empty pixels
– Compute motion image gradient
• Force high gradient at image edges
Blur = 0 Blur = 3 pixels Blur = 10 pixels Blur = 60 pixels
Motion Gradient
Result
![Page 23: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/23.jpg)
Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics
• During Main Loop:
– Identify holes as empty pixels near high-gradient regions
– Fill holes with nearby solid neighbors along motion path
![Page 24: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/24.jpg)
Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics
• During Main Loop:
– Identify holes as empty pixels near high-gradient regions
– Fill holes with nearby solid neighbors along motion path
– Solid pixels at image edges are always presumed holes
![Page 25: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/25.jpg)
Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics
• During Main Loop:
– Identify holes as empty pixels near high-gradient regions
– Fill holes with nearby solid neighbors along motion path
– Solid pixels at image edges are always presumed holes
StaticImage
MotionGradient
Hole Fill: off
Hole Fill: on
[demo]
![Page 26: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/26.jpg)
Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics
• After Main Loop:
– Option to fix matte for remaining holes:
Unpremultiply partial pixels with solid neighbors(distance along motion vector times user constant)
• Helps create a solid matte where needed for compositingBut: can harden some soft edges
![Page 27: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/27.jpg)
Implementation:Hole-Filling HeuristicsImplementation:Hole-Filling Heuristics
• After Main Loop:
– Option to fix matte for remaining holes:
Unpremultiply partial pixels with solid neighbors(distance along motion vector times user constant)
• Helps create a solid matte where needed for compositingBut: can harden some soft edges
FixMatte: 0.0
FixMatte: 1.0
![Page 28: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/28.jpg)
ResultsResults
• No overlap or transparency: excellent
3D Motion Blur
Pixmotor
![Page 29: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/29.jpg)
ResultsResults
• No overlap or transparency: excellent
• With overlap or transparency: not so good
3D Motion Blur
Pixmotor
![Page 30: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/30.jpg)
Results: Linear 3D MB (easy case)Results: Linear 3D MB (easy case)
![Page 31: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/31.jpg)
Results:Pixmotor (easy case)Results:Pixmotor (easy case)
![Page 32: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/32.jpg)
Results:Linear 3D MB (hard case)Results:Linear 3D MB (hard case)
![Page 33: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/33.jpg)
Results:Pixmotor (hard case)Results:Pixmotor (hard case)
![Page 34: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/34.jpg)
Motion-Blurred Shadows:Our ApproachMotion-Blurred Shadows:Our Approach
• Need to store partial occlusion in depth map use matte channel
Matte Depth
![Page 35: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/35.jpg)
Motion-Blurred Shadows:Our ApproachMotion-Blurred Shadows:Our Approach
• Need to store partial occlusion in depth map use matte channel
• Process depth, matte channels using Pixmotor
Pixmotor
Matte Depth
Motion
![Page 36: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/36.jpg)
Motion-Blurred Shadows:Our ApproachMotion-Blurred Shadows:Our Approach
• Need to store partial occlusion in depth map use matte channel
• Process depth, matte channels using Pixmotor
• Use unpremultiplied values for depth
• Depth map camera settings are unmodified
Pixmotor
Matte Depth
Motion
Matte Depth
![Page 37: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/37.jpg)
Motion-Blurred Shadows:ResultsMotion-Blurred Shadows:Results
![Page 38: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/38.jpg)
Motion-Blurred Shadows:ResultsMotion-Blurred Shadows:Results
![Page 39: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/39.jpg)
Frustrations & Future WorkFrustrations & Future Work
• Transparency, occlusion artifacts
use more fragments per pixelBut: speed will suffer
• Nonlinear motion (e.g. propeller)
Use acceleration + velocity for motion
• Memory-hungry on deep inputs
Process channels in batchesBut: counteracts multithreading
• Apply to reflections, refractions
– Compute motion vectors for these
![Page 40: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/40.jpg)
Demo VideoDemo Video
• Real-time capture of interactive Icy session on aquad-CPU Opteron 2200
demo
![Page 41: Pixmotor: A Pixel Motion Integrator](https://reader035.vdocument.in/reader035/viewer/2022062500/5681516c550346895dbf9c04/html5/thumbnails/41.jpg)
Thank YouThank You
• Jonathan Robbins
• Martin Ryan
www.rhythm.com/~ivan/pdfs/sketch2007.pdfwww.rhythm.com/~ivan/pub/sketch2007.ppt