flexible voxels for motion-aware videography

Flexible Voxels for Motion-Aware Videography Mohit Gupta Srinivasa G. NarasimhanAmit Agrawal

Sampling of the Space-Time VolumeMotion-Aware Camera

Ashok Veeraraghavan

Post Capture Motion-Aware Interpretation of Data

Hardware Implementation: Fast Per-pixel Shutter

High Frame Rate (500 fps) for Fast Moving Regions

High Spatial Resolution (1 MP) for Static Regions

Post-Capture control of frame rate and spatial resolution

Time

Camera

Sensor Plane

Integration Time

Conventional Sampling Scheme:Frame 1 Frame 2 Frame N

Fat and short voxels

Low SR, High TR

Flexible voxels

Motion Aware Video

Thin and long voxels

High SR, Low TR

x

ty

Integration Time Time

Camera

Sensor Plane

Our Sampling Scheme:Frame 1 Frame 2 Frame N

Why is Building Such a Camera Hard?

Motion-Aware video requires flexible voxels and a priori knowledge of the scene

Image Plane

ProjectorImage Plane

Camera

Scene

Beam Splitter

Time

Pixel 1

Pixel 2

Pixel K

Camera Integration

Projector Pattern

Motion-Aware Video Results Fan Rotating (‘Wagon-wheel effect’)

Multi-Use Light Engine (MULE)

Pico-Projector (1440Hz.) ($350)

Co-located projector camera setup

Pixel on

Pixel off

Fast optical modulationFuture work: Passive

implementation using LCoS

[Mukaigawa et al]

Input Frame (Captured at 7.5 FPS)

Anisotropic diffusion Motion-Aware reconstruction using motion information

Captured Data Different Spatio-temporal Interpretations

Re-binningTR = 1X SR = 1X

TR = 2X SR = ½ X

TR = 4X SR = ¼X

Time

Spa

ce

Captured Frame

Simple re-binning results in aliasing artifacts

High SR High TR

Diffusion Kernels

Optical Flow Red = Fast, Green = Slow,

Blue = Stationary

Diffusion Kernels Aligned Along Flow Directions

Motion-Aware Reconstruction

High SR High TR

Reduced AliasingIncreasing Temporal Resolution

Coded Motion Blur

Reconstructed Frame (60 FPS) (1 out of 8)

High TR High SR