real-time video effects using a depth sensor robbie fraser and ridho jeftha

REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR

Robbie Fraser and Ridho Jeftha

The ProblemVideo effects:

• Difficult

• Expensive

• Time consuming

The SolutionApplication:

• Automated

• Real time

AutomationDepth sensor

• Capture colour with camera (RGBD)

• Capture depth with sensor (RGBD)

Research QuestionCan image based effects be applied to video

footage in real-time?

AimDevelop a system to:

• Capture RGBD from a camera and depth sensor

• Render a variety of different video effects

Test the system:

• Benchmark performance

• Evaluate the viability of real-time video effects

System Design

Mapping SystemPrepare the data

• Depth map

• Point cloud

• Normal map

Refine the data

• Real time input

• Reflectance map

Video Capture

Normal Map GenerationDepth Map to Normal Map

Normal Map GenerationDepth Map to Normal Map

Reflectance Map

• Estimate initial lighting conditions

• Use normals

• Estimate reflectance at each point

Map Refinement

Map Refinement

Effects SystemFour effects:

• Fog Synthesis

• Scene Relighting

• Depth of Field

• Cartoon Shading

Fog Synthesis

Scene Relighting

Depth of Field

Cartoon Shading

User Interface

• Interface with cameras

• View effects

• Edit parameters

PerformanceUtilise graphics processors

• Fragment shaders

• CUDA

Testing

• Benchmarko Different effectso Different parameterso Different resolutions

• Performance profiling

Success CriterionPerformance

• Entire process at 25fps

Work Allocation

Thank YouQuestions?