real-time video effects using a depth sensor robbie fraser and ridho jeftha
TRANSCRIPT
![Page 1: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/1.jpg)
REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR
Robbie Fraser and Ridho Jeftha
![Page 2: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/2.jpg)
The ProblemVideo effects:
• Difficult
• Expensive
• Time consuming
![Page 3: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/3.jpg)
The SolutionApplication:
• Automated
• Real time
![Page 4: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/4.jpg)
AutomationDepth sensor
• Capture colour with camera (RGBD)
• Capture depth with sensor (RGBD)
![Page 5: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/5.jpg)
Research QuestionCan image based effects be applied to video
footage in real-time?
![Page 6: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/6.jpg)
AimDevelop a system to:
• Capture RGBD from a camera and depth sensor
• Render a variety of different video effects
Test the system:
• Benchmark performance
• Evaluate the viability of real-time video effects
![Page 7: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/7.jpg)
System Design
![Page 8: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/8.jpg)
Mapping SystemPrepare the data
• Depth map
• Point cloud
• Normal map
Refine the data
• Real time input
• Reflectance map
![Page 9: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/9.jpg)
Video Capture
![Page 10: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/10.jpg)
Normal Map GenerationDepth Map to Normal Map
![Page 11: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/11.jpg)
Normal Map GenerationDepth Map to Normal Map
![Page 12: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/12.jpg)
Reflectance Map
• Estimate initial lighting conditions
• Use normals
• Estimate reflectance at each point
![Page 13: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/13.jpg)
Map Refinement
![Page 14: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/14.jpg)
Map Refinement
![Page 15: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/15.jpg)
Effects SystemFour effects:
• Fog Synthesis
• Scene Relighting
• Depth of Field
• Cartoon Shading
![Page 16: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/16.jpg)
Fog Synthesis
![Page 17: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/17.jpg)
Scene Relighting
![Page 18: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/18.jpg)
Depth of Field
![Page 19: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/19.jpg)
Cartoon Shading
![Page 20: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/20.jpg)
User Interface
• Interface with cameras
• View effects
• Edit parameters
![Page 21: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/21.jpg)
PerformanceUtilise graphics processors
• Fragment shaders
• CUDA
![Page 22: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/22.jpg)
Testing
• Benchmarko Different effectso Different parameterso Different resolutions
• Performance profiling
![Page 23: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/23.jpg)
Success CriterionPerformance
• Entire process at 25fps
![Page 24: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/24.jpg)
Work Allocation
![Page 25: REAL-TIME VIDEO EFFECTS USING A DEPTH SENSOR Robbie Fraser and Ridho Jeftha](https://reader036.vdocument.in/reader036/viewer/2022081515/56649e895503460f94b8d953/html5/thumbnails/25.jpg)
Thank YouQuestions?