![Page 1: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/1.jpg)
Three-Dimensional VideoPostproduction and Processing
Ibraheem Alhashim - July 10th 2013CMPT 880
![Page 2: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/2.jpg)
2
Overview
History + Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
![Page 3: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/3.jpg)
3
Overview
History + Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
![Page 4: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/4.jpg)
4
A bit of history
Imaging technology
James Maxwell 1855
JC d'Almeida 1858Joseph Niépce 1826
Stereoscope 1860
Underwood 1901
The Power of Love (1922 film)
The Jazz Singer 1927
![Page 5: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/5.jpg)
5
A bit of history
Becky Sharp (1935) 1950s
![Page 6: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/6.jpg)
6
A bit of history
Timeline of 3D movies
1950s 1980s 2010s
Worldwide: $2,782,275,172
![Page 7: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/7.jpg)
7
3D Feature Films Highest-grossing films 2012: nine of the top 15 were
in 3D [businessinsider.com]
Industry forecast > 20% of TVs by 2015
Year 3D Films2005 42006 72008 82009 202010 302011 382012 38
![Page 8: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/8.jpg)
8
Other uses
WW2 maps Virtual reality 3D Electron Microscopy Video games + virtual cinema 1968
1945
198020132013 2013
![Page 9: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/9.jpg)
9
3D Video
Usually marketed as objects popping off screen
![Page 10: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/10.jpg)
10
3D Video
In reality.. It’s the same old concept Present slightly different image per eye The brain combines them and perceives depth
Trick human visual system Stereo 3D content production:
technical, psychological, and creative skills
![Page 11: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/11.jpg)
11
3D Video - Issues
Not as straightforward as 2D production Several considerations for a good 3D experience Balance between 3D effect and overall experience Minimize viewing discomfort
Stereoscopic comfort zone Scene depth adaptation Control of global and local disparity Video composition
![Page 12: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/12.jpg)
12
Stereoscopic Comfort Zone Comfort zone
Stereographer “bring the whole real world inside this virtual space”
![Page 13: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/13.jpg)
13
Control of Absolute Disparity Convergence is controlled by shifting the views
![Page 14: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/14.jpg)
14
Scene Depth Adaptation
Keep in mind disparity range (screen size and resolution)
Carefully plan a scene’s 3D effect Consider transitions and provide resting periods Post-production depth adaptation
Manual changes per display
![Page 15: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/15.jpg)
15
3D Display Adaptation From cinema to TV Depth composition has to modified (stereographer)
Depth information allows for virtual view interpolation
![Page 16: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/16.jpg)
16
Local Disparity Adaptation Objects should be within stereoscopic window
Intentional depth changes Physical / multiple camera rigs Synthetic / depth-editing
Objects at the border Can cause retinal rivalry Should be cropped by virtually shifting screen plane closer However, not applicable to live broadcast
![Page 17: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/17.jpg)
17
Live auto-correction
Automatic correction & manipulation of stereo live broadcast Live sport events (big player) Close objects could abruptly appear
Open research problem Some kind of novel view synthesis
![Page 18: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/18.jpg)
18
Video composition
Mixing and Composition of 3D Material, Real and Animated Content Graphics overlay / subtitles Cannot be simply pasted over other footage Leverage knowledge about depth range of footage
![Page 19: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/19.jpg)
19
3D Video - Issues
Summary 3D Production has an “art” component Different medium requires different parameters Content makers / directors need to think about 3D issues Stereographer try to balance 3D effect with overall
viewing experience
![Page 20: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/20.jpg)
20
3D video
Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
![Page 21: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/21.jpg)
21
3D Display Technologies
“Offer immersive experience” 3D Glasses (cinema + TV) Head-mounted displays Volumetric and holographic displays Autostereoscopic displays
![Page 22: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/22.jpg)
22
Autostereoscopic displays
Best choice for mobile devices Backward compatible & closer to viewer expectation Most common
Parallax barrier Lenticular sheet
![Page 23: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/23.jpg)
23
Autostereoscopic displays Crosstalk (ghosting) is most important parameter
“information meant for one eye intrudes into the other eye’s view”
![Page 24: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/24.jpg)
24
Autostereoscopic displays Issues
Generally less available depth range More ghosting artifacts Also, depth information is essential
Synthesize different views
![Page 25: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/25.jpg)
25
3D video
Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
![Page 26: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/26.jpg)
26
Basic processing Signal processing to avoid visual artifacts Any small visual discrepancy could cause discomfort
Three main categories1. Correction of Geometrical Distortions2. Color Matching3. Adjustment of Stereo Geometry
![Page 27: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/27.jpg)
27
Correction of Geometrical Distortions
Camera rigs might not be perfectly aligned Real lenses impose radial distortions by nature Other lens parameters might not sync E.g. geometrical lens distortions or chromatic
aberration
![Page 28: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/28.jpg)
28
Color Matching Color discrepancy can lead to eyestrain and visual
fatigue Manual calibration need to be done on cameras Automatic methods exist (histogram filtering)
![Page 29: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/29.jpg)
29
Color Matching
Modern professional postproduction tools incorporate stereo color matching and grading
![Page 30: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/30.jpg)
30
Adjustment of Stereo Geometry
Convergence need to be selected and balanced carefully to achieve good stereo content
Usually by shifting images horizontally in contrary directions, however, cropping & scaling is needed
“shift-crop-scale” Demo http://stereo.nypl.org/create
![Page 31: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/31.jpg)
31
Adjustment of Stereo Geometry
Stereo baseline is fixed during shooting Several hardware solutions help camera team
analyze the disparity range Also help visualize result of shift-crop-scale
![Page 32: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/32.jpg)
32
3D video
Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
![Page 33: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/33.jpg)
33
3D Depth information
Depth information is needed for novel output images in post-production adjusting the view parallax (different screens) many different uses in postproduction
![Page 34: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/34.jpg)
34
Depth info
Extracting depth information (time of flight camera / SfS)
Structure-from Stereo (SfS) Advanced computer vision problem Stereo matching
Local – block matching, optical flow est. Global – graph cuts, simulated annealing
![Page 35: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/35.jpg)
35
Structure-from Stereo
Global methods are more accurate Slow + don’t work well on video / motion
Local methods are more widely used Window-based methods Some system are in real-time Blocky output
![Page 36: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/36.jpg)
36
Example of Depth-based method
Apply hybrid recursive matching (HRM) Follow by cross-trilateral median filtering (ACTMF) Semi-automatic
![Page 37: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/37.jpg)
37
Post processing depth
Align depth discontinues to object borders Remove noise and mismatches Fill occlusions Approaches
Use image segmentation Neighborhood filtering
![Page 38: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/38.jpg)
38
View synthesis
Synthesize new virtual stereo views by image-based rendering
Input – depth + color Output – image with new view
Two types depth-based + warping-based
![Page 39: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/39.jpg)
39
Depth-based
Computer vision techniques Image-based rendering (IBR) Depth-based rendering (DIBR) Layered-depth images (LDI) Intermediate view reconstruction (IVR)
![Page 40: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/40.jpg)
40
Depth-Image-Based Rendering
Need pixel-by-pixel depth maps Recent focus
Handle depth discontinues Better depth boundaries
![Page 41: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/41.jpg)
41
Warping-based Methods that deform the image content directly Compress or stretch by nonlinear warp function Do not need camera calibration, segmentation, fill holes Worst case, visible wobbling artifact
![Page 42: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/42.jpg)
42
View synthesis
Summary Depth maps are computed using computer vision
techniques (still active) Generate new views by image-based rendering or warping Warping methods can potentially have less visual problems
DEMO
![Page 43: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/43.jpg)
43
3D video
Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
![Page 44: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/44.jpg)
44
2D to 3D
3D to 2D is trivial Hot topic for 3DTV and 3D cinema Methods so far are
Manual (computer assisted) Automatic
![Page 45: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/45.jpg)
45
Depth cues
Human visual systemMonocular cuesBinocular cues
![Page 46: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/46.jpg)
46
Depth cues
Monocular depth cues - things can be seen by one eye (2D Camera)
Focus / defocus, perspective, relative size Light and shading and texture Motion parallax
Binocular – 2 eyes or 3D camera rigs accommodation, convergence, and binocular discrepancy
![Page 47: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/47.jpg)
47
Manual 2D to 3D
Applicable to prerecorded video Utilize depth cues to generate a stereoscopic view
for each frame Time consuming and costly Cost vs. quality
![Page 48: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/48.jpg)
48
Manual 2D to 3D Three major steps
Rotoscoping / segmentation Depth assignment Inpainting
Few companies provide process as a service
![Page 49: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/49.jpg)
49
Depth Assignment
Shifting different parts of scene to simulate 3D To avoid cardboard effect hire a “3D compositor”
Create displacement maps for each pixel Use 3D primitives, spheres or cubes Use DIBR or 3D warping to synthesize view cannot handle transparencies well
![Page 50: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/50.jpg)
50
Occlusion filling
Also known as “in-painting” One of the most challenging parts in 2D/3D
conversion
![Page 51: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/51.jpg)
51
Automatic conversion
Automatic systems Extract depth information + synthesize stereoscopic images
![Page 52: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/52.jpg)
52
Automatic segmentation
Automatic segmentation of background Background subtraction techniques (ML) Optical-flow-based (uses motion as cue)
Objects in scene segmentation Contour-based (color, edges, texture) “SnapCut” state of the art algorithm in After Effects
![Page 53: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/53.jpg)
53
Automatic depth estimation 3D Structure recovery (Shape From X)
Depth from focus/defocus Depth from geometric cues
Relative sizes, gradients Depth from color and intensity
Lights and shadow Depth from motion
Motion parallax
![Page 54: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/54.jpg)
54
Automatic in-painting
Relay on - texture synthesis or border continuation Some work is done on spatio-temporal inpainting
Enforce global temporal consistency for patches Object motion could reveal background
Not viable in practice
![Page 55: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/55.jpg)
55
Real-time and Offline conversion
Real-time Approximate using motion parallax Could use color + intensity information Height-depth cues Hybrid approaches are better
Offline Apply structure from motion Create virtual scene using different views Slow cloudification?
![Page 56: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/56.jpg)
56
2D to 3D
Summary Hire specialist to rebuild depth and frame objects Relay on computer vision methods to build depth info Fill background using in-painting Still an active research topic
segmentation + depth estimation + synthesis
![Page 57: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/57.jpg)
57
3D video
Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
![Page 58: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/58.jpg)
58
3D media for mobile devices
Consider size of display Computing power + Network constraints Encoding
3D Video coding (V+D) Multiview Video Coding (MVC)
![Page 59: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/59.jpg)
59
Issues
Stereoscopic images have different disparity Coding artifacts
Dealing with different channels Transmission artifacts
Image distortion on loss Temporal mismatch
Display artifacts autostereoscopic display problems (ghosting)
![Page 60: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/60.jpg)
60
3D media for mobile devices Not only mobile devices, Internet connected devices Standards for 3D video encoding are still developing
Challenges for 2D video on mobile and more (depth)
3D Video conferencing demand real-time processing
Display technologies are evolving Resolution, need an adaptive and future ready format
![Page 61: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/61.jpg)
61
3D video
Fundamentals 3D display technologies Basic processing View synthesis 2D to 3D conversion 3D media for mobile devices Outlook
![Page 62: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/62.jpg)
62
Outlook
Today, 3D video technology are becoming practical Content creation process have matured
still a lot of room for improvement and extension There is a huge commercial incentive in this field A lot of areas to work on…
![Page 63: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/63.jpg)
63
Outlook
![Page 64: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/64.jpg)
64
Commentary
3D has been historically about generating revenue using mind tricks
Tricks are getting much better! Immersive experience ≠ quality
![Page 65: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/65.jpg)
65
Commentary
Future of 3D Video Consumers decide it Good for short media If all devices support it
Cinema products areprofitable!
![Page 66: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/66.jpg)
66
Commentary Similar strategies could apply on different technology
Light-field camera – Lytro first product 2011 (2006) Nokia / Pelican Imaging 16-lens array camera 2014
![Page 67: Three-Dimensional Video Postproduction and Processing](https://reader035.vdocument.in/reader035/viewer/2022062813/56816523550346895dd7a560/html5/thumbnails/67.jpg)
67 Thank you!