3d material style transfer
DESCRIPTION
3D Material Style Transfer. Chuong H. Nguyen 1 , Tobias Ritschel 2 , Karol Myszkowski 1 , Elmar Eisemann 2 , Hans-Peter Seidel 1 1 Max-Planck-Institut Informatik 2 Telecom ParisTech / CNRS . Background – Material Perception. [ Ce Liu et al. 2010]. Background - Mood. Background. - PowerPoint PPT PresentationTRANSCRIPT
Eurographics 2012, Cagliari, Italy
3D Material Style TransferChuong H. Nguyen1, Tobias Ritschel2, Karol Myszkowski1, Elmar Eisemann2, Hans-Peter Seidel1
1Max-Planck-Institut Informatik 2Telecom ParisTech / CNRS
Eurographics 2012, Cagliari, Italy
Background – Material Perception
[Ce Liu et al. 2010]
Eurographics 2012, Cagliari, Italy
Background - Mood
Eurographics 2012, Cagliari, Italy
Background
Eurographics 2012, Cagliari, Italy
Motivation
Scene without MaterialsGuide Image Scene with
Materials
Eurographics 2012, Cagliari, Italy
Previous Works - Colorization
Color TransferReinhard et al: Color Transfer between Images, IEEE CG&A 2001
ColorizationWelsh et al: Transfering Color to Greyscale Images, Siggraph 2001Irony et al: Colorization by Examples, EGSR 2005
Eurographics 2012, Cagliari, Italy
Previous Works – Detail Synthesis
Improving the RealismJohnson et al: CG2Real: Improving the Realism of Computer Generated Images using a Large Collection of Photographs, IEEE TVCG 2010
AnalogyHertzmann et al: Image Analogies, Siggraph 2001Mertens et al: Texture Transfer Using Geometry Correlation, EGSR 2007
Eurographics 2012, Cagliari, Italy
Previous Work – Materials
Image-Based Material EditingKhan et al: Image-Based Material Editing , Siggraph 2006
Material PerceptionVangorp et al: The Influence of Shape on the Perception of Material Reflectance , Siggraph 2007
Eurographics 2012, Cagliari, Italy
Problem statement
Guide Source3D Scene
Objects Materials
Eurographics 2012, Cagliari, Italy
Material Extraction
Guide Source3D Scene
Objects Materials
Eurographics 2012, Cagliari, Italy
Material ExtractionOriginal
Diffuse
Specular
1st K-Mean Merging 2nd K-Mean
Extracted MaterialsTexture
Diffuse and Specular ExtractionYang et al: Real-time Specular Highlight Removal Using Bilateral Filtering, ECCV, 2010
Eurographics 2012, Cagliari, Italy
12
Material Extraction Results
Eurographics 2012, Cagliari, Italy
Material Assignment
Guide Source3D Scene
Objects Materials
Eurographics 2012, Cagliari, Italy
Material Assignment Goal
Guide Source
Objects Materials
1.
2.
(Image Cost)
(Geometry Cost)
Eurographics 2012, Cagliari, Italy
Material Assignment Goal – Image Cost
Guide Source
Objects Materials
1.
2.
(Image Cost)
(Geometry Cost)
Eurographics 2012, Cagliari, Italy
Material Assignment - Image CostGuide Source3D Scene
∑ ϴ
Eurographics 2012, Cagliari, Italy
Material Assignment Goal – Geometry Cost
Guide Source
Objects Materials
1.
2.
(Image Cost)
(Geometry Cost)
Eurographics 2012, Cagliari, Italy
3D Scene
Material Assignment - Geometry CostGuide Source
Eurographics 2012, Cagliari, Italy
Geometry Cost – Shape Descriptor2D Angular Radial Transform (ART)Bober et al: MPEG-7 visual shape descriptors, IEEE Circuits and Sys. Vid. Tech. 11, 6 (2001)
35 coefs (B)
35 coefs (A)35 coefs (C)
35 coefs (G)
35 coefs (H) 35 coefs(E) 35 coefs (D)
35 coefs (F)
35 coefs (O)
ϴ ¿min {‖𝐴−𝑂‖,‖𝐵−𝑂‖,‖𝐶−𝑂‖,.. } ,…
Eurographics 2012, Cagliari, Italy
Cost Function = Image Cost + Geometry Cost
ϴ
ϴ
ϴ
+
…
ϴ
ϴ
ϴ
…
Eurographics 2012, Cagliari, Italy
Material Assignment
Guide Source3D Scene
Objects Materials
Eurographics 2012, Cagliari, Italy
Optimization – Simulated Annealingϴ
ϴ
ϴ
+
…
ϴ
ϴ
ϴ
…
ϴ
ϴ
ϴ
+
…
ϴ
ϴ
ϴ
…
otherwise
assignment ith
assignment i+1th
Eurographics 2012, Cagliari, Italy
Results: Pebbles
Eurographics 2012, Cagliari, Italy
Results: Temple
Eurographics 2012, Cagliari, Italy
Results: Kitchen
Eurographics 2012, Cagliari, Italy
Results: Bedroom
Eurographics 2012, Cagliari, Italy
Perceptual Study – Step 1
Eurographics 2012, Cagliari, Italy
Perceptual Study – Step 2 60% best
Eurographics 2012, Cagliari, Italy
Limitations• Plausible material extraction• Image segmentation errors• Fix lighting
Eurographics 2012, Cagliari, Italy
Conclusion and Future Work
• Conclusion– Plausible material extraction from images– An automatic system for 3D material style transfer
• Future work– Material database for different objects– Better object recognition techniques
Eurographics 2012, Cagliari, Italy
Thank you!