model-driven 3d content creation as variation
DESCRIPTION
NUDT. TAU. ZJU. SFU. Model-Driven 3D Content Creation as Variation. Hao (Richard) Zhang – 张皓 GrUVi Lab, Simon Fraser University (SFU) Talk @ HKUST, 04/21/11. 3D content creation. Inspiration?. Inspiration a readily usable digital 3D model. Realistic reconstruction. - PowerPoint PPT PresentationTRANSCRIPT
Model-Driven 3D Content Creation as
Variation
Model-Driven 3D Content Creation as
VariationHao (Richard) Zhang – 张皓
GrUVi Lab, Simon Fraser University (SFU)
Talk @ HKUST, 04/21/11
Hao (Richard) Zhang – 张皓GrUVi Lab, Simon Fraser University (SFU)
Talk @ HKUST, 04/21/11
TAUTAU ZJUZJUNUDTNUDTSFUSFU
3D content creation
Inspiration a readily usable digital 3D modelInspiration a readily usable digital 3D model
Inspiration?Inspiration?
Realistic reconstruction
• Inspiration = real-world data
[Nan et al., SIGGRAPH 2010][Nan et al., SIGGRAPH 2010]
Creative inspiration
•Creation of novel 3D shapes
• Inspiration = design concept, mental
picture, …
sketchsketch
High demand in VFX, games, simulation, VR, …
High demand in VFX, games, simulation, VR, …
3D content creation is hard
•2D-to-3D: an ill-posed problem
▫Shape from shading, sketch-based modeling, …
•Creation from scratch is hard: job for skilled
artistsOne of the most central problems in
graphics; One of the most discussed at SIG’10 panel
One of the most central problems in graphics; One of the most discussed at
SIG’10 panel
Usable 3D content even harder
•Models created are meant for subsequent
use
•Creation of readily usable 3D models
Usable 3D content even harder
•Models created are meant for subsequent use
•Creation of readily usable 3D models
•Higher-level information beyond low-level mesh
▫Part or segmentation information
▫Structural relations between parts
▫Correspondence to relevant models, etc.Hard shape analysis problems!Hard shape analysis problems!
Key: model reuse
•Reuse existing 3D models and associated
info
•Model-driven approach: creation is driven
by or based on existing (pre-analyzed)
models
Key: model reuse
•Reuse existing 3D models and associated
info
•Model-driven approach: creation is driven
by or based on existing (pre-analyzed)
models
•Two primary modes of reuse:
▫New creation via part composition
Key: model reuse
•Reuse existing 3D models and associated info
•Model-driven approach: creation is driven by
or based on existing (pre-analyzed) models
•Two primary modes of reuse:
▫New creation via part composition
▫New creation as variation or modification of
existing model(s), e.g., a warp or a
deformation
Modeling by example
•New models composed by parts retrieved
from an existing data repository
•Key: retrieve relevant parts
•Many variants …
[Funkhouser et al., SIGGRAPH 2004][Funkhouser et al., SIGGRAPH 2004]
Pros and cons
•Pros:▫Significant deviation from existing models
▫Exploratory modeling via part suggestions
[Chaudhuri & Koltun., SIG Asia 2010][Chaudhuri & Koltun., SIG Asia 2010]
Pros and cons
•Pros:▫Significant deviation from existing models
▫Exploratory modeling with part suggestions
•Cons:▫Are models composed by parts readily
usable?
Pros and cons
•Pros:▫Significant deviation from existing models
▫Exploratory modeling with part suggestions
•Cons:▫Are models composed by parts readily
usable? structure lost by part composition; how to stitch?
Pros and cons
•Pros:▫Significant deviation from existing models
▫Exploratory modeling with part suggestions
•Cons:▫Are models composed by parts readily
usable? structure lost by part composition; how to stitch?
▫Does part exploration always reflect user design intent?
Model-driven creation as variation
•New creation as variation of existing model(s)
Enrich a set; generate “more of
the same” …
Enrich a set; generate “more of
the same” …
Photo-inspired 3D model creation
Photo-inspired 3D model creation
Inspiration = photographsInspiration = photographs
Inspiration = a model set
Inspiration = a model set
Model-driven creation as variation
•New creation as variation of existing model(s)
Enrich a set; generate “more of
the same” …
Enrich a set; generate “more of
the same” …
Inspiration = a model set
Inspiration = a model set
Style-Content Separation by Anisotropic Part Scales
Kai Xu1,2, Honghua Li2, Hao Zhang2, Daniel Cohen-Or3
Yueshan Xiong2, and Zhi-Quan Cheng2
1Simon Fraser Universtiy 2National Univ. of Defense Tech. 3Tel-Aviv University
Motivation
•Enrich a set of 3D models with their derivatives
Set belongs to the same family or class
Set belongs to the same family or class
Variations in shape parts in the set
Geometric or content difference
Part proportion (= style) difference
?
Style transfer as a derivative
Part proportion style
?
Style transfer as a derivative
Part proportion style
Difficulty with style transfer
•Style transfer needs part correspondence
•Part correspondence is difficult
▫Unsupervised problem
▫Both content and style variations
Variations can be significant!
Work at part and OBB level
Parts enclosed and characterized by tight oriented bounding boxes (OBBs)Parts enclosed and characterized by tight oriented bounding boxes (OBBs)
Style content separation
•To address both shape variations in the set▫Separate treatment of “style” and “content”
Style 1
Style 2
Style 3
ContentContent
Sty
leS
tyle
Style transfer as a derivative
•Creation = filling in the style-content table
Style vs. content
•Fundamental to human perception
Content Style
Language Words Accents
Text Letters Fonts
Human face Identities Expressions
Style content separation
•Previous works on faces, motion, etc.
▫Prerequisite: data correspondence
▫Correspondence dealt with independently
▫Correspondence itself is the very challenge!
Our approach
• One particular style:
Anisotropic part scales or part proportions
Our approach
• One particular style:
Anisotropic part scales or part proportions
• The approach:
Style-content separation with style
clustering in a correspondence-free way
Algorithm overview
•Pipeline
Style clustering Co-segmentation Inter-style part correspondence
Contentclassification
Anisotropic part scales
•Measure style distance between two shapes
Anisotropic part scales
•Measure style distance between two shapes
Part OBB graphs of
given segmentatio
n
Anisotropic part scales
•Measure style distance between two shapes
Computestyle
signatures
……
Part OBB graphs of
given segmentatio
n
Anisotropic part scales
•Measure style distance between two shapes
……
Part OBB graphs of
given segmentatio
n
Euclideandistance
Computestyle
signatures
Style distance issues
•Unknown segmentation
•Unknown correspondence
?
?
Style distance
•Search over all part compositions and part counts
……
……
Style distance
•For each part count, find minimal distance
……
……
A good signature will return min distance across all part counts to reflect corresponding part decompositions …
Correspondence-free style signature
Binary relations: difference of part scales between adjacent OBBs
Use Laplacian graph spectra:
OBB graph
Correspondence-free style signature
Unary attributes: anisotropy of parts
Use Laplacian graph spectra:
OBB graph linear planar spherical
Graph spectra is permutation-free
Style clustering
•Spectral clustering using style distances
Pipeline
Style clustering Co-segmentation Inter-style part correspondence
Contentclassification
Co-segmentation
•Approach:▫Consistent segmentation [Golovinskiy & Funkhouser,
SMI 09]
▫ Initial guess: global alignment (ICP)
[Golovinskiy & Funkhouser 09]
Co-segmentation
•Approach:▫Consistent segmentation [Golovinskiy & Funkhouser, SMI 09]
▫ Initial guess: global alignment (ICP)
•We co-segment within a style cluster▫Removing non-homogeneous part scaling from analysis
[Golovinskiy & Funkhouser 09]
Co-segmentation
•Approach:▫Consistent segmentation [Golovinskiy & Funkhouser, SMI 09]
▫ Initial guess: global alignment (ICP)
•We co-segment within a style cluster▫Removing non-homogeneous part scaling from analysis
[Golovinskiy & Funkhouser 09] After style separation
Pipeline
Style clustering Co-segmentation Inter-style part correspondence
Contentclassification
Inter-style part correspondence
•Approach: deform-to-fit
▫Deformation-driven correspondence [Zhang et al., SGP 08]
▫Consider common interactions between OBBs
1D-to-1D 1D-to-2D 2D-to-2D 2D-to-3D
Inter-style part correspondence
•Deform-to-fit: appropriate deformation energy
Pruned priority-driven search
Pipeline
Style clustering Co-segmentation Inter-style part correspondence
Contentclassification
Content classification
•Use Light Field Descriptor [Chen et al. 2003]
•Compare corresponding parts
Part-level LFD Global LFD
Synthesis by style transfer
•OBBs are scaled
•Underlying geometry via space
deformationcontent
style style transfer
Results: hammers
Results: goblets
Results: humanoids
Results: chairs
Pros and cons
•Pros:▫Automatic generation of many variations
▫Unsupervised
▫Deals with anisotropic part scales
▫Variation = part scaling: structure preservation
Pros and cons
•Pros:▫Automatic generation of many variations
▫Unsupervised
▫Deals with anisotropic part scales
▫Variation = part scaling: structure preservation
•Cons:▫Rely on sufficiently good initial segmentations
▫Variation does not create new content
Interesting future work
•Learn and synthesize with generic styles
Model-driven creation as variation
•New creation as variation of existing model(s)
Photo-inspired 3D model creation
Photo-inspired 3D model creation
Inspiration = photographsInspiration = photographs
Photo-inspired 3D modeling
Photo-Inspired Model-Driven 3D Object Modeling
Kai Xu1,2, Hanlin Zheng4, Hao Zhang2, Daniel Cohen-Or3
Ligang Liu4, and Yueshan Xiong2
1NUDT 2SFU 3TAU 4ZJU Conditionally acceptedConditionally accepted
Overview
Input: single photograph + pre-analyzed datasetInput: single photograph + pre-analyzed dataset
Overview
1. Model-driven labelled segmentation of photographed object
1. Model-driven labelled segmentation of photographed object
Overview
2. Choosing of a candidate model from the database
2. Choosing of a candidate model from the database
Overview
3. Silhouette-constrained deform-to-fit of candidate
3. Silhouette-constrained deform-to-fit of candidate
Overview
OutputOutput
Structure preservation
•Any higher-level structural info in the candidate
models is preserved during deform-to-fit
▫Symmetry relations
▫Part-level correspondence in the set
▫Controller structures [Zheng et al. @ HKUST, EG 11]
Structure preservation
•Any higher-level structural info in the candidate
models is preserved during deform-to-fit
▫Symmetry relations
▫Part-level correspondence in the set
▫Controller structures [Zheng et al. @ HKUST, EG 11]
•Structures also serve to constrain deformation of
candidate model
Controller representations
•Controllers: cuboids and generalized cylinders
•Relations: symmetry, proximity, etc.
Fitting primitivesFitting primitives
Controller representations
•Controllers: cuboids and generalized cylinders
•Relations: symmetry, proximity, etc.
Fitting primitivesFitting primitives
Deformation of controllers
photophoto
Controller primitivesController primitives
Deformation of controllers
photophoto candidate modelcandidate model
Controller primitivesController primitives
Deformation of controllers
Result of silhouette-driven deform-to-fit
Result of silhouette-driven deform-to-fit
photophoto candidate modelcandidate model
Structure preservation at work
symmetrysymmetry
Structure preservation at work
symmetrysymmetry
proximityproximity
Structure preservation at work
symmetrysymmetry
proximityproximity
optimizationoptimization
Structure preservation at work
symmetrysymmetry
proximityproximity
optimizationoptimization
outputoutput
Short videoShort video
Results
•Guidance in single view but coherent 3D results
Results
The Google chair challenge
Not just chairs …
Pros and cons
•Pros:▫Photos: immensely rich source of inspiration
▫Silhouette-driven deformation
▫Variation is less “intrusive” to retain high-level info of source model readily usable
Pros and cons
•Pros:▫Photos: immensely rich source of inspiration
▫Silhouette-driven deformation
▫Variation is less “intrusive” to retain high-level info of source model more readily usable
•Cons▫Variation does not create new structures
Future work
•Photo-inspired model deformation only a start
•Further model refinement, e.g., via sketches
Future work
•Photo-inspired model deformation only a start
•Further model refinement, e.g., via sketches
•Model-driven structure modification
Future work
•Photo-inspired model deformation only a start
•Further model refinement, e.g., via sketches
•Model-driven structure modification
•Other inspirations for 3D content creation
▫Sketch-inspired model variation
Future work
•Photo-inspired model deformation only a start
•Further model refinement, e.g., via sketches
•Model-driven structure modification
•Other inspirations for 3D content creation
▫Sketch-inspired model variation
•Style transfer with unknown style in a set
Thank you, 谢谢
TAUTAU ZJUZJUNUDTNUDTSFUSFU