Texture Synthesis from Multiple Sources

Texture Synthesis from Multiple Sources

Li-Yi WeiStanford University (was) NVIDIA Corporation (now)

Texture Synthesis(Single-Source) Texture Synthesis(Single-Source)

Input

Output

Synthesis

Tong et. al. 2002Soler et. al. 2002

Hertzmann et. al. 2001Efros & Freeman 2001

Yin et. al. 2001Turk 2001

Wei & Levoy 2001 Wei & Levoy 2000

Portilla & Simoncelli 1999Efros & Leung 1999

Heeger & Bergen 1995De Bonet 1997

Limitations of Single-Source SynthesisLimitations of Single-Source Synthesis

• Non-uniform, varying patterns– Junction of 2 textures– Varying scale, orientation, color, shape– Creating new textures

• Textures of different dimensions– Solid textures from 2D views

Multiple-Source Texture SynthesisMultiple-Source Texture Synthesis

• Solid textures from multiple 2D views

View 3

View 2

View 1

• Texture mixtures from multiple sources

+ = ?Source 1 Source 2

Previous WorkPrevious Work

• Solid texture from 2D views– Heeger & Bergen 1995– Ghazanfarpour & Dischler 1999

Images from [Heeger&Bergen 1995]

2D Source 3D result

Previous WorkPrevious Work

• Texture varying, morphing, and mixture– Portilla & Simoncelli 1999– Bar-Joseph, El-Yaniv, Lichinski, Werman 2001– Z. Liu, C. Liu, Shum, Yu 2003– PVT paper in SIGGRAPH 2003

Image from [Portilla&Simoncelli 1999]

Source 1 Source 2morphing

AlgorithmAlgorithm

Source

Synthesis Result

N(p)

search

N(pi)

Single-source algorithm [ICCV 99, SIGGRAPH 2000]

copy p

AlgorithmAlgorithm

Source 1

Source 2

Source 3

Synthesis Result

search

search

search

N1(p1)

N2(p2)

N3(p3)

E(p, {pi}) = Σ wi ×(|p-pi|2 + |Ni(p)-Ni(pi)|2)i= 1,2,3

L2 distance

Fix p, search {pi}Fix {pi}, set p = Σ wi×pi

User weights

Average

AlgorithmAlgorithm

Source 1

Source 2

Source 3

Synthesis Result

search

search

search

E(p, {pi}) = Σ wi ×(|p-pi|2 + |Ni(p)-Ni(pi)|2)i= 1,2,3

L2 distance

Fix p, search {pi}Fix {pi}, set p = Σ wi×pi

User weights

Average

Texture Mixture ExampleTexture Mixture Example

+ = Uniform

Transition

Use weights wi to control the result

Source 1 Source 2

Texture Mixture ResultsTexture Mixture Results

Source1 Source2 Mixture-uniform Mixture-transition

Texture Mixture ResultsTexture Mixture Results

Source1 Source2 Mixture-uniform Mixture-transition

Solid Texture SynthesisSolid Texture Synthesis

View 3View 2

View 1

Use {Ni} with different orientationsN2

N3

N1

Specifying ViewsSpecifying Views

Input

Views Result

×Inconsistent

О

О

Solid Texture ResultsSolid Texture Results

Solid Texture ResultsSolid Texture Results

Comparison : Surface/Volume SynthesisComparison : Surface/Volume Synthesis

Surface[SIGGRAPH 2001]

Volume[this work]

ConclusionConclusion

• Single-source synthesis– mostly mature and well done– limited capability

• Multiple-source synthesis– hard (patch-copying won’t work!)

– more useful– requires more work!

Future WorkFuture Work

• Combine the texton mask idea in SIGGRAPH 2003 PVT paper with this algorithm?

ENDEND

AlgorithmAlgorithm

1. Fix p, search {pi} to minimize |Ni(p)-Ni(pi)|2

2. Fix {pi}, set p = Σ wi×pi

Goal : Minimize E(p, {pi}) = Σ wi ×(|p-pi|2 + |Ni(p)-Ni(pi)|2)Step 1Step 2

How :

iterate

ComparisonComparison

Reusable Distortion Quality Efficiency Tunable

Surface better better better

Volume better better