andrew nealen and marc alexa, discrete geometric modeling group, tu darmstadt, 2003 hybrid texture...
Post on 21-Dec-2015
216 views
TRANSCRIPT
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Hybrid Texture SynthesisHybrid Texture Synthesis
Andrew NealenAndrew NealenMarc AlexaMarc Alexa
Discrete Geometric Modeling Group (DGM)Discrete Geometric Modeling Group (DGM)Technische Universität DarmstadtTechnische Universität Darmstadt
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
2D Texture Synthesis
nxm Input Texture
NxM Output Texture
► The goal: Synthesize an output texture which is perceptually similar to the input texture. Also ensure that the result contains sufficient variation.
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Existing (and Impressive) Technology• Pixel-Based:
─ Non-parametric Sampling [Efros and Leung 1999]─ Tree-structured Vector Quantization [Wei and Levoy 2000]─ Synthesizing Natural Textures [Ashikhmin 2001]─ Image Analogies [Hertzmann et al. 2001]
• Patch-Based:─ Patch-Based Sampling [Guo et al. 2001]─ Image Quilting [Efros and Freeman 2001]─ Hierarchical Pattern Mapping [Soler et al. 2002]
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Possible Drawbacks ?• Loss of global structure• Loss of scale• Boundary mismatch• Blurring
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Possible Drawbacks ?• Loss of global structure• Loss of scale• Boundary mismatch• Blurring
Wei/Levoy Algorithm:Can occur when using structured Textures with rich histograms.This is due to the L2 norm.
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Possible Drawbacks ?• Loss of global structure• Loss of scale• Boundary mismatch• Blurring
Patch-Based Sampling Algorithm:Can occur when texture features are not well-aligned across patch boundaries
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Possible Drawbacks ?• Loss of global structure• Loss of scale• Boundary mismatch• Blurring
► Is it possible to reduce all deficiencies for an even larger class of textures ?
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
Step n in Algorithm
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
Build L-shaped Neighborhood
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
Find Matching Neighborhood(s) in Input
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
Select Best Neighborhood from all Candidates
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
Copy Selected Pixel to Output
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
Repeat for all Pixels in this Row...
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input Output
...and all other Output Rows
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
► Main difference between Efros/Leung (EL), Wei/Levoy (WL) and Ashikhmin (AS): Per-Pixel Search Strategy.
► Ashikhmin’s insight: “WL tends to produce textures with smoothed out edges in some cases.”
► Ashikhmin’s solution: consider only coherent pixels in the search, which encourages region growing.
► Problem: inconsistencies can still occur for highly-structured textures.
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Step n in Algorithm
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Patch to be Synthesized in this Step
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Build Neighborhood (Image Mask)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Find Matching Neighborhood(s) in Input
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Select Best Neighborhood from all Candidates
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Copy Selected Patch to Output
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Apply some Overlap Repair Strategy ...
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
... And we are done with this patch
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input Output
Repeat for all Patches in Output
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture SynthesisGeneralization of Pixel-Based Synthesis
► Main (visual) difference between Image Quilting (IQ) and Patch-based Sampling (PBS): Overlap Repair Strategy.
► IQ performs a minimum-error-boundary-cut (MEBC) through the error surface (least cost path; Dijkstra).
► PBS feathers the overlap region (alpha-blending).
► Pro: patch-based algorithms are good at maintaining global structure.
► Problems: boundary mismatch (MEBC, top) and blurring (PBS, bottom).
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture SynthesisGeneralization of Pixel-Based Synthesis
► Main (visual) difference between Image Quilting (IQ) and Patch-based Sampling (PBS): Overlap Repair Strategy.
► IQ performs a minimum-error-boundary-cut (MEBC) through the error surface (least cost path; Dijkstra).
► PBS feathers the overlap region (alpha-blending).
► Pro: patch-based algorithms are good at maintaining global structure.
► Problems: boundary mismatch (MEBC, top) and blurring (PBS, bottom).
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture SynthesisGeneralization of Pixel-Based Synthesis
► Main (visual) difference between Image Quilting (IQ) and Patch-based Sampling (PBS): Overlap Repair Strategy.
► IQ performs a minimum-error-boundary-cut (MEBC) through the error surface (least cost path; Dijkstra).
► PBS feathers the overlap region (alpha-blending).
► Pro: patch-based algorithms are good at maintaining global structure.
► Problems: boundary mismatch (MEBC, top) and blurring (PBS, bottom).
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Hybrid Texture Synthesis (HTS)
Algorithm
► Hybrid, two-fold approach to texture synthesis• Adaptive patch sampling [Soler et al. 2002]
─ Start with a uniform quadrilateral grid of patches (the output)─ Recursively split a patch if the best fit taken from the input
texture is not good enough (a user-defined tradeoff max)
• Overlap re-synthesis (novel overlap repair strategy)─ Mark invalid pixels in the overlap region
─ Validity map is defined by the error surface E(x)=(xsrc – xdst)2
and a user-defined tolerance parameter max
─ Define an ordering for the invalid pixels and re-synthesize them using a per-pixel strategy
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch SamplingPrinciple
► Essentially: Hierarchical Pattern Mapping applied to the plane [Soler et al. 2002]
► Analogy: Assembling a quadtree puzzle
p > max Split PatchSynthesize White Patch p < max
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch SamplingTrade-off
► Trade-off between preserving global structure and avoiding detail artifacts
max = 1.0max = 0.01 max = 0.04
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch SamplingTrade-off
► Trade-off between preserving global structure and avoiding detail artifacts
max = 1.0max = 0.01 max = 0.04
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch SamplingTrade-off
► Trade-off between preserving global structure and avoiding detail artifacts
max = 1.0max = 0.01 max = 0.04
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch SamplingTrade-off
► Trade-off between preserving global structure and avoiding detail artifacts
max = 1.0max = 0.01 max = 0.04
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesSynthesizing a single patch
Synthesize Black Patch i
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesExtracting the image mask
Synthesize Black Patch i
► Grow patch by overlap (toroidally)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesExtracting the image mask
Image Mask (Ii)Synthesize Black Patch i Binary Support (Ji)
► Extract image mask (Ii) and binary support (Ji), and circularly shift to upper left corner
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesComputing the error image
Image Mask (Ii)Input Texture (T) Binary Support (Ji)
► Compute error Ei(x0) between Ii and T for each circular shift x0=(x,y) of the input texture T
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesComputing the error image
Error Image (Ei)Input Texture (T)
x
xxxxxc
cicicii
i TIwJκ
E 20,,0 ))()(()(
1)(
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesSelecting a Patch
Error Image (E)Input Texture (T) Selected Patch (P)
x
xxxxxc
cicicii
i TIwJκ
E 20,,0 ))()(()(
1)(
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good PatchesSelecting a Patch
Error Image (E)Input Texture (T) Selected Patch (P)
► Note: error image can be computed efficiently in the Fourier domain. Complexity: O(n log n) per patch.
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisOverlap error and pixel invalidation
Patch and Image Mask Selected Patch (P)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisOverlap error and pixel invalidation
Image Mask (I) Selected Patch (P)
c
cicicii PIwJS 2,, ))()(()()( xxxx
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisOverlap error and pixel invalidation
Image Mask (I) Selected Patch (P)Error Surface (S)
c
cicicii PIwJS 2,, ))()(()()( xxxx
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisOverlap error and pixel invalidation
Patch and Image Mask P with Invalid Pixels (blue)Error Surface (S)
► Trade-off: user-defined max is the pixel error tolerance. Setting to 1 results in pure feathering.
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisCompositing
Intermediate Result Patch with Invalid Pixels
P OVER Ri-1
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisCompositing
Composited Result Patch with Invalid Pixels
► Simple scanline re-synthesis: possibly insufficient causal neighborhood within valid pixel region
► Solution: alternative ordering (Pixel Traversal Map)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
► Pixel Traversal Map: repeated morphological dilation of the binary support for valid pixels with a euclidian disk of radius 1 (city-block distance transform).
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
► Pixel Traversal Map: step 1
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
► Pixel Traversal Map: step 2 ...
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
► Pixel Traversal Map: ... step n
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPixel Traversal Map (Ordering)
Composited Result Patch with Invalid PixelsPixel Traversal Map (M)
► Art Restorer Analogy: stepwise restoration of the hole from the boundary of the existing image.
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Pixel Traversal Map (M)
► Synthesize red pixel from valid (or already re-synthesized) pixels
Intermediate Result
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Image Mask (Ij)
Binary Support (Jj)
► Extract image mask (Ij) and binary support (Jj) for best-pixel search in the input texture. Complexity: O(n log n) per pixel.
Intermediate Result
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Intermediate Result Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Intermediate Result Shifted
Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Intermediate Result with Traversal Map
Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesisStep 1
Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesisStep 2
Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesisStep 3
Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesisPer-Pixel Re-synthesis
Overlap Re-synthesisResult
Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsOverlap Repair Comparisons
No Repair HTSPBSIQ
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
Efros/Leung Wei/Levoy
IQ PBS HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
PBS HTS
PBS HTS
Input
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
PBS HTS
PBS HTS
Input
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
ResultsSynthesis Comparisons
Input
PBS HTS
PBS HTS
Input
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Results
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Conclusions and Future Work
► Improve Computational Complexity• Pixel neighborhoods of patch and pixel stage are not
known a priori, so precomputation is not straightforward• We can solve this in the pixel stage by employing k-
coherence search [Tong et al. 2002] [Ashikhmin 2001]
► Improve Error Metric• Still using the L2 norm due to its simplicity
• Develop a metric which takes feature mismatch into account
• Texton approach looks promising [Zhang et al. 2003]
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Questions ?
► Contact Information
Andrew [email protected]
Marc [email protected]
http://www.dgm.informatik.tu-darmstadt.de
Matlab code and extended Thesis:http://www.nealen.com