application into image inpainting
DESCRIPTION
Application into Image Inpainting. Criminisi, A.; Perez, P.; Toyama, K., "Region filling and object removal by exemplar-based image inpainting," IEEE Trans. on Image Proc. , vol.13, no.9, pp. 1200- 1212, Sept. 2004. At the Intersection. Error Concealment, Image Recovery … …. - PowerPoint PPT PresentationTRANSCRIPT
Application into Image Inpainting
Criminisi, A.; Perez, P.; Toyama, K., "Region filling and object removal by exemplar-based image inpainting," IEEE Trans. on Image Proc., vol.13, no.9, pp. 1200- 1212, Sept. 2004
At the Intersection
Error Concealment, Image Recovery … …
Importance of Scanning Order
Heuristic Rules
Green/red color: high (low) priority pixels
Geometric cue: ConvexityPhotometric cue: gradient
Image Inpainting
SpecificScanning
order
Repair byNonparametric
samplingImage torepaired
Repairedimage
Synthetic Image Example
Kanisza Triangle
FinalIntermediateInitial
Real-world Image Example
Computer Assignment 5 Implement Efros-Leung’s scheme Test it on image inpainting (you
might want to try different textures) When does it work and when does it
fail? Computational complexity reduction
Open Questions Where to inpaint? Sequential vs. parallel inpainting Performance evaluation
Seeing is not always believing Failure examples
Few researchers report them Comparison with other approaches
Why does A work better than B?
Error Concealment in Image Communication
videoimage
Sequential Parallel
Why Parallel? Avoid choosing scanning order and its
associated error propagation problem Modeling multimedia signal at a patch
(e.g., speech frame, image block) level instead of data (e.g., speech sample, image pixel) level
Parallelism has to work with distributed processing together (redundancy is exploited to fight against uncertainty)
Why Nonlocal?
B4
B2
B3
B0
B1
RPP Local neighborsin range (photometricsimilarity)
nonlocal patchesin domain (geometricproximity)
Algorithm Flowchart
Projectiononto
observationconstraint
Projectionontoprior
constraint
Data clustering
in Rp
Coringalgorithm
Better estimate of local neighborhood in the patch space
Inner loopOuter loop
Better estimate of missing data in the time domain
Sparsity Enforcement
B0 BkB1 Pack into3D Array D 3D-FFT
Thresholding
…
3D-IFFTPack into
3D Array DB0 BkB1 …^ ^ ^
Projectionontoprior
constraint
Image Examples
27.06dB 29.14dB 30.16dB
original damagedexemplar
basedGuleryuz’s
schemeour
work
18.06dB 17.57dB
35.72dB 17.93dB 37.37dB
22.36dB
26.75dB 24.63dB 31.84dB
original damagedexemplar
basedGuleryuz’s
schemeour
work
26.52dB 25.84dB 33.08dB
28.78dB 28.09dB 34.26dB
21.21dB 14.66dB
17.83dB 17.66dB
26.58dB
20.24dB
Parallel and Distributed Processing (PDP) Dominant form of
Connectionism (models mental or behavioral phenomena as the emergent processes of interconnected networks of simple units.)
More widely known as neural networks since 1980s
Image Quilting for Texture Synthesis &
TransferAlexei Efros and
Bill Freeman (SIGGRAPH’2001)
++ ==
Pixel-level Nonparametric Sampling
Assuming Markov property, compute P(p|N(p)) Building explicit probability tables infeasible Instead, let’s search the input image for all similar
neighborhoods — that’s our histogram for p To synthesize p, just pick one match at random
pp
non-parametricsampling
Input image Synthesizing a pixel
Block-level Chaos Mosaic
Process: 1) tile input image; 2) pick random blocks and place them in random locations 3) Smooth edges
input
idea result
Comparison with Periodic Mosaic
periodic chaotic
input
result
Doesn’t work for structured textures
pp
Image Quilting=Nonparametric Sampling+Chaos Mosaic
Observation: neighbour pixels are highly correlated
Input image
non-parametricsampling
BB
Idea:Idea: unit of synthesis = block unit of synthesis = block• Exactly the same but now we want P(B|N(B))• Much faster: synthesize all pixels in a block at once• Not the same as multi-scale!
Synthesizing a block
Input texture
B1 B2
Random placement of blocks
block
B1 B2
Neighboring blocksconstrained by overlap
B1 B2
Minimal errorboundary cut
min. error boundary
Minimal error boundaryoverlapping blocks vertical boundary
__ ==22
overlap error
Search for the Shortest Path
start
end
An Elegant Solution Exhaustive search is
computationally infeasible Fortunately, there exists a fast and
optimum solution known as Dynamic Programming (DP) in Operations Research and Viterbi Algorithm (VA) in communication
DP/VA Example
start end
Basic Idea: Only keep the best Q survivors in sequential decision
DP/VA Step-by-Step Result
Image Quilting Algorithm Pick size of block and size of overlap Synthesize blocks in raster order
Search input texture for block that satisfies minimum error constraints (above and left)
Paste new block into resulting texture
++ ==
++ ==
++ ==
parmesan
rice
Summary for Texture Synthesis Parametric approach
Histogram/high order statistics matching in the wavelet/steerable pyramid space
Nonparametric approach Conceptually simple and effective
Interesting application into image inpainting & quilting
Produce visually convincing results Parametric vs. nonparametric: which is the
winner? It depends on the application How to combine them together?