real-time exemplar-based face sketch synthesis
DESCRIPTION
Real-Time Exemplar-Based Face Sketch Synthesis. Pipeline illustration. Qingxiong Yang 1. Ming-Hsuan Yang 2. Yibing Song 1. Linchao Bao 1. 1 City University of Hong Kong. 2 University of California at Merced. Note: containing animations. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Real-Time Exemplar-Based Face Sketch Synthesis](https://reader035.vdocument.in/reader035/viewer/2022081511/56816678550346895dda1484/html5/thumbnails/1.jpg)
Real-Time Exemplar-Based Face Sketch Synthesis
Pipeline illustration
Note: containing animations
Yibing Song1 Linchao Bao1 Qingxiong Yang1 Ming-Hsuan Yang2
1City University of Hong Kong2University of California at Merced
![Page 2: Real-Time Exemplar-Based Face Sketch Synthesis](https://reader035.vdocument.in/reader035/viewer/2022081511/56816678550346895dda1484/html5/thumbnails/2.jpg)
Our assumption: a database containing photo-sketch pairs
1. photo database 2. sketch database
Aligned
![Page 3: Real-Time Exemplar-Based Face Sketch Synthesis](https://reader035.vdocument.in/reader035/viewer/2022081511/56816678550346895dda1484/html5/thumbnails/3.jpg)
Coarse Sketch GenerationStep 1: KNN search
p
Test photo patch Test photo
Training photo dataset
๐ป ๐๐ป ๐
๐ป ๐
Matched photo patch
Relative position
Similarly
Matched photo patch
Relative position
โ๐๐ฒ[ ]โ๐ =
![Page 4: Real-Time Exemplar-Based Face Sketch Synthesis](https://reader035.vdocument.in/reader035/viewer/2022081511/56816678550346895dda1484/html5/thumbnails/4.jpg)
Test photo patch
Matched photo patch
Matched photo patch
Matched photo patch
๐๐๐ โ +๐๐
๐ โ +๐๐๐ฒ โ ยฟ
2. Compute linear mapping function defined by
Coarse Sketch GenerationStep 2: Linear Estimation from Photos
![Page 5: Real-Time Exemplar-Based Face Sketch Synthesis](https://reader035.vdocument.in/reader035/viewer/2022081511/56816678550346895dda1484/html5/thumbnails/5.jpg)
Matched sketch pixel
p
Matched sketch pixel
Test photo
๐บ๐ท +โ๐๐
โ
๐บ๐ท +โ๐๐
โ
Matched sketch pixel๐บ๐ท +โ๐๐ฒ
โ
๐๐๐ โ +๐๐
๐ โ +๐๐๐ฒ โ ยฟ
Estimation on pixel p
Repeat for every pixel
Coarse sketch
Coarse Sketch GenerationStep 3: Apply Linear Mapping to Sketches
๐ฌ๐
![Page 6: Real-Time Exemplar-Based Face Sketch Synthesis](https://reader035.vdocument.in/reader035/viewer/2022081511/56816678550346895dda1484/html5/thumbnails/6.jpg)
Because: coarse sketch image is not natural. is not a good similarity measurement between p and r.
Denoising: State-of-the-art Image Denoising Algorithms
Coarse sketch
Nonlocal Means (NLM)
p
r
๐๐๐๐ฟ๐=ยฟ ๐ธ๐
๐ค(๐ ,๐ )+โฏ
For all pixels in the neighbor of p:
Little improvement
After NLM
q
๐ธ๐๐ค(๐ ,๐)+ยฟ
[NLM] A. Buades, B. Coll and J.-M. Morel, A non-local algorithm for image denoising, CVPR 2005.
![Page 7: Real-Time Exemplar-Based Face Sketch Synthesis](https://reader035.vdocument.in/reader035/viewer/2022081511/56816678550346895dda1484/html5/thumbnails/7.jpg)
Motivation โ BM3D
BM3D groups correlated patches in the noisy image to create multiple estimations.
Our idea for sketch denoising: group highly similar sketch estimations.
How BM3D works
[BM3D] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, โImage denoising by sparse 3D transform-domain collaborative filtering,โ IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080-2095, August 2007.
![Page 8: Real-Time Exemplar-Based Face Sketch Synthesis](https://reader035.vdocument.in/reader035/viewer/2022081511/56816678550346895dda1484/html5/thumbnails/8.jpg)
๐ค(๐ ,๐) โ
Proposed Spatial Sketch Denoising Algorithm (SSD)
Test photo
q
๐บ๐+โ๐๐
โ
p
Matched sketch
๐บ๐+โ๐๐
โ
Similarly ,
๐บ๐+โ๐๐
โ
๐บ๐+โ๐๐
โ
,
๐บ๐+โ๐๐ฒ
โ
๐บ๐+โ๐๐ฒ
โ๐๐๐ โ +๐๐
๐ โ +๐๐๐ฒ โ ยฟ
๐ฌ๐๐
p
Estimations from pixels in local region
r ๐ฌ๐๐
Averaging estimations to generate output sketch value.
Nonlocal Means (NLM):
๐๐๐๐ฟ๐=ยฟ ๐ธ๐ +โฏ๐ธ๐๐ค(๐ ,๐ )โ+ยฟ
Proposed SSD:
๐๐๐๐๐ท=ยฟ +โฏ+ยฟ1 โ๐ฌ๐
๐ 1 โ๐ฌ๐๐
![Page 9: Real-Time Exemplar-Based Face Sketch Synthesis](https://reader035.vdocument.in/reader035/viewer/2022081511/56816678550346895dda1484/html5/thumbnails/9.jpg)
p
Proposed SSD is robust to
Input 5x5 local region
11x11 local region
17x17 local region
23x23 local region
Note: When is sufficient large (i.e., >100), the proposed SSD can effectivelysuppress noise while preserving facial details like the tiny eye reflections (see close-ups).
Robustness to the region size - the only parameter involved