cse 659a: advances in computer visionayan/courses/cse659a/pdfs/lec14.pdf · neural style transfer...
TRANSCRIPT
![Page 1: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/1.jpg)
CSE 659A: Advances inComputer Vision
Spring 2020: T-R: 1:00-2:20pm @ Whitaker 216
Instructor: Ayan Chakrabarti ([email protected]).
Feb 27, 2020
http://www.cse.wustl.edu/~ayan/courses/cse659a/
1
![Page 2: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/2.jpg)
COURSE ADMINCOURSE ADMIN
Reminder: Project 1 Report Deadline Extended. Now due March 17th.
Reminder: Tuesday's Class (March 2) canceled.
2
![Page 3: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/3.jpg)
COMPUTATIONAL PHOTOGRAPHYCOMPUTATIONAL PHOTOGRAPHY
Broadly defined as algorithms that enable high-capability generation of images and other visualmeasurements, through improved camera design and/or providing advanced image editing abilities.
Intersection of Three disciplines
Optics - Vision - Graphics
Optics: Visual Measurement
Vision: Visual Inference
Graphics: Visual Generation
Two kinds of Comp. Photography Work
Intersection of Optics + Vision: Better cameras that make inference easier
Intersection of Vision + Graphics: Generate better images using vision-based reasoning
We'll first talk about the latter.
3
![Page 4: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/4.jpg)
CG2REALCG2REAL
The problem: Photo-realistic scenes take many many hours to render.
Need to have complex shape and texture models for a scene, simulate complex light-transport (inter-
reflections, scattering, etc.) by ray tracing.
A simpler rendering on the other hand doesn't look real.
4
![Page 5: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/5.jpg)
CG2REALCG2REAL
Use a simpler renderer, and "transfer" detail by looking at a dataset of real images.
5
![Page 6: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/6.jpg)
CG2REALCG2REAL
Johnson et al., "CG2Real: Improving the Realism of Computer Generated Images using a Large Collection
of Photographs," IEEE Transactions on Visualization and Computer Graphics, 2010.
1. Find Similar Images
Use SIFT and other scene features to recover a small set of similar images from a real image dataset, given aquery 'rendered' image.
Similar idea to content-based image retrieval: except that the query is a rendered image.
Incorporate user input: automatically select 30 images, and have a user select the best 5.
6
![Page 7: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/7.jpg)
CG2REALCG2REAL
Johnson et al., "CG2Real: Improving the Realism of Computer Generated Images using a Large Collection
of Photographs," IEEE Transactions on Visualization and Computer Graphics, 2010.
2. Co-Segmentation: Find corresponding regions to transfer local detail and color properties.
For each potential real image, consider the pair of (rendered, real) image and do a co-segmentation.
Assign every pixel in each image a label to minimize an energy which promotes:
Same label for spatial neighbors in each image independently.
Same label for pixels with similar appearance across images.
7
![Page 8: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/8.jpg)
CG2REALCG2REAL
Johnson et al., "CG2Real: Improving the Realism of Computer Generated Images using a Large Collection
of Photographs," IEEE Transactions on Visualization and Computer Graphics, 2010.
3. Local Style Transfer
Now that you have matched real regions to rendered regions, transfer color and texture properties.
Color and tone transfer by matching histograms of intensities and gradients across regions.
Texture transfer by aligning different shi�ed copies of real region with rendered region, and solve a graph-cuts problem to figure out which shi�ed copy to transfer gradients from at each location.
8
![Page 9: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/9.jpg)
CG2REALCG2REAL
Johnson et al., "CG2Real: Improving the Realism of Computer Generated Images using a Large Collection
of Photographs," IEEE Transactions on Visualization and Computer Graphics, 2010.
9
![Page 10: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/10.jpg)
CG2REALCG2REAL
Johnson et al., "CG2Real: Improving the Realism of Computer Generated Images using a Large Collection
of Photographs," IEEE Transactions on Visualization and Computer Graphics, 2010.
10
![Page 11: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/11.jpg)
CG2REALCG2REAL
Example of where computational image generation can be done by approximate image-analysis rather than'rendering'.
In general, there is a large class of methods that allow you to manipulate images in this way.
Instead of "inverse rendering" an image to get a full description of the scene (texture, material, depth,lighting, etc.), modifying that description, and re-rendering ...
approximate with operations that work directly in the image domain.
Sometimes called image-based rendering.
11
![Page 12: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/12.jpg)
SEAM CARVINGSEAM CARVING
Avidan and Shamir, "Seam Carving for Content-Aware Image Resizing," ACM ToG, 2007.
Consider the case where you are given an image, and want to change it's aspect ratio: from landscape toportrait.
Option 1: Just do imresize.
But this warps the image and makes things look stretched out.
12
![Page 13: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/13.jpg)
SEAM CARVINGSEAM CARVING
Avidan and Shamir, "Seam Carving for Content-Aware Image Resizing," ACM ToG, 2007.
Option 2: Do a crop.
But this is sub-optimal. You might be forced to crop out interesting things in the image.
In this case, you might have wanted to show the shape of the rock and its boundary with the sea.
But the original image has a lot of redundant information. Don't need to see as much of the sea or the beach.
Could we remove 'un-interesting' pixels, in a way that overall content structure is preserved ?
13
![Page 14: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/14.jpg)
SEAM CARVINGSEAM CARVING
Avidan and Shamir, "Seam Carving for Content-Aware Image Resizing," ACM ToG, 2007.
Option 3: Seam Carving !
14
![Page 15: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/15.jpg)
SEAM CARVINGSEAM CARVING
Basic idea, to reduce the height (or width) by one pixel, find an "optimal" line of pixels from le�-to-right (top-to-bottom) to delete from the image.
This line is determined by solving an optimization problem: minimize the gradient magnitudes at all pixelsyou are deleting, while making sure they form a connected line.
To decrease height by N pixels, apply this procedure successively N times.
15
![Page 16: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/16.jpg)
SEAM CARVINGSEAM CARVING
Let's say we want to decrease height.
Define an energy function
Sum of magnitudes of x- and y- derivatives.
The idea is that if we delete a pixel in the image with low , it won't be "missed".
Now our goal is to find a horizontal line defined as:
Where .
Such that is minimized. This is the energy of the line or "seam".
Once we have found the optimal seam, we delete it by shi�ing all pixels up:
Same idea applies for finding a 'vertical' seam.
e(x, y) = | ∗ I(x, y)| + | ∗ I(x, y)|∇x ∇y
(x, y) e
(1, ), (2, ), (3, ),… (W , )y1 y2 y3 yW
| − | ≤ 1yx yx−1
e(x, )∑W
x=1 yx
Inew = np.zeros(np.shape(I[:-1,:]))
for x in range(W):
Inew[:y[x],x] = I[:y[x],x]
Inew[y[x]:,x] = I[y[x]+1:,x]
16
![Page 17: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/17.jpg)
SEAM CARVINGSEAM CARVING
This is an optimization over discrete pixel locations , where each .
You can solve this by dynamic programming. How ?
Viterbi Decoder
Initialization:
Go from le�-to-right:
Once you've reached the end, gives you the end of the optimal seam.
Backtrack by looking at to get the rest of the line.
, ,… = argmin e(x, ), | − | ≤ 1, ∀xy1 y2 yW ∑x=1
W
yx yx yx−1
{ }yx ∈ {1,…H}yx
M(1, y) = e(1, y) ∀y
M(x, y) = e(x, y) + min {M(x − 1, y − 1),M(x − 1, y),M(x − 1, y + 1)}
C(x, y) = argmin {M(x − 1, y − 1),M(x − 1, y),M(x − 1, y + 1)}
(W , arg M(W , y))miny
C(x, y)
17
![Page 18: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/18.jpg)
SEAM CARVINGSEAM CARVING
Original Image and
Crop: Find optimal crop (contiguous rectangle kept) that minimizes of removed pixels.
Column: remove vertical lines that minimize .
Seam Carving: Arbitrary connected lines that go from top-bottom.
Pixel: Remove pixels independently.
e(x, y)
e(x, y)
e(x, y)
18
![Page 19: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/19.jpg)
SEAM CARVINGSEAM CARVING
Now let's say you want to reduce both height and width: from to
Option 1: Uniformly scale by and then do seam-carving to fix aspect-ratio.
, . Carve vertical seams.
, . Carve horizontal seams.
Option 2: Remove both horizontal and vertical seams.
But in what order ? Select order to minimize energy of all removed pixels (computed on the image they were
removed from).
Define as the cost of reducing width and height by and pixels respectively. is correspondingreduced image.
, is .
Given and (unless or are 0, in which case consider only one)
Consider reducing width from or height from .
The corresponding energies will be or plus the energy of their respective seams.
Pick whichever is lower, and set and respectively.
Do this in order till you get to for your desired reduction size.
(H,W) ( , )H ′ W ′
s
sH = H ′ sW ≥ W ′
sW = W ′ sH ≥ H ′
(H − )H ′ (W − )W ′
T(r, c) r c Ir,c
T(0, 0) = 0 Ir,c I
T(r − 1, c) T(r, c − 1) r c
Ir−1,c Ir,c−1
T(r − 1, c) T(r, c − 1)
T(r, c) Ir,c
T(r, c)
19
![Page 20: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/20.jpg)
SEAM CARVINGSEAM CARVING
Content Amplification
Let's say you don't want to reduce the size of your image. Just make the content in it larger.
Scale up by uniform scaling, and then use seam carving to get back to original size.
20
![Page 21: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/21.jpg)
SEAM CARVINGSEAM CARVING
Seam Insertion
You can also use seam carving to "increase" width or height, by expanding instead of deleting seams.
Find all the seams in your image. And now, instead of deleting them, insert additional pixels equal to theaverage of the pixels on either side of the seam.
(Le�) Non-uniform Scaling. (Right) Seam Insertion.
21
![Page 22: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/22.jpg)
SEAM CARVINGSEAM CARVING
Seam Insertion
22
![Page 23: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/23.jpg)
SEAM CARVINGSEAM CARVING
Object Removal
User selects an object---set of pixels---to remove from a photograph.
Apply seam-carving, where each seam passes through atleast one of the pixels to be removed.
Keep applying carving till all object pixels are removed.
23
![Page 24: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/24.jpg)
SEAM CARVINGSEAM CARVING
Object Removal
24
![Page 25: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/25.jpg)
SEAM CARVINGSEAM CARVING
Object Removal
A�er object removal, you can also do seam insertion to change image back to the original size.
25
![Page 26: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/26.jpg)
SEAM CARVINGSEAM CARVING
Object Removal
Top Le�: Original. Everything else has a different shoe removed.
26
![Page 27: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/27.jpg)
SEAM CARVINGSEAM CARVING
Provides a basic framework for image re-targeting.
Simple gradient energy works surprisingly well in many cases.
There's a lot of work though to augment this with "saliency" information, to make sure you don't delete"important" parts of the image.
For example, apply a face detector, and make the energy of regions with detected faces very high.
27
![Page 28: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/28.jpg)
TEXTURE SYNTHESIS AND TRANSFERTEXTURE SYNTHESIS AND TRANSFER
Efros and Freeman, "Image Quilting for Texture Synthesis and Transfer," CGIA 2001.
Basic Task: I give you an example of texture of some size, fill in a larger region with the same texture.
28
![Page 29: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/29.jpg)
TEXTURE SYNTHESIS AND TRANSFERTEXTURE SYNTHESIS AND TRANSFER
Version 1: Divide target region into non-overlapping blocks. For each block, select a random block from the
source.
But this causes obvious boundaries between blocks.
29
![Page 30: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/30.jpg)
TEXTURE SYNTHESIS AND TRANSFERTEXTURE SYNTHESIS AND TRANSFER
Version 2: Place overlapping blocks, and when selecting a new block, make sure that it matches the existing
block in area of overlap (error is less than some threshold).
Do this by going in a raster scan order over blocks. When searching for a new block at a position, consideroverlap with block to its le� and top.
Gives better results, but still some boundary artifacts.
30
![Page 31: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/31.jpg)
TEXTURE SYNTHESIS AND TRANSFERTEXTURE SYNTHESIS AND TRANSFER
Version 3: When placing overlapping blocks, find an optimal cut: which pixels to take from existing le�/top
and which from new right/bottom block.
The cut is defined as the line which minimizes deviation between the two overlapping blocks.
31
![Page 32: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/32.jpg)
TEXTURE SYNTHESIS AND TRANSFERTEXTURE SYNTHESIS AND TRANSFER
Simple idea, but works surprisingly well in practice! Only parameter choice is block size.
32
![Page 33: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/33.jpg)
TEXTURE SYNTHESIS AND TRANSFERTEXTURE SYNTHESIS AND TRANSFER
Simple idea, but works surprisingly well in practice! Only parameter choice is block size.
33
![Page 34: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/34.jpg)
TEXTURE SYNTHESIS AND TRANSFERTEXTURE SYNTHESIS AND TRANSFER
Simple idea, but works surprisingly well in practice! Only parameter choice is block size.
34
![Page 35: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/35.jpg)
TEXTURE SYNTHESIS AND TRANSFERTEXTURE SYNTHESIS AND TRANSFER
You can apply this idea also for texture transfer.
Now given a source texture and a target image.
The idea remains the same, fill in blocks selected from the source image on the target image plane. But inthis case, you have an additional cost for selecting blocks: want to promote blocks that are 'similar' to thetarget image in the location you're filling in.
35
![Page 36: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/36.jpg)
NEURAL STYLE TRANSFERNEURAL STYLE TRANSFER
Similar idea. Make one image have the "style" or texture quality of another.
But define content and texture based on activations of a deep CNN trained for some task
Gatys et al., Image Style Transfer Using Convolutional Neural Networks, CVPR 2016:
Use a pre-trained network for classification on ImageNet.
Values of higher layers represent "content": Try to preserve them
Covariances of other layers represent style: Try to match them with other image
36
![Page 37: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/37.jpg)
NEURAL STYLE TRANSFERNEURAL STYLE TRANSFER
Set this up as an optimization problem, and minimize with SGD+Backprop from a random init.
37
![Page 38: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/38.jpg)
NEURAL STYLE TRANSFERNEURAL STYLE TRANSFER
38
![Page 39: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/39.jpg)
POISSON IMAGE EDITINGPOISSON IMAGE EDITING
Perez et al., "Poisson Image Editing," ACM ToG 2003.
Basic Idea: If I have horizontal and vertical gradients of an image at each pixel, as well as some absoluteintensities at boundaries, this is enough to reconstruct the image.
It is easier to edit, replace gradients than pixels, and reconstructing images from edited gradients looks morenatural.
Poisson Solver
Given estimates of of differences in intensities between pixels and , i.e., :
For example, can be horizontal and vertical gradients at all locations:
And values of for a small set of 'boundary' locations .
Reconstruct by minimization:
This is a simple least-squares minimization. Can solve by conjugate gradient (or Frankot-Chellappa, if youhave the same gradient at all pixels).
Remember, you did this for Normals -> Depth in CSE 559A.
{ }Vpq p q = −Vpq Ip Iq
I[x + 1, y] − I[x − 1, y], ∀(x, y)
{ }I ∗p p ∈ ∂Ω
I
I = arg min( − − , such that~ = , ∀p ∈ ∂ΩIp Iq Vpq)2 Ip I ∗p
39
![Page 40: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/40.jpg)
POISSON IMAGE EDITINGPOISSON IMAGE EDITING
Applications: Conceal objects/textures. Remove for values in regions where you don't want to retainobjects.
Vpq
40
![Page 41: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/41.jpg)
POISSON IMAGE EDITINGPOISSON IMAGE EDITING
Applications: Texture Flattening: Set all gradients below a threshold to 0.
41
![Page 42: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/42.jpg)
POISSON IMAGE EDITINGPOISSON IMAGE EDITING
Applications: Insert Objects by copying over their gradient field.
42
![Page 43: CSE 659A: Advances in Computer Visionayan/courses/cse659a/pdfs/lec14.pdf · NEURAL STYLE TRANSFER Similar idea. Make one image have the "style" or texture quality of another. But](https://reader034.vdocument.in/reader034/viewer/2022042308/5ed457c3d83ad958f471bd50/html5/thumbnails/43.jpg)
POISSON IMAGE EDITINGPOISSON IMAGE EDITING
Applications: Insert translucent objects by mixing gradient fields. (O�en, selecting the max by magnitude ofthe source and destination gradient).
43