a neural algorithm of artistic styleheld/teaching/wiss_arbeiten/... · problem statement the...
TRANSCRIPT
A Neural Algorithm of Artistic Style
Liyuan Su Elaheh YousefiAmiri
February 1, 2019
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 1 / 16
Overview
1 Problem Statement
2 Methods of Artistic Styles
3 Deep image representations
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 2 / 16
Problem Statement
The generated image B combines the ”content” of the image A with the”style” of image S.
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 3 / 16
Methods of Artistic StylesImage Style Transfer
Traditional Methods: Non-parametric
Deep Learning based Methods
� Optimization� Convolutional Neural Networks(Gatys et al.)� Feed-forward(Johnson et al.)
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 4 / 16
Methods of Artistic StylesOptimization Method
Optimization
”Neural style transfer used an optimization technique that is, starting offwith a random noise image and making it more and more desirable withevery training iteration of the neural network.”
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 5 / 16
Methods of Artistic StylesNeural Style Transfer Algorithm-CNN
Figure: Style and content representations taken at each layer of a NeuralNetwork.Image from(Gatys et)
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 6 / 16
Methods of Artistic StylesFeed-Forward
Feed-Forward
”By pre-training a feed-forward network rather than directly optimizing theloss functions.”
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 7 / 16
Methods of Artistic StylesFeed-Forward
Figure: ”System overview. We train an image transformation network totransform input images into output images. We use a loss network pretrained forimage classification to define perceptual loss functions that measure perceptualdifferences in content and style between images. The loss network remains fixedduring the training process.” Figure from Johnson et al
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 8 / 16
Deep image representations
Content representation
Style representation
Style transfer
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 9 / 16
Deep image representationsContent representations
Define the squared-error loss between the two feature representations
Lcontent(~p, ~x , l) =1
2
∑i ,j
(F lij − P l
ij)2
The derivative of this loss with respect to the activations in layer l equal
∂Lcontent
∂F lij
=
{(F l − P l)ij F l
ij > 0,
0 otherwise
from which the gradient with respect to the image ~x can be computedusing standard error back-propagation.
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 10 / 16
Deep image representationsstyle representation
Feature correlations are given by the Gram matrix G l ∈ RNl×Nl ,where G lij
is the inner product between the vectorised feature maps i and j in layer l:
G lij =
∑k
F likF
ljk .
The contribution of layer l to the total loss is then:
El =1
4N2l M
2l
∑i ,j
(G lij − Al
ij)2
and the total style loss is
Lstyle(~a, ~x) =L∑
l=0
wlEl
where wl are weighting factors of the contribution of each layer to thetotal loss
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 11 / 16
Deep image representationsStyle representation
The derivative of El with respect to the activations in layer l can becomputed analytically:
∂El
∂F lij
=
{1
N2l M
2l
((F l)T (G l − Al))ji if F lij > 0,
0 otherwise
The gradients of El with respect to the pixel values ~x can be readilycomputed using standard error back-propagation.
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 12 / 16
Deep image representationsStyle transfer
The loss function we minimise is
Ltotal(~p, ~a, ~x) = αLcontent(~p, ~x) + βLstyle(~a, ~x)
Figure: Style transfer algorithm
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 13 / 16
online Aplications
Style transfer Apps:http://deepart.iohttp://www.pikazoapp.comhttps://artisto.my.com (Video and Photo Editor)
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 14 / 16
References
Leon A. Gatys, Alexander S. Ecker, Matthias Bethge(2015)
A Neural Algorithm of Artistic Style
https://arxiv.org/abs/1508.06576 .
Justin Johnson, Alexandre Alahi, Li Fei-Fei(2016)
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
https://arxiv.org/pdf/1603.08155.pdf
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 15 / 16
Thank YouThe End
Liyuan Su, Elaheh YousefiAmiri A Neural Algorithm of Artistic Style February 1, 2019 16 / 16