The document discusses recent work on fast neural style transfer using feed-forward texture networks. It begins with examples of texture synthesis, image style transfer, and neural doodles produced with optimization-based methods. The author then describes their work using texture networks - feed-forward generator networks trained to synthesize textures and transfer styles. Their networks can perform these tasks 500 times faster than optimization-based methods while producing comparable results. The networks have enabled real-time mobile apps for neural doodling and style transfer.

1. Image Style Transfer,
Neural Doodles &
Texture Synthesis
Dmitry Ulyanov
11/13/2016 1
MIXAR
Moscow, 2016

2. Dmitry Ulyanov
• Consist of repeated
• Convolutions
• ReLU
• MaxPool
+
• FC + Softmax at the end
• Activations (feature maps)
• Tensor of size CxWxH
VGG-style neural networks
11/13/2016 Style transfer, neural doodles & texture synthesis 2
Image credit: Xavier Giro, DeepFix slides
H
W
C

3. Dmitry Ulyanov
Image generation examples
11/13/2016 Style transfer, neural doodles & texture synthesis 3
Simonyan et al. 2014Mordvintsev, 2015

4. Dmitry Ulyanov
• General overview:
1. Texture synthesis
2. Image style transfer
3. Neural doodles
• Our work “Texture networks” (ICML 2016):
• Fast texture synthesis
• Fast image style transfer
• Fast neural doodles
Presentation structure
11/13/2016 Style transfer, neural doodles & texture synthesis 4

5. Dmitry Ulyanov
Examples: Texture Synthesis
11/13/2016 Style transfer, neural doodles & texture synthesis 5
Source Synthesized
L. A. Gatys, A. S. Ecker, M. Bethge; “Texture Synthesis Using Convolutional Neural Networks“; NIPS 2015

6. Dmitry Ulyanov
Examples: Image Artistic Style Transfer
11/13/2016 Style transfer, neural doodles & texture synthesis 6
L. A. Gatys, A. S. Ecker, M. Bethge; “Image Style Transfer Using Convolutional Neural Networks“; CVPR 2016
Content Style Result

7. Dmitry Ulyanov
Examples: Neural Doodles
11/13/2016 Style transfer, neural doodles & texture synthesis 7
A. J. Champandard. “Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks”, 2016
Template Mask
Synthesized image User mask

8. Dmitry Ulyanov11/13/2016 Style transfer, neural doodles & texture synthesis 8
How does it work?

9. Dmitry Ulyanov
Image generation by optimization
11/13/2016 Style transfer, neural doodles & texture synthesis 9

10. Dmitry Ulyanov
Texture:
Activations at layer :
Gram matrix at layer :
Gatys et. al.: Optimization-based texture synthesis
11/13/2016 Style transfer, neural doodles & texture synthesis 10

11. Dmitry Ulyanov
Image:
Gram matrix at layer :
Loss at layer :
Gatys et. al.: Optimization-based texture synthesis
11/13/2016 Style transfer, neural doodles & texture synthesis 11

12. Dmitry Ulyanov
Loss:
Solve
By gradient descent
Gatys et. al.: Optimization-based texture synthesis
11/13/2016 Style transfer, neural doodles & texture synthesis 12

13. Dmitry Ulyanov
Examples: Texture Synthesis
11/13/2016 Style transfer, neural doodles & texture synthesis 13
Source Synthesized
L. A. Gatys, A. S. Ecker, M. Bethge; “Texture Synthesis Using Convolutional Neural Networks“; NIPS 2015

14. Dmitry Ulyanov
How to: Neural Doodles
11/13/2016 Style transfer, neural doodles & texture synthesis 14
github.com/DmitryUlyanov/fast-neural-doodle
Template Mask
Synthesized image User mask

15. Dmitry Ulyanov
Total loss:
Texture loss:
Content loss:
Gatys et. al.: Content loss for style transfer
11/13/2016 Style transfer, neural doodles & texture synthesis 15
Content Style Result

16. Dmitry Ulyanov
Content image:
Activations at layer :
Gatys et. al.: Content loss for style transfer
11/13/2016 Style transfer, neural doodles & texture synthesis 16

17. Dmitry Ulyanov
Total loss:
Texture loss:
Content loss:
Gatys et. al.: Content loss for style transfer
11/13/2016 Style transfer, neural doodles & texture synthesis 17
Content Style Result

18. Dmitry Ulyanov
The results are excellent, but…
It is slow! Several minutes on a high-end GPU.
What else?
11/13/2016 Style transfer, neural doodles & texture synthesis 18

19. Texture Networks:
Feed-forward Synthesis of Textures and Stylized Images
Dmitry Ulyanov1,2, Vadim Lebedev1,2, Andrea Vedaldi3, Victor Lempitsky2
11/13/2016 Style transfer, neural doodles & texture synthesis 19
ICML 2016
1 2 3

20. Dmitry Ulyanov
Instead of solving Solve
Our method: learn a neural net to generate
11/13/2016 Style transfer, neural doodles & texture synthesis 20
• Now
• Generation requires a single evaluation
• But
• Need to make sure does not collapse everything into one point

21. Dmitry Ulyanov
Solve
By gradient descent
Generate :
We propose: texture network
11/13/2016 Style transfer, neural doodles & texture synthesis 21

22. Dmitry Ulyanov
Solve
By gradient descent
Generate :
We propose: stylization network
11/13/2016 Style transfer, neural doodles & texture synthesis 22

23. Dmitry Ulyanov
Qualitative evaluation: textures
11/13/2016 Style transfer, neural doodles & texture synthesis 23
Texture Gatys et. al.
(90 sec.)
Ours
(0.06 sec.)
Almost similar but ours 500 times faster.

24. Dmitry Ulyanov
Qualitative evaluation: textures
11/13/2016 Style transfer, neural doodles & texture synthesis 24
Texture Gatys et. al.
(90 sec.)
Ours
(0.06 sec.)

25. Dmitry Ulyanov
Qualitative evaluation: textures
11/13/2016 Style transfer, neural doodles & texture synthesis 25
Texture Gatys et. al. Ours Texture Gatys et. al. Ours
(90 sec.) (0.06 sec.) (90 sec.) (0.06 sec.)

26. Dmitry Ulyanov
Qualitative results: stylization
11/13/2016 Style transfer, neural doodles & texture synthesis 26
Content
Ours
(0.06 sec.)
Style
Gatys et. al.
(90 sec.)

27. Dmitry Ulyanov
Qualitative results: stylization
11/13/2016 Style transfer, neural doodles & texture synthesis 27
Content
Ours Gatys et. al.
Style

28. Dmitry Ulyanov
Generator network
11/13/2016 Style transfer, neural doodles & texture synthesis 28
• Works good with any fully convolutional architectures.
• Use Instance normalization instead of Batch Normalization.

29. Dmitry Ulyanov11/13/2016 Style transfer, neural doodles & texture synthesis 29
Was the technology used somewhere?
Yes!

30. Dmitry Ulyanov
Online neural doodles: likemo.net
11/13/2016 Style transfer, neural doodles & texture synthesis 30
GIF: prostheticknowledge-online-neural-doodle
Code: github.com/DmitryUlyanov/online-neural-doodle

31. Dmitry Ulyanov
• Made possible many stylization apps for mobile devices
Fast stylization
11/13/2016 Style transfer, neural doodles & texture synthesis 31

32. Dmitry Ulyanov
Source code is open at
https://github.com/DmitryUlyanov/
Source code
11/13/2016 Style transfer, neural doodles & texture synthesis 32

33. Dmitry Ulyanov
Thank you!
The last slide
11/13/2016 Style transfer, neural doodles & texture synthesis 33

34. Dmitry Ulyanov
• Generative Adversarial Networks (Goodfellow et. al., NIPS 2014): a neural
network aims to produce samples that are indistinguishable from real examples
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution, (Johnson
et. al., ECCV 2016): very similar approach fast stylization approach.
• Precomputed Real-Time Texture Synthesis with Markovian Generative
Adversarial Networks (Li & Wand, ECCV 2016): similar patch-based style transfer
acceleration approach.
Related work
11/13/2016 Style transfer, neural doodles & texture synthesis 34
Similar concurrent work
Feed-forward generator