A fantastic presentation explaining how texture synthesis, neural doodles, and image style transfer work. The presentation by Dmitry Ulyanov also covers issues of efficiency of algorithms.

1. Image Style Transfer,
Neural Doodles &
Texture Synthesis
Dmitry Ulyanov
10/21/16 1
BMMO seminar
Moscow, 2016

2. Dmitry Ulyanov
• Consist of repeated
• Convolutions
• ReLU
• MaxPool
+
• FC + Softmax at the end
• Activations (feature maps)
• Tensor of size CxWxH
VGG-style networks
10/21/16 Style transfer, neural doodles & texture synthesis 2
Image credit: Xavier Giro, DeepFix slides
H
W
C

3. Dmitry Ulyanov
Image generation examples
10/21/16 Style transfer, neural doodles & texture synthesis 3
Simonyan et al. 2014Mordvintsev, 2015

4. Dmitry Ulyanov
• General overview:
1. Texture synthesis
2. Image style transfer
3. Neural doodles
• Our work “Texture networks” (ICML 2016):
• Fast texture synthesis
• Fast image style transfer
• Fast neural doodles
Presentation structure
10/21/16 Style transfer, neural doodles & texture synthesis 4

5. Dmitry Ulyanov
Examples: Texture Synthesis
10/21/16 Style transfer, neural doodles & texture synthesis 5
Source Synthesized
L. A. Gatys, A. S. Ecker, M. Bethge; “Texture Synthesis Using Convolutional Neural Networks“; NIPS 2015

6. Dmitry Ulyanov
Examples: Image Artistic Style Transfer
10/21/16 Style transfer, neural doodles & texture synthesis 6
L. A. Gatys, A. S. Ecker, M. Bethge; “Image Style Transfer Using Convolutional Neural Networks“; CVPR 2016
Content Style Result

7. Dmitry Ulyanov
Examples: Neural Doodles
10/21/16 Style transfer, neural doodles & texture synthesis 7
A. J. Champandard. “Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks”, 2016
Template Mask
Synthesized image User mask

8. Dmitry Ulyanov10/21/16 Style transfer, neural doodles & texture synthesis 8
How does it work?

9. Dmitry Ulyanov
Image generation by optimization
10/21/16 Style transfer, neural doodles & texture synthesis 9

10. Dmitry Ulyanov
Texture:
Activations at layer :
Gram matrix at layer :
Gatys et. al.: Optimization-based texture synthesis
10/21/16 Style transfer, neural doodles & texture synthesis 10

11. Dmitry Ulyanov
Image:
Gram matrix at layer :
Loss at layer :
Gatys et. al.: Optimization-based texture synthesis
10/21/16 Style transfer, neural doodles & texture synthesis 11

12. Dmitry Ulyanov
Loss:
Solve
By gradient descent
Gatys et. al.: Optimization-based texture synthesis
10/21/16 Style transfer, neural doodles & texture synthesis 12

13. Dmitry Ulyanov
Examples: Texture Synthesis
10/21/16 Style transfer, neural doodles & texture synthesis 13
Source Synthesized
L. A. Gatys, A. S. Ecker, M. Bethge; “Texture Synthesis Using Convolutional Neural Networks“; NIPS 2015

14. Dmitry Ulyanov
How to: Neural Doodles
10/21/16 Style transfer, neural doodles & texture synthesis 14
github.com/DmitryUlyanov/fast-neural-doodle
Template Mask
Synthesized image User mask

15. Dmitry Ulyanov
Total loss:
Texture loss:
Content loss:
Gatys et. al.: Content loss for style transfer
10/21/16 Style transfer, neural doodles & texture synthesis 15
Content Style Result

16. Dmitry Ulyanov
Content image:
Activations at layer :
Gatys et. al.: Content loss for style transfer
10/21/16 Style transfer, neural doodles & texture synthesis 16

17. Dmitry Ulyanov
Total loss:
Texture loss:
Content loss:
Gatys et. al.: Content loss for style transfer
10/21/16 Style transfer, neural doodles & texture synthesis 17
Content Style Result

18. Dmitry Ulyanov
The results are excellent, but…
It is slow! Several minutes on a high-end GPU.
What else?
10/21/16 Style transfer, neural doodles & texture synthesis 18

19. Texture Networks:
Feed-forward Synthesis of Textures and Stylized Images
Dmitry Ulyanov1,2, Vadim Lebedev1,2, Andrea Vedaldi3, Victor Lempitsky2
10/21/16 Style transfer, neural doodles & texture synthesis 19
ICML 2016
1 2 3

20. Dmitry Ulyanov
Instead of solving Solve
Our method: learn a neural net to generate
10/21/16 Style transfer, neural doodles & texture synthesis 20
• Now
• Generation requires a single evaluation
• But
• Need to make sure does not collapse everything into one point

21. Dmitry Ulyanov
Solve
By gradient descent
Generate :
We propose: texture network
10/21/16 Style transfer, neural doodles & texture synthesis 21

22. Dmitry Ulyanov
Solve
By gradient descent
Generate :
We propose: stylization network
10/21/16 Style transfer, neural doodles & texture synthesis 22

23. Dmitry Ulyanov
Qualitative evaluation: textures
10/21/16 Style transfer, neural doodles & texture synthesis 23
Texture Gatys et. al.
(90 sec.)
Ours
(0.06 sec.)
Almost similar but ours 500 times faster.

24. Dmitry Ulyanov
Qualitative evaluation: textures
10/21/16 Style transfer, neural doodles & texture synthesis 24
Texture Gatys et. al.
(90 sec.)
Ours
(0.06 sec.)

25. Dmitry Ulyanov
Qualitative evaluation: textures
10/21/16 Style transfer, neural doodles & texture synthesis 25
Texture Gatys et. al. Ours Texture Gatys et. al. Ours
(90 sec.) (0.06 sec.) (90 sec.) (0.06 sec.)

26. Dmitry Ulyanov
Qualitative results: stylization
10/21/16 Style transfer, neural doodles & texture synthesis 26
Content
Ours
(0.06 sec.)
Style
Gatys et. al.
(90 sec.)

27. Dmitry Ulyanov
Qualitative results: stylization
10/21/16 Style transfer, neural doodles & texture synthesis 27
Content
Ours Gatys et. al.
Style

28. Dmitry Ulyanov
Generator network
10/21/16 Style transfer, neural doodles & texture synthesis 28
• Works good with any fully convolutional architectures.
• Use Instance normalization instead of Batch Normalization.

29. Dmitry Ulyanov10/21/16 Style transfer, neural doodles & texture synthesis 29
Was the technology used somewhere?
Yes!

30. Dmitry Ulyanov
Online neural doodles: likemo.net
10/21/16 Style transfer, neural doodles & texture synthesis 30
GIF: prostheticknowledge-online-neural-doodle
Code: github.com/DmitryUlyanov/online-neural-doodle

31. Dmitry Ulyanov
• Made possible many stylization apps for mobile devices
Fast stylization
10/21/16 Style transfer, neural doodles & texture synthesis 31

32. Dmitry Ulyanov
Source code is open at
https://github.com/DmitryUlyanov/
Source code
10/21/16 Style transfer, neural doodles & texture synthesis 32

33. Dmitry Ulyanov
Instead of solving Solve
Our method: learn a neural net to generate
10/21/16 Style transfer, neural doodles & texture synthesis 33
• Now
• Generation requires a single evaluation
• But
• Need to make sure does not collapse everything into one point

34. Dmitry Ulyanov
Learning to sample
10/21/16 Style transfer, neural doodles & texture synthesis 34

35. Dmitry Ulyanov
Learning to sample
10/21/16 Style transfer, neural doodles & texture synthesis 35

36. Dmitry Ulyanov
Learning to sample
10/21/16 Style transfer, neural doodles & texture synthesis 36

37. Dmitry Ulyanov
• Process each frame independently
Video stylization
10/21/16 Style transfer, neural doodles & texture synthesis 37

38. Dmitry Ulyanov
• Use optical flow (OF)
Video stylization
10/21/16 Style transfer, neural doodles & texture synthesis 38

39. Dmitry Ulyanov
• Use optical flow (OF)
Video stylization
10/21/16 Style transfer, neural doodles & texture synthesis 39
Frame 1 Frame 2
Stylized 1
(random init)
Frame 3OF(1,2) OF(2,3)
Stylized 2
(init with OF applied to Stylized 1)
Stylized 3
init: apply OF to Stylized 2

40. Dmitry Ulyanov
• Use optical flow (OF)
Video stylization
10/21/16 Style transfer, neural doodles & texture synthesis 40

41. Dmitry Ulyanov
• Use optical flow (OF)
Video stylization
10/21/16 Style transfer, neural doodles & texture synthesis 41

42. Dmitry Ulyanov
• Ganin, Kononenko, Sungatullina, Lempitsky, ECCV 2016
DeepWarp
10/21/16 Style transfer, neural doodles & texture synthesis 42

43. Dmitry Ulyanov
• Ganin, Kononenko, Sungatullina, Lempitsky, ECCV 2016
DeepWarp
10/21/16 Style transfer, neural doodles & texture synthesis 43

44. Dmitry Ulyanov
Thank you!
The last slide
10/21/16 Style transfer, neural doodles & texture synthesis 44

45. Dmitry Ulyanov
• Generative Adversarial Networks (Goodfellow et. al., NIPS 2014): a neural
network aims to produce samples that are indistinguishable from real examples
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution, (Johnson
et. al., ECCV 2016): very similar approach fast stylization approach.
• Precomputed Real-Time Texture Synthesis with Markovian Generative
Adversarial Networks (Li & Wand, ECCV 2016): similar patch-based style transfer
acceleration approach.
Related work
10/21/16 Style transfer, neural doodles & texture synthesis 45
Similar concurrent work
Feed-forward generator