Workshops

1. 1
Introduction to Generative
Adversarial Networks
Vladyslav Kolbasin
Lead Software Developer, Consultant
14.03.2018

2. 2
Agenda
1. Introduction to generative adversarial network
2. What can be done with GANs?
3. Pros and cons
4. How to train it? How to construct your own GAN
5. Application examples
1. Faces in-painting
2. Lesion segmentation
6. Cool results
7. Conclusion

3. 3
Generative vs discriminative models
• Pros:
• Know about
data distribution
• Cons:
• Very expensive
to get
• Need lots of
data
• Pros:
• Easy to model
• Cons:
• Can classify,
but not
generate data
Generative Discriminative
P(c|x)

4. 4
Generative adversarial network. Architecture
• GAN – two neural networks competing against
each other in a zero-sum game framework.
(Ian Goodfellow et al. in 2014)
• G tries to “trick” D by generating samples that
are hard for D to distinguish from data
• Some kind of unsupervised learning
• Networks try to:
• D(G(z)) => max
• D(x)(1 — D(G(z))) => max
=> Nash equilibrium:
Credit:
https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generati
ve-models-and-adversarial-training-upc-2016

5. 5
What can be done with GANs?
• Simulated environments and training data
• Missing data
• Multiple correct answers
• Realistic generation tasks
• Super-resolution
• Learn useful embeddings

6. 6
Simulated environments and training data
• Good generator, but too few real data
• Can’t use real data
• Medical data (EHR, EMR, ...)
• Drug discovery
• Autonomous driving data
• Many others..

7. 7
Missing data. Image inpainting

8. 8
Multiple correct answers

9. 9
Multiple correct answers
(Lotter et al 2016)

10. 10
Realistic generation tasks

11. 11
Realistic generation tasks
https://github.com/tkarras/progressive_growing_of_gans
https://arxiv.org/abs/1710.10196

12. 12
Linear Latent Space
G(z0
) Linear Interpolation in z space: G(z0
+t*(z1
-z0
)) G(z1
)

13. 13
• “Unsupervised Representation Learning with Deep Convolutional
Generative Adversarial Models”. https://arxiv.org/pdf/1511.06434.pdf
Vector Space Arithmetics

14. 14
GAN Pros and Cons
• Pros
• Provide great results
• Vector Space Arithmetics
• Designing loss function
• Cons
• Generality penalty:
for any given problem, application-tailored solutions might work better
• It is very hard to train it (properly)
■ Sometimes training longer makes it worse
■ Discriminator almost always wins
■ Mode-collapse
• Hard to evaluate

15. 15
Design loss function
• No need in hand-engineering our loss functions for Img2Img task

16. 16
Design loss function
• No need in hand-engineering our loss functions for Img2Img task

17. 17
Design loss function
• No need in hand-engineering our loss functions for Img2Img task

18. 1818
Demo. Mode-collapse.
Oscillations in GAN

19. 19
• Optimization criteria:
• We want to make distributions equal
• It is equivalent to maximizing log-likelihood or
• KL-divergence:
• We can use other divergence too
• Jensen-Shannon divergence
GAN. How to train?
True data Noise for generating data

20. 20
KL-divergence

21. 21
KL-divergence

22. 22
• Introduced other metric: Wasserstein metric (earth mover's distance)
• Summary about wGAN:
• For mathematicians: it uses Wasserstein distance instead of
Jensen-Shannon divergence to compare distributions
• For engineers: it gets rid of a few unnecessary logarithms, and clips
weights
• For others: it employs an art critic instead of forgery expert
• More math details: https://www.cph-ai-lab.com/wasserstein-gan-wgan
Fixing mode collapse. wGAN(s)
Wasserstein GAN. https://arxiv.org/pdf/1701.07875.pdf

23. 23
Fixing mode collapse. Improved wGAN.
• Penalize the norm of the gradient
of the critic with respect to its
input instead of clipping weights.
• This ‘gradient penalty’ is simply
added to the Wasserstein
distance for the total loss.
Improved training of Wasserstein GAN. https://arxiv.org/pdf/1704.00028.pdf

24. 24
Improved WGAN: +gradient penalty

25. 2525
Demo. Mode-collapse.
Fixing oscillations in GANs

26. 2626
How to train it?

27. 27
GAN. Training Discriminator
Credit: https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016

28. 28
GAN. Training Generator
Credit: https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016

29. 29
GAN. Training algorithm

30. 3030
Let’s construct GAN

31. 31
Generator

32. 32
Discriminator

33. 33
Init GAN

34. 34
Init GAN

35. 35
Train

36. 36
Train

37. 37
Train

38. 38
GAN hacks
• https://github.com/soumith/ganhacks
• About 17 hacks:
• Normalize the inputs
• Use Soft and Noisy Labels
• Avoid Sparse Gradients: ReLU,
MaxPool. Use LeakyReLU
• Use SGD for discriminator and
ADAM for generator
• …
• Sample from a gaussian
distribution
• Batch Normalization

39. 39
GANs Taxonomy

40. 4040
Application Examples

41. 41
1. Faces in-painting
• Can we restore image with
distortion?
• Faces dataset: CelebA
• Solution: Conditional GAN

42. 42
1. Faces mode-collapse
3800 4250 11000
12000 12300 14300

43. 43
• Competition Skin Lesion Analysis Towards Melanoma Detection
• Segment image - find lesion in the picture
• https://challenge.kitware.com/#challenge/560d7856cad3a57cfde481ba
• Solution: Analogue of pix2pix architecture
2. Lesion segmentation

44. 44

45. 45

46. 4646
Cool Results

47. 47
Updates at the GAN Zoo
https://github.com/hindupuravinash/the-gan-zoo

48. 48
StackGAN
• Text-to-Image Synthesis https://github.com/hanzhanggit/StackGAN

49. 49

50. 50

51. 51
medGAN
• “Generating Multi-label Discrete Patient Records using Generative
Adversarial Network” https://arxiv.org/pdf/1703.06490.pdf

52. 52
Pix2Pix
• Image-to-image translation with conditional adversarial nets
• https://github.com/phillipi/pix2pix

53. 53
CycleGAN
• Does image-to-image translation
without input-output pairs
• Unsupervised learning
• https://github.com/junyanz/CycleGAN

54. 54
CycleGAN

55. 55
UNIT
• Unsupervised Image-to-Image Translation Networks
• https://arxiv.org/pdf/1703.00848.pdf
• https://github.com/mingyuliutw/unit

56. 56
UNIT

57. 57
Conclusions
• GANs can do very cool things
• GANs training is hard and slow
• GANs is an architecture that is rapidly developing
• Three view of GANs:
• Generative models: G(z) - unsupervised learning
• Trainable regression loss .
Learn L - what should I do.
• Domain matching loss
■ Domain adaptation - train on source, adapt to target
■ Adversarial Discriminative Domain Adaptation
https://arxiv.org/pdf/1702.05464.pdf

58. 58
References
• Good tutorials from Ian Goodfellow:
• https://arxiv.org/pdf/1701.00160.pdf
• https://arxiv.org/pdf/1406.2661.pdf
• https://youtu.be/RvgYvHyT15E
• Training hacks:
• https://github.com/soumith/ganhacks
• Simplest GAN:
• https://deeplearning4j.org/generative-adversarial-network
• GAN collections:
• https://github.com/nightrome/really-awesome-gan
• https://github.com/wiseodd/generative-models
• https://github.com/hindupuravinash/the-gan-zoo

59. 59
Vladyslav Kolbasin
Lead Software Developer, Consultant
vladyslav.kolbasin@globallogic.com
ва
Thank you!