2017-07-12 NN論文を肴に酒を飲む会 #3 @ TFUG

1. Wasserstein GAN
2017-07-12 NN #3 @ TFUG

2. Yuta Kashino ( )
BakFoo, Inc. CEO
Astro Physics /Observational Cosmology
Zope / Python
Realtime Data Platform for Enterprise / Prototyping

3. Yuta Kashino ( )
arXiv
stat.ML, stat.TH, cs.CV, cs.CL, cs.LG
math-ph, astro-ph
- PyCon2016
- PyCon2017 Edward
- 2017 8 TFUG
@yutakashino
https://www.slideshare.net/yutakashino/pyconjp2016

4. Wasserstein GAN

5. …
- WGAN: GAN
-
- DCGAN
-
-

6. Generative Adversarial
Networks

7. GAN 1
- Generative Adversarial Networks
- Ian Goodfelow
- Bengio , Theano/Pyleran2
- Google Brain
- 2016 NIPIS Tutorial
- : The GAN Zoo
https://goo.gl/uC8xn2
https://github.com/hindupuravinash/the-gan-zoo

8. GAN 2
- GAN …
- Meow Generator
- HDCGAN, WGAN, LSGAN…
https://ajolicoeur.wordpress.com/cats/
https://github.com/hindupuravinash/the-gan-zoo

9. Vanila GAN
- Generator Discriminator min/max
- G D
- MLP
https://goo.gl/vHUpqG https://goo.gl/7u4zS6

10. DCGAN
-
- CNN
- G /D
- Pool/ Full Batch Norm, Leaky ReLU
Unsupervised Representation Learning with
Deep Convolutional Generative Adversarial Networks
https://arxiv.org/abs/1511.06434
https://goo.gl/8EmZgT

11. GAN
- G/D JS
-
-
-
-
-

12. Wasserstein GAN

13. WGAN 1
- / /
- /
-
-
-
-
- (

14. WGAN 2
-
- Read-through: Wasserstein GAN
- Wasserstein GAN and the Kantorovich-Rubinstein
Duality
-
https://goo.gl/7ywVwc
https://goo.gl/40eCbR

15. WGAN
GAN Descriminator/Critic W
1. W
( 1, 2)
2. W
( 3)
3. W
4.

16. WGAN

17. 1. WGAN:

18. 4
- Total Variation(TV)
- Kullback-Leibler (KL) divergence
- Jenson-Shannon (JS) divergence
- Earth Mover (EM) / Wasserstein
(Pr, Pg) = sup
A
|Pr(A) Pg(A)|
KL(PrkPg) =
Z
x
log
✓
Pr(x)
Pg(x)
◆
Pr(x) dx
JS(Pr, Pg) =
1
2
KL(PrkPm) +
1
2
KL(PgkPm)
M = Pr/2 + Pg/2M
W(Pr, Pg) = inf
2⇧(Pr,Pg)
E(x,y)⇠
⇥
kx yk
⇤

19. 4
- W
- JS
- KL
- TV
KL(P0kP✓) = KL(P✓kP0) =
(
+1 if ✓ 6= 0 ,
0 if ✓ = 0 ,
(P0, P✓) =
(
1 if ✓ 6= 0 ,
0 if ✓ = 0 .
JS(P0, P✓) =
(
log 2 if ✓ 6= 0 ,
0 if ✓ = 0 ,
W(P0, P✓) = |✓|
U[0, 1]
https://goo.gl/40eCbR

20. 3 1
- 1:
W
- 2:
W
W
- 1, 2
W GAN Loss

21. 3 2
3: Kantorovich-Rubinstein
- W
- W
max
w2W
Ex⇠Pr
[fw(x)] Ex⇠P✓
[fw(x)]  sup
kfkLK
Ex⇠Pr
[f(x)] Ex⇠P✓
[f(x)]
= K · W(Pr, P✓)
r✓W(Pr, P✓) = r✓(Ex⇠Pr
[fw(x)] Ez⇠Z[fw(g✓(z))])
= Ez⇠Z[r✓fw(g✓(z))]

22. W/EM 1
-
-
W(Pr, Pg) = inf
2⇧(Pr,Pg)
E(x,y)⇠
⇥
kx yk
⇤
scypy.optimize.linprog
γ
https://goo.gl/7ywVwc
https://goo.gl/7ywVwc

23. W/EM 2
: Kantorovich-Rubinstein
-
W(Pr, Pg) = inf
2⇧(Pr,Pg)
E(x,y)⇠
⇥
kx yk
⇤
W(Pr, Pg) = sup
kfkL1
Ex⇠Pr
[f(x)] Ex⇠Pg
[f(x)]

24. 3 2( )
3: Kantorovich-Rubinstein
- W
- W
max
w2W
Ex⇠Pr
[fw(x)] Ex⇠P✓
[fw(x)]  sup
kfkLK
Ex⇠Pr
[f(x)] Ex⇠P✓
[f(x)]
= K · W(Pr, P✓)
r✓W(Pr, P✓) = r✓(Ex⇠Pr
[fw(x)] Ez⇠Z[fw(g✓(z))])
= Ez⇠Z[r✓fw(g✓(z))]

25. 2. WGAN:

26. r✓W(Pr, P✓) = r✓(Ex⇠Pr
[fw(x)] Ez⇠Z[fw(g✓(z))])
= Ez⇠Z[r✓fw(g✓(z))]

27. PyTorch
https://goo.gl/unktzn

28. 3. WGAN:

29. - WGAN

30. - W
DCGAN
JS
WGAN
W

31. DCGAN
WGAN
DCGAN

32. BatchNorm OK
BatchNorm
WGAN
BatchNorm
DCGAN

33. OK
MLP
WGAN
MLP
DCGAN

35. -
-
- WGAN
- GAN G D
Improved Training of Wasserstein GANs
https://arxiv.org/abs/1704.00028
Do GANs actually learn the distribution? An empirical study
https://arxiv.org/abs/1706.08224

36. WGAN
GAN Descriminator/Critic W
1. W
( 1, 2)
2. W
( 3)
3. W
4.

37. Questions
kashino@bakfoo.com
@yutakashino

38. BakFoo, Inc.
NHK NMAPS: +

39. BakFoo, Inc.
PyConJP 2015
Python

40. BakFoo, Inc.

41. BakFoo, Inc.
: SNS +