Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Pycon2017
Similaire à Pycon2017 (20)
Plus de Yuta Kashino
Plus de Yuta Kashino (20)
Dernier
Dernier (20)
Pycon2017
- 2. Yuta Kashino ( ) BakFoo, Inc. CEO Astro Physics /Observational Cosmology Zope / Python Realtime Data Platform for Enterprise / Prototyping
- 3. Yuta Kashino ( ) arXiv PyCon2015 Python PyCon2016 PyCon2017 DNN PPL Edward @yutakashino
- 8. - Denker, Schwartz, Wittner, Solla, Howard, Jackel, Hopfield (1987) Denker and LeCun (1991) MacKay (1992) Hinton and van Camp (1993) Neal (1995) Barber and Bishop (1998) Graves (2011) Blundell, Cornebise, Kavukcuoglu, and Wierstra (2015) Hernandez-Lobato and Adam (2015)
- 9. - Yarin Gal Zoubin Ghahramani Shakir Mohamed Dastin Tran Rajesh Ranganath David Blei Ian Goodfellow Columbia U U of Cambridge
- 10. - - : - : - : - : - : SGD + BackProp … …x1 x2 xd ✓(2) ✓(1) x y y(n) = X j ✓ (2) j ( X i ✓ (1) ji x (n) i ) + ✏(n) p(y(n) | x(n) , ✓) = ( X i ✓ (n) i x (n) i ) ✓ D = {x(n) , y(n) }N n=1 = (X, y)
- 11. : - + - 2012 ILSVRC → 2015 - - - -
- 12. : - - ReLU, DropOut, Mini Batch, SGD(Adam), LSTM… - - ImageNet, MSCoCo… - : GPU, - : - Theano, Torch, Caffe, TensorFlow, Chainer, MxNet, PyTorch…
- 15. - = = -
- 16. - - : - : - : - : - : SGD + BackProp … …x1 x2 xd ✓(2) ✓(1) x y y(n) = X j ✓ (2) j ( X i ✓ (1) ji x (n) i ) + ✏(n) p(y(n) | x(n) , ✓) = ( X i ✓ (n) i x (n) i ) ✓ D = {x(n) , y(n) }N n=1 = (X, y)
- 17. 1. → 2. → DropOut ✓
- 18. 1. - data hypothesis( ) - : - - P(H | D) = P(H)P(D | H) P H P(H)P(D|H) P(x) = X y P(x, y) P(x, y) = P(x)P(y | x) posterior likelihoodprior evidence
- 19. 1. - : - : - - - : P(H | D) = P(H)P(D | H) P H P(H)P(D|H) likelihood priorposterior P(✓ | D, m) = P(D | ✓, m)P(✓ | m) P(D | m) m: P(x | D, m) = Z P(x | ✓, D, m)P(✓ | D, m)d✓ P(m | D) = P(D | m)P(m) P(D) evidence ✓ ⇠ Beta(✓ | 2, 2)
- 20. 1. - - : - : - : … …x1 x2 xd ✓(2) ✓(1) x y ✓ D = {x(n) , y(n) }N n=1 = (X, y) P(✓ | D, m) = P(D | ✓, m)P(✓ | m) P(D | m) m:
- 21. 1. - (MCMC) - (Variational Inference) P(✓ | D, m) = P(D | ✓, m)P(✓ | m) P(D | m) Z P(D | ✓, m)P(✓)d✓ evidence
- 22. 1. - - P(✓ | D, m) = P(D | ✓, m)P(✓ | m) liklihood priorposterior ✓ https://github.com/dfm/corner.py
- 24. 1.
- 25. P(θ|D,m) KL q(θ) ELBO 1. ⇤ = argmin KL(q(✓; ) || p(✓ | D)) = argmin Eq(✓; )[logq(✓; ) p(✓ | D)] ELBO( ) = Eq(✓; )[p(✓, D) logq(✓; )] ⇤ = argmax ELBO( ) P(✓ | D, m) = P(D | ✓, m)P(✓ | m) P(D | m)
- 26. 1. - KL =ELBO - P q q(✓; 1) q(✓; 5) p(✓, D) p(✓, D) ✓✓ ⇤ = argmax ELBO( ) ELBO( ) = Eq(✓; )[p(✓, D) logq(✓; )]
- 27. 1. - P q - : - ADVI: Automatic Differentiation Variational Inference - BBVI: Blackbox Variational Inference arxiv:1603.00788 arxiv:1401.0118 https://github.com/HIPS/autograd/blob/master/examples/bayesian_neural_net.py
- 28. 1. - VI - - David MacKay “Lecture 14 of the Cambridge Course” - PRML 10 http://www.inference.org.uk/itprnn_lectures/
- 29. 1. Reference - Zoubin Ghahramani “History of Bayesian neural networks” NIPS 2016 Workshop Bayesian Deep Learning - Yarin Gal “Bayesian Deep Learning"O'Reilly Artificial Intelligence in New York, 2017
- 30. 2. - - : - : - : - : - : SGD + BackProp … …x1 x2 xd ✓(2) ✓(1) x y y(n) = X j ✓ (2) j ( X i ✓ (1) ji x (n) i ) + ✏(n) p(y(n) | x(n) , ✓) = ( X i ✓ (n) i x (n) i ) ✓ D = {x(n) , y(n) }N n=1 = (X, y) Dropout
- 31. 2.Dropout - Yarin Gal ”Uncertainty in Deep Learning” - Dropout - Dropout : conv - LeNet with Dropout http://mlg.eng.cam.ac.uk/yarin/blog_2248.html
- 32. 2.Dropout - LeNet DNN - conv Dropout MNIST
- 33. 2.Dropout - CO2
- 34. - : - : - : - : - (MCMC) - (Variational Inference) … …x1 x2 xd ✓(2) ✓(1) x y ✓ D = {x(n) , y(n) }N n=1 = (X, y) P(✓ | D, m) = P(D | ✓, m)P(✓ | m) P(D | m)
- 35. Edward
- 36. Edward - Dustin Tran (Open AI) - Blei Lab - (PPL) - 2016 2 PPL - / TensorFlow - George Edward Pelham Box Box-Cox Trans., Box-Jenkins, Ljung-Box test box plot Tukey, 3 2 RA Fisher
- 37. - Probabilistic Programing Library/Langage - Stan, PyMC3, Anglican, Church, Venture,Figaro, WebPPL, Edward - : Edward / PyMC3 - (VI) Metropolis Hastings Hamilton Monte Carlo Stochastic Gradient Langevin Dynamics No-U-Turn Sampler Blackbox Variational Inference Automatic Differentiation Variational Inference
- 40. 1. TF: - - :
- 41. 1. TF:
- 42. 1. TF: - - - GPU / TPU Inception v3 Inception v4 # of parameters: 42,679,816 # of layers: 48
- 43. 1. TF: - Keras, Slim - TensorBoard
- 45. 2. x: edward x⇤ s P(x | ↵) ✓⇤ ⇠ Beta(✓ | 1, 1)
- 46. 2. - ( ) Edward p(x, ✓) = Beta(✓ | 1, 1) 50Y n=1 Bernoulli(xn | ✓),
- 47. 2. - log_prob() - mean() - sample() - tf.contrib.distributions 51 : https://www.tensorflow.org/api_docs/python/tf/contrib/distributions
- 48. 3. Edward TF
- 49. 3. Wh Wh Wx Wx bh bh xtxt 1 ht 1 ht Wy Wy by by yt 1 yt ht = tanh(Whht 1 + Wxxt + bh) yt ⇠ Normal(Wyht + by, 1).
- 51. 4. - : - : - : - : - (MCMC) - (Variational Inference) … …x1 x2 xd ✓(2) ✓(1) x y ✓ D = {x(n) , y(n) }N n=1 = (X, y) P(✓ | D, m) = P(D | ✓, m)P(✓ | m) P(D | m)
- 52. 4.
- 53. Edward MCMC 4. : MCMC
- 54. Edward : KLqp 4. :
- 55. 5. Box’s loop George Edward Pelham Box Blei 2014
- 56. 5. Box’s loop
- 57. Edward - Edward = TensorFlow + + + - TensorFlow - - TF GPU, TPU, TensorBoard, Keras - - TensorFlow
- 59. Refrence •D. Tran, A. Kucukelbir, A. Dieng, M. Rudolph, D. Liang, and D.M. Blei. Edward: A library for probabilistic modeling, inference, and criticism.(arXiv preprint arXiv:1610.09787) •D. Tran, M.D. Hoffman, R.A. Saurous, E. Brevdo, K. Murphy, and D.M. Blei. Deep probabilistic programming.(arXiv preprint arXiv:1701.03757) •Box, G. E. (1976). Science and statistics. (Journal of the American Statistical Association, 71(356), 791–799.) •D.M. Blei. Build, Compute, Critique, Repeat: Data Analysis with Latent Variable Models. (Annual Review of Statistics and Its Application Volume 1, 2014)
- 62. BakFoo, Inc. NHK NMAPS: +
- 64. BakFoo, Inc.
- 65. BakFoo, Inc. : SNS +
- 66. 3. 256 28*28