Synthetic Gradients Tutorial
•
2 j'aime•848 vues
Synthetic Gradients were introduced in 2016 by Max Jaderberg and other researchers at DeepMind. They are designed to replace backpropagation, and they can make all sorts of deep neural networks much faster to train, and even perform better. Moreover, they can allow Recurrent Neural Networks to learn long term patterns in the data. Papers: - Decoupled Neural Interfaces using Synthetic Gradients, Max Jaderberg et al., 2016: https://arxiv.org/abs/1608.05343 - Understanding Synthetic Gradients and Decoupled Neural Interfaces, Wojciech Marian Czarnecki et al., 2017: https://arxiv.org/abs/1703.00522 Blog posts: - By Max Jaderberg, DeepMind: https://deepmind.com/blog/decoupled-neural-networks-using-synthetic-gradients/ - By iamtrask: https://iamtrask.github.io/2017/03/21/synthetic-gradients/ Implementations: - DNI-TensorFlow by Andrew Liao: https://github.com/vyraun/DNI-tensorflow - Jupyter Notebook with TensorFlow by Nitarshan Rajkumar: https://github.com/nitarshan/decoupled-neural-interfaces/ - DNI.PyTorch by Andrew Liao: https://github.com/andrewliao11/dni.pytorch/ The painting on the first slide is by Annie Clavel, a great French artist currently living in Los Angeles. Visit her website: http://www.annieclavel.com/.
Recommandé
Contenu connexe
Dernier
Dernier (20)
En vedette
En vedette (20)
Synthetic Gradients Tutorial
- 1. Synthetic Gradients Aurélien Géron December 2017 Painting by Annie Clavel, annieclavel.com
- 2. Aurélien Géron, 2017Paper by Jaderberg et al.
- 3. Aurélien Géron, 2017 X Inputs Hidden Layer Hidden Layer Backpropagation Output Layer h1 h2 h3 θ1 θ2 θ3
- 4. Aurélien Géron, 2017 X Inputs Hidden Layer Hidden Layer Backpropagation Output Layer h1 h2 h3 Forward Loss Labels θ1 θ2 θ3
- 5. Aurélien Géron, 2017 X Inputs Backpropagation h1 h2 h3 Backward Labels Hidden Layer θ1 Hidden Layer Output Layer Loss δ3 δ2 δ1 θ2 θ3
- 6. Aurélien Géron, 2017 X Inputs Backpropagation h1 h2 h3 Labels Hidden Layer θ1 Hidden Layer Output Layer Loss δ3 δ2 δ1 θ2 θ3 Gradient Descent Step
- 7. Aurélien Géron, 2017 Inputs Model Parallelism Hidden Layer Hidden Layer Output Layer Loss
- 8. Aurélien Géron, 2017 X Inputs Model Parallelism h1 h2 h3 Labels Hidden Layer θ1 Hidden Layer Output Layer Loss θ2 θ3 Forward Lock
- 9. Aurélien Géron, 2017 X Inputs Model Parallelism h1 h2 h3 Labels Hidden Layer θ1 Hidden Layer Output Layer Loss θ2 θ3 Update Lock
- 10. Aurélien Géron, 2017 X Inputs Model Parallelism h1 h2 h3 Labels Hidden Layer θ1 Hidden Layer Output Layer Loss δ3 δ2 θ2 θ3 δ1 Backward Lock
- 11. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X
- 12. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 θ1
- 13. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 θ1 M1 Synthetic Gradient Model
- 14. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 θ1 δ1 ^ M1 Synthetic Gradients Synthetic Gradient Model
- 15. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 θ1 δ1 ^ M1 Synthetic Gradients Synthetic Gradient Model
- 16. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 θ1 δ1 ^ M1 Synthetic Gradients Synthetic Gradient Model DNI
- 17. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 h2 θ2 θ1 δ1 ^ M1
- 18. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 h2 θ2 θ1 δ1 ^ M1 δ2 ^ M2
- 19. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 h2 θ2 θ1 δ1 ^ M1 δ2 ^ M2
- 20. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 h2 θ2 θ1 h3 θ3 δ1 ^ M1 δ2 ^ M2
- 21. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 h2 θ2 θ1 h3 Labels Loss θ3 δ1 ^ M1 δ2 ^ M2
- 22. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 h2 θ2 θ1 h3 δ3 θ3 δ3 Labels δ1 ^ M1 δ2 ^ M2 Loss
- 23. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 h2 θ2 θ1 h3 θ3
- 24. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer θ2 θ1 Loss θ3 Labels M1 M2
- 25. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer θ2 θ1 Loss θ3 Labels M1 M2 1 X
- 26. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 θ2 θ1 Loss θ3 Labels δ1 ^ M1 M2 2 1
- 27. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 h2 θ2 θ1 Loss θ3 Labels δ1 ^ M1 δ2 ^ M2 3 2 1
- 28. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 h2 θ2 θ1 Loss θ3 Labels δ1 ^ M1 δ2 ^ M2 4 3 2 h3 δ31
- 29. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 h2 θ2 θ1 Loss θ3 Labels δ1 ^ M1 δ2 ^ M2 5 4 3 h3 δ32
- 30. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 h2 θ2 θ1 Loss θ3 Labels δ1 ^ M1 δ2 ^ M2 6 5 4 h3 δ33
- 31. Aurélien Géron, 2017 Inputs Hidden Layer Hidden Layer Decoupled Neural Interfaces Output Layer X h1 h2 θ2 θ1 Loss θ3 Labels δ1 ^ M1 δ2 ^ M2 6 5 4 h3 δ33
- 32. Aurélien Géron, 2017 Inputs Hidden Layer Decoupled Neural Interfaces Output Layer θ20 Loss θ30 Labels δ10 ^M10 δ20 ^M20 δ30 Hidden Layer θ10
- 33. Aurélien Géron, 2017 Hidden Layer Training a Synthetic Gradient Model hi θi δi ^ Mi
- 34. Aurélien Géron, 2017 Hidden Layer Training a Synthetic Gradient Model hi θi δi ^ Mi
- 35. Aurélien Géron, 2017 Hidden Layer Training a Synthetic Gradient Model hi ǁ δi – δi ǁ² ^ θi Synthetic Gradient Model Loss Mi
- 36. Aurélien Géron, 2017 Synthetic Gradient Model LossHidden Layer Training a Synthetic Gradient Model hi θi ? Mi ǁ δi – δi ǁ² ^
- 37. Aurélien Géron, 2017 Hidden Layer Hidden Layer Training a Synthetic Gradient model hi hi+1 θi+1 θi Mi δi+1 ^Mi+ 1 ǁ δi – δi ǁ² ^ Synthetic Gradient Model Loss ?
- 38. Aurélien Géron, 2017 Hidden Layer Hidden Layer Training a Synthetic Gradient Model hi hi+1 θi+1 θi Synthetic Gradient Model Loss Mi δi+1 ^Mi+ 1 ǁ δi – δi ǁ² ^
- 39. Aurélien Géron, 2017 Hidden Layer Output Layer Training a Synthetic Gradient Model hi hi+1 θi+1 θi Loss δi+1 Synthetic Gradient Model Loss Mi ǁ δi – δi ǁ² ^
- 40. Aurélien Géron, 2017CNNs with SGs
- 41. Aurélien Géron, 2017CNNs with SGs M1 M2 M3 M4
- 42. Aurélien Géron, 2017CNNs with SGs M1 M2 M3 M4
- 43. Aurélien Géron, 2017CNNs with SGs M2 M3 M4
- 44. Aurélien Géron, 2017 Hidden Layer Providing Context hi θi Mi δi ^ DNI
- 45. Aurélien Géron, 2017 Hidden Layer Providing Context hi θi Labels Mi δi ^ cDNI
- 46. Aurélien Géron, 2017Results on MNIST Hidden Output Hidden Hidden 10 256 256 256 28x28
- 47. Aurélien Géron, 2017Results on MNIST Hidden Output Hidden Hidden 10 256 256 256 28x28 Linear + Batch Normalization + ReLU
- 48. Aurélien Géron, 2017Results on MNIST (Figure 2) Hidden Output Hidden Hidden 10 256 256 256 28x28
- 49. Aurélien Géron, 2017Results on MNIST (Figure 2) Hidden Output Hidden Hidden 10 256 256 256 28x28 3 6
- 50. Aurélien Géron, 2017Results on MNIST (Figure 2) Hidden Output Hidden Hidden 10 256 256 256 28x28 M3 M2 M1 3 6
- 51. Aurélien Géron, 2017Results on MNIST (Figure 2) Hidden Output Hidden 256 1024 1024 Mi 3 6
- 52. Aurélien Géron, 2017Results on MNIST (Figure 2) Hidden Output Hidden Hidden 10 256 256 256 28x28 M3 M2 M1 3 6
- 53. Aurélien Géron, 2017Results on MNIST (Figure 2) Hidden Output Hidden Hidden 10 256 256 256 28x28 M3 M2 M1 3 6
- 54. Aurélien Géron, 2017Results on MNIST (Figure 2) Output256 Mi
- 55. Aurélien Géron, 2017RNNs Recurrent Layer
- 56. Aurélien Géron, 2017RNNs Xt ht Recurrent Layer ht-1
- 65. Aurélien Géron, 2017RNNs Xt=0 ht=0 Xt=1 ht=1 Xt=2 ht=2 Xt=3 ht=3 0 ... ht=0 ht=1 ht=2 ht=3
- 70. Aurélien Géron, 2017RNNs with SGs Xt=0 Xt=1 Xt=2 Xt=3 0 ht=0 ht=1 ht=2 ht=0 ht=1 ht=2 ht=3 Loss
- 71. Aurélien Géron, 2017RNNs with SGs Xt=0 Xt=1 Xt=2 Xt=3 0 ht=0 ht=1 ht=2 ht=0 ht=1 ht=2 ht=3 Loss ht=3 Xt=4
- 72. Aurélien Géron, 2017RNNs with SGs Xt=0 Xt=1 Xt=2 Xt=3 0 ht=0 ht=1 ht=2 M δt=4 ^ht=0 ht=1 ht=2 ht=3 Loss ht=3 ht=4 Xt=4
- 73. Aurélien Géron, 2017RNNs with SGs Xt=0 Xt=1 Xt=2 Xt=3 0 ht=0 ht=1 ht=2 M ht=0 ht=1 ht=2 ht=3 Loss δt=4 ^ ht=3 ht=4 δt=3 Xt=4
- 74. Aurélien Géron, 2017RNNs with SGs Xt=0 Xt=1 Xt=2 Xt=3 0 ht=0 ht=1 ht=2 ht=0 ht=1 ht=2 ht=3 Loss M δt=4 ^ ht=3 ht=4 δt=3 Xt=4
- 75. Aurélien Géron, 2017RNNs with SGs Xt=0 Xt=1 Xt=2 Xt=3 0 ht=0 ht=1 ht=2 ht=0 ht=1 ht=2 ht=3 Loss M δt=4 ^ ht=3 ht=4 δt=3 Xt=4
- 76. Aurélien Géron, 2017RNNs with SGs Xt=0 Xt=1 Xt=2 Xt=3 0 ht=0 ht=1 ht=2 ht=0 ht=1 ht=2 ht=3 Loss M δt=4 ^ ht=3 ht=4 δt=3 Xt=4
- 77. Aurélien Géron, 2017RNNs with SGs Xt=0 Xt=1 Xt=2 Xt=3 ht=0 ht=1 ht=2 ht=0 ht=1 ht=2 ht=3 Loss M δt=4 ^ ht=3 ht=4 δt=3 Xt=4
- 78. Aurélien Géron, 2017RNNs with SGs Xt=0 Xt=1 Xt=2 Xt=3 ht=0 ht=1 ht=2 ht=1 ht=2 ht=3 Loss M δt=4 ^ ht=3 ht=4 δt=3 Xt=4
- 79. Aurélien Géron, 2017RNNs with SGs Xt=1 Xt=2 Xt=3 ht=1 ht=2 ht=1 ht=2 ht=3 Loss M δt=4 ^ ht=3 ht=4 δt=3 Xt=4
- 80. Aurélien Géron, 2017RNNs with SGs Xt=2 Xt=3 ht=1 ht=2 ht=2 ht=3 Loss M δt=4 ^ ht=3 ht=4 δt=3 Xt=4
- 81. Aurélien Géron, 2017RNNs with SGs t=2 Xt=3 ht=2 ht=3 M δt=4 ^ht=4 δt=3 Xt=4
- 82. Aurélien Géron, 2017RNNs with SGs M δt=4 ^ht=4 Xt=4 Xt=3
- 83. Aurélien Géron, 2017RNNs with SGs M δt=4 ^ht=4 Xt=4 Xt=3
- 84. Aurélien Géron, 2017RNNs with SGs M δt=4 ^ ht=4 Xt=4 Xt=3
- 85. Aurélien Géron, 2017RNNs with SGs M δt=4 ^ ht=4 Xt=4 Xt=3
- 86. Aurélien Géron, 2017RNNs with SGs M δt=4 ^ ht=4 Xt=4 Xt=3
- 87. Aurélien Géron, 2017RNNs with SGs M δt=4 ^ ht=4 Xt=4 ht=4 Xt=3
- 88. Aurélien Géron, 2017RNNs with SGs M δt=4 ^ ht=4 Xt=4 Xt=5 ht=4 ht=4 ht=5 Xt=3
- 89. Aurélien Géron, 2017RNNs with SGs M δt=4 ^ ht=4 Xt=4 Xt=5 Xt=6 ht=4 ht=5 ht=4 ht=5 ht=6 Xt=3
- 90. Aurélien Géron, 2017RNNs with SGs M δt=4 ^ ht=4 Xt=4 Xt=5 Xt=6 Xt=7 ht=4 ht=5 ht=6 ht=4 ht=5 ht=6 ht=7 Xt=3
- 91. Aurélien Géron, 2017RNNs with SGs M δt=4 ^ ht=4 Xt=4 Xt=5 Xt=6 Xt=7 ht=4 ht=5 ht=6 ht=4 ht=5 ht=6 ht=7 Loss Xt=3
- 92. Aurélien Géron, 2017RNNs with SGs M δt=4 ^ ht=4 Xt=4 Xt=5 Xt=6 Xt=7 ht=4 ht=5 ht=6 ht=4 ht=5 ht=6 ht=7 M δt=8 ^ ht=7 ht=8 Xt=8 Loss Xt=3
- 93. Aurélien Géron, 2017RNNs with SGs M δt=4 ^ ht=4 Xt=4 Xt=5 Xt=6 Xt=7 ht=4 ht=5 ht=6 ht=4 ht=5 ht=6 ht=7 M δt=8 ^ ht=7 ht=8 δt=7 Xt=8 Loss Xt=3
- 94. Aurélien Géron, 2017RNNs with SGs M δt=4 ^ ht=4 Xt=4 Xt=5 Xt=6 Xt=7 ht=4 ht=5 ht=6 ht=4 ht=5 ht=6 ht=7 Loss M δt=8 ^ ht=7 ht=8 δt=7 Xt=8 Xt=3
- 95. Aurélien Géron, 2017RNNs with SGs M ǁδt=4 - δt=4 ǁ ^ ht=4 Xt=4 Xt=5 Xt=6 Xt=7 ht=4 ht=5 ht=6 ht=5 ht=6 ht=7 Loss M δt=8 ^ ht=7 ht=8 δt=7 Xt=8 ht=4 Xt=3
- 96. Aurélien Géron, 2017RNNs with SGs M ǁδt=4 - δt=4 ǁ ^ ht=4 Xt=4 Xt=5 Xt=6 Xt=7 ht=4 ht=5 ht=6 ht=5 ht=6 ht=7 Loss M δt=8 ^ ht=7 ht=8 δt=7 Xt=8 ht=4 Xt=3
- 97. Aurélien Géron, 2017RNNs with SGs M ǁδt=4 - δt=4 ǁ ^ ht=4 Xt=4 Xt=5 Xt=6 Xt=7 ht=4 ht=5 ht=6 ht=5 ht=6 ht=7 Loss M δt=8 ^ ht=7 ht=8 δt=7 Xt=8 ht=4 Xt=3
- 98. Aurélien Géron, 2017Penn Treebank Results (Figure 4)
- 99. Aurélien Géron, 2017 X Inputs Synthetic Inputs h1 h2 h3 Labels Hidden Layer θ1 Hidden Layer Output Layer Loss θ2 θ3 Forward Lock
- 100. Aurélien Géron, 2017 Inputs Synthetic Inputs h3 Labels Hidden Layer θ1 Hidden Layer Output Layer Loss θ2 θ3 I3 X h2 ^ Synthetic Input model h1
- 101. Aurélien Géron, 2017 Inputs Synthetic Inputs h3 Labels Hidden Layer θ1 Output Layer Loss θ3 I3 X ǁh2 - h2 ǁ² ^ Hidden Layer θ2 Synthetic Input model h1
- 102. Aurélien Géron, 2017 Inputs Synthetic Inputs h3 Labels Hidden Layer θ1 Output Layer Loss θ3 I3 X ǁh2 - h2 ǁ² ^ Hidden Layer θ2 Synthetic Input model h1
- 103. Aurélien Géron, 2017 Inputs Synthetic Inputs h3 Labels Hidden Layer θ1 Output Layer Loss θ3 I3 X Synthetic Input model ǁh2 - h2 ǁ² ^ Hidden Layer θ2 I2 h1 ^
- 104. Aurélien Géron, 2017 Hidden Layer Fully Decoupled Neural Interface θi Mi Ii Inputs
- 105. Aurélien Géron, 2017 Hidden Layer Fully Decoupled Neural Interface θi Mi Ii Inputs hi-1 ^
- 106. Aurélien Géron, 2017 Hidden Layer Fully Decoupled Neural Interface hi θi Mi δi ^ Ii Inputs hi-1 ^
- 107. Aurélien Géron, 2017 Hidden Layer Fully Decoupled Neural Interface hi θi Mi δi ^ Ii Inputs hi-1 ^ δi-1
- 108. Aurélien Géron, 2017 Hidden Layer Fully Decoupled Neural Interface hi θi Mi δi ^ Ii Inputs hi-1 ^
- 109. Aurélien Géron, 2017 Hidden Layer Fully Decoupled Neural Interface hi θi Mi δi ^ Ii Inputs ǁhi-1 - hi-1 ǁ² ^
- 110. Aurélien Géron, 2017 Hidden Layer Fully Decoupled Neural Interface hi θi Mi δi ^ Ii Inputs ǁhi-1 - hi-1 ǁ² ^
- 111. Aurélien Géron, 2017 Hidden Layer Fully Decoupled Neural Interface hi θi Mi Ii Inputs ǁ δi – δi ǁ² ^
- 112. Aurélien Géron, 2017 Hidden Layer Fully Decoupled Neural Interface hi θi Mi Ii Inputs ǁ δi – δi ǁ² ^
- 113. Aurélien Géron, 2017 Hidden Layer Fully Decoupled Neural Interface hi θi Mi Ii Inputs