My presentation of one of the ICLR2017 best paper by Google Brain. (arxiv.org/abs/1611.03530). I believe that generalization deserves more attention as we go deep into over-parameterization zone.

1. UNDERSTANDING
DEEP LEARNING
REQUIRES
RETHINKING
Ahmet KUZUBAŞLI
16.03.2018

2. Quick Facts
• Google Brain
• ICLR 2017 - Best Paper Award
• Love it / Hate it
• Interesting Experiments
• Questioning the Traditional Explanations
• A “This is also not useful!” paper

3. large networks
generalize well in practice?
Answers…
more interpretable networks.
principled and reliable designs.

4. neural networks
that generalize well
from those that don’t?
Answers…
more interpretable networks.
principled and reliable designs.

5. Are enough to
explain the results we are observing?
Answers…
more interpretable networks.
principled and reliable designs.

6. Conventional Wisdom
Test
Training
Error
Model Complexity
Where should we put
Deep Learning?

7. Conventional Wisdom
Deep Learning

8. Conventional Wisdom
Counting the number of parameters are NOT USEFUL!
How are we going to measure the complexity?
Deep Learning
(from poster)

9. Randomization Tests
1. Random labeling of true data
2. Partially corrupted labels
3. Shuffled pixels
4. Completely random pixels
5. Interpolations between no-noise / complete-noise pixels

10. Random Labeling of True Data
(from poster)

11. Random Labeling of True Data
(from poster)

12. Random Labeling of True Data
But it still fits perfectly!
It memorizes all random labels!
(from paper)

13. Random Labeling of True Data
(from paper)

14. Implications
Rademacher complexity and VC-dimension:
Uniform stability:
The networks fit the training set with random labels perfectly.
How sensitive the algorithm is to replacement of a single example.
(from paper)

15. What do they mean? - Implications
Rademacher complexity and VC-dimension:
Uniform stability:
The networks fit the training set with random labels perfectly.
How sensitive the algorithm is to replacement of a single example.

16. What do they mean? - Implications
Rademacher complexity and VC-dimension:
Uniform stability:
The networks fit the training set with random labels perfectly.
How sensitive the algorithm is to replacement of a single example.

17. Explicit Regularizations
• “The original hypothesis space is too large to generalize,
so confine learning to a subset of the hypothesis space
with manageable complexity.”
• Data augmentation
• Weight decay
• Dropout
Can this be the reason why the networks generalize well?

18. Explicit Regularizations
(see Table-2 at the appendix for details)

19. Explicit Regularizations
(see Table-2 at the appendix for details)

20. Implicit Regularizations
• Early Stopping (helps in ImageNet but not in CIFAR10)
• Batch-normalization: Improves 3~4%
(see Table-2 at the appendix for details)

21. Implicit Regularizations
• Early Stopping (helps in ImageNet but not in CIFAR10)
• Batch-normalization: Improves 3~4%
(see Table-2 at the appendix for details)

22. So what is the source of generalization?

23. Finite Sample Expressivity
a depth k network with O(n/k) also does the job.
(see Proof at the appendix for details)

24. An Appeal to Linear Models
• “Is there a way to determine when one global minimum
will generalize whereas another will not?”
• Common way: check the curvature of the loss at the minimum
• In linear case: Hessian is degenerate at all minima. NOT USEFUL!
• A good-old-friend: SGD!
• SGD provides “Kernel trick” as an implicit regularization.
• Often converges with minimum norm, providing guidance.
• But minimum norm is NOT -totally- predictive for generalization.

25. Conclusion
• Effective capacity of successful NNs is large enough to
shatter the training data.
“Rich enough to memorize the training data”.
• A conceptual challenge to statistical learning theory.
• Model complexity struggle to explain the generalization ability of large
ANNs.
• Optimization is easy for large neural networks.
• The source of optimization and generalization are different.
• We have yet to discover a precise formal measure
under which these enormous models are simple.

26. References
• https://arxiv.org/pdf/1611.03530.pdf (paper)
• https://danieltakeshi.github.io/2017/05/19/understanding - deep - learning - requires -
rethinking - generalization - my - thoughts - and - notes (blog)
• https://www.slideshare.net/JungHoonSeo2/understanding - deep - learning - requires -
rethinking - generalization - 2017 - 12 (slideshare)
• https://www.youtube.com/watch?v=kCj51pTQPKI (presentation, YouTube)
• http://pluskid.org/slides/ICLR2017 - Poster.pdf (poster)
• https://github.com/pluskid/fitting - random - labels (code)
• https://openreview.net/forum?id=Sy8gdB9xx (open review comments)
• Google Images

27. Feel free to ask questions…