In this presentation, I would like to review basis techniques, models, and applications in deep learning. Hope you find the slides are interesting. Further information about my research can be found at "https://sites.google.com/site/ihaiphan/." NhatHai Phan CIS Department, University of Oregon, Eugene, OR

1. Tutorial on Deep Learning and
Applications
Hai Phan
AIM Lab, University of Oregon
1Includes slide material sourced from Yoshua Bengio, Geoff Hinton, Yann LeCun, Andrew Ng, Honglak Lee, and
Marc’Aurelio Ranzato

2. Outline
• Deep learning
– Greedy level-wise training (supervised learning)
– Restricted Boltzmann machine (RBM)
– Deep belief networks
– Stacked autoassociators
– Deep Boltzmann machines
• Applications
– Human motion modeling
– Vision
– Language
2

3. Motivation: why go deep?
• Deep Architectures can be representationally
efficient
• Deep representation might allow for a
hierarchy of representations
– Comprehensibility
• Multiple levels of latent variables allow
combinatorial sharing of statistical strength
• Deep architectures work well (vision, audio,
NLP, etc.)!
3

4. Motivation
4

5. 5

6. 6

7. 7

8. 8

9. 9

10. 10

11. Outline
• Deep learning
– Greedy level-wise training (supervised learning)
– Restricted Boltzmann machine (RBM)
– Deep belief networks
– Stacked autoassociators
– Deep Boltzmann machines
• Applications
– Human motion modeling
– Vision
– Language
11

12. Restricted Boltzmann Machine (RBM)
• Binary visible units
– Pixels
• Binary hidden units
– Feature detectors
12

13. Training RBM
• Free energy
– inspired from physics
•
13

14. Different Types of Unit
• Gaussian visible units
• Gaussian visible and hidden units
• Softmax units
14

15. Outline
• Deep learning
– Greedy level-wise training (supervised learning)
– Restricted Boltzmann machine (RBM)
– Deep belief networks
– Stacked autoassociators
– Deep Boltzmann machines
• Applications
– Human motion modeling
– Vision
– Language
15

16. 16

17. 17

18. 18

19. 19

20. 20

21. Outline
• Deep learning
– Greedy level-wise training (supervised learning)
– Restricted Boltzmann machine (RBM)
– Deep belief networks
– Stacked autoassociators
– Deep Boltzmann machines
• Applications
– Human motion modeling
– Vision
– Language
21

22. Stacked Autoassociators
• Reconstruct x
22

23. Stacked Autoassociators
23

24. Benchmarks
24

25. Benchmarks
• Consistent improvement over Neural Net
25

26. Outline
• Deep learning
– Greedy level-wise training (supervised learning)
– Restricted Boltzmann machine (RBM)
– Deep belief networks
– Stacked autoassociators
– Deep Boltzmann machines
• Applications
– Human motion modeling
– Vision
– Language
26

27. 27

28. 28

29. 29

30. Greedy Layer-wise Retraining of DBM’s
30

31. Discriminative Fine-tuning of DBM’s
31

32. 32

33. 33

34. 34

35. Outline
• Deep learning
– Greedy level-wise training (supervised learning)
– Restricted Boltzmann machine (RBM)
– Deep belief networks
– Stacked autoassociators
– Deep Boltzmann machines
• Applications
– Human motion modeling
– Vision
– Language
35

36. Conditional Restricted Boltzmann
Machines (CRBM)
Auto-regression
36

37. Conditional Restricted Boltzmann
Machines (CRBM)
• http://www.uoguelph.ca/~gwtaylor/thesis/4/
37

38. Outline
• Deep learning
– Greedy level-wise training (supervised learning)
– Restricted Boltzmann machine (RBM)
– Deep belief networks
– Stacked autoassociators
– Deep Boltzmann machines
• Applications
– Human motion modeling
– Vision
– Language
38

39. 39

40. Convolutional Deep Belief Networks
40

41. 41

42. Outline
• Deep learning
– Greedy level-wise training (supervised learning)
– Restricted Boltzmann machine (RBM)
– Deep belief networks
– Stacked autoassociators
– Deep Boltzmann machines
• Applications
– Human motion modeling
– Vision
– Language
42

43. 43

44. 44

45. 45

46. 46

47. 47

48. Thank you!
48