Deep Learning Made Easy with Deep Features
•Télécharger en tant que PPTX, PDF•
10 j'aime•2,189 vues
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (16)
Similaire à Deep Learning Made Easy with Deep Features
Similaire à Deep Learning Made Easy with Deep Features (20)
Plus de Turi, Inc.
Plus de Turi, Inc. (20)
Dernier
Dernier (20)
Deep Learning Made Easy with Deep Features
- 1. Deep learning Made Easy with Deep Features Piotr Teterwak Dato, Machine Learning Engineer
- 2. 2 Hello, my name is… Piotr Teterwak Machine Learning Engineer, Dato
- 4. Graphlab Create: Production ML Pipeline DATA YourWebServiceor IntelligentApp ML Algorithm Data cleaning & feature eng Offline eval & Parameter search Deploy model Data engineering Data intelligence Deployment Goal: Platform to help implement, manage, optimize entire pipeline
- 6. Today’s talk
- 7. Today’s talk Features in ML Deep Neural Networks learn features Using those learned features for new tasks Productionizing
- 8. Features are key to machine learning
- 9. 9 Simple example: Spam filtering • A user opens an email… - Will she thinks its spam? • What’s the probability email is spam? Text of email User info Source info Input: x MODEL Yes! No Output: Probability of y
- 10. 10 Feature engineering: the painful black art of transforming raw inputs into useful inputs for ML algorithm • E.g., important words, complex transformation of input,… MODEL Yes! No Output: Probability of y Feature extraction Features: Φ(x) Text of email User info Source info Input: x
- 11. Deep Learning for Learning Features
- 12. 12 Linear classifiers • Most common classifier - Logistic regression - SVMs - … • Decision correspond to hyperplane: - Line in high dimensional space w0 + w1 x1 + w2 x2 > 0 w0 + w1 x1 + w2 x2 < 0
- 13. 13 What can a simple linear classifier represent? AND 0 0 1 1
- 14. 14 What can a simple linear classifier represent? OR 0 0 1 1
- 15. 15 What can’t a simple linear classifier represent? XOR 0 0 1 1 Need non-linear features
- 17. 17 Graph representation of classifier: Useful for defining neural networks x 1 x 2 x d y … 1 w2 w0 + w1 x1 + w2 x2 + … + wd xd > 0, output 1 < 0, output 0 Input Output
- 18. 18 What can a linear classifier represent? x1 OR x2 x1 AND x2 x 1 x 2 1 y -0.5 1 1 x 1 x 2 1 y -1.5 1 1
- 19. Solving the XOR problem: Adding a layer XOR = x1 AND NOT x2 OR NOT x1 AND x2 z 1 -0.5 1 -1 z1 z2 z 2 -0.5 -1 1 x 1 x 2 1 y 1 -0.5 1 1 Thresholded to 0 or 1
- 21. 21 Deep Neural Networks • Can model any function with enough hidden units. • This is tremendously powerful: given enough units, it is possible to train a neural network to solve arbitrarily difficult problems. • But also very difficult to train, too many parameters means too much memory+computation time.
- 22. 22 Neural Nets and GPU’s • Many operations in Neural Net training can happen in parallel • Reduces to matrix operations, many of which can be easily parallelized on a GPU.
- 23. 24 Convolutional Neural Nets • Strategic removal of edges Input Layer Hidden Layer
- 24. 25 Convolutional Neural Nets • Strategic removal of edges Input Layer Hidden Layer
- 25. 26 Convolutional Neural Nets • Strategic removal of edges Input Layer Hidden Layer
- 26. 27 Convolutional Neural Nets • Strategic removal of edges Input Layer Hidden Layer
- 27. 28 Convolutional Neural Nets • Strategic removal of edges Input Layer Hidden Layer
- 28. 29 Convolutional Neural Nets • Strategic removal of edges Input Layer Hidden Layer
- 29. 30 Convolutional Neural Nets http://ufldl.stanford.edu/wiki/images/6/6c/Convolution_schematic.gif
- 30. 31 Pooling layer Ranzato, LSVR tutorial @ CVPR, 2014. www.cs.toronto.edu/~ranzato
- 32. 33 Final Network Krizhevsky et al. ‘12
- 33. Applications to computer vision
- 34. 35 Image features • Features = local detectors - Combined to make prediction - (in reality, features are more low-level) Face! Eye Eye Nose Mouth
- 35. 36 Standard image classification approach Input Computer$vision$features$ SIFT$ Spin$image$ HoG$ RIFT$ Textons$ GLOH$ Slide$Credit:$Honglak$Lee$ Extract features Use simple classifier e.g., logistic regression, SVMs Face
- 36. 37 Many hand crafted features exist… Computer$vision$features$ SIFT$ Spin$image$ HoG$ RIFT$ Textons$ GLOH$ Slide$Credit:$Honglak$Lee$ … but very painful to design
- 37. 38 Change image classification approach? Input Computer$vision$features$ SIFT$ Spin$image$ HoG$ RIFT$ Textons$ GLOH$ Slide$Credit:$Honglak$Lee$ Extract features Use simple classifier e.g., logistic regression, SVMs FaceCan we learn features from data?
- 38. 39 Use neural network to learn features Input Learned hierarchy Output Lee et al. ‘Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations’ ICML 2009
- 39. Sample results • Traffic sign recognition (GTSRB) - 99.2% accuracy • House number recognition (Google) - 94.3% accuracy 40
- 40. Krizhevsky et al. ’12: 60M parameters, won 2012 ImageNet competition 41
- 41. 42 ImageNet 2012 competition: 1.2M images, 1000 categories 42
- 42. 43 Application to scene parsing ©Carlos Guestrin 2005-2014 Y LeCun MA Ranzato Semantic Labeling: Labeling every pixel with the object it belongs to [ Farabet et al. ICML 2012, PAMI 2013] Would help identify obstacles, targets, landing sites, dangerous areas Would help line up depth map with edge maps
- 43. A quick demo! 44
- 44. Challenges of deep learning
- 45. Deep learning score card Pros • Enables learning of features rather than hand tuning • Impressive performance gains on - Computer vision - Speech recognition - Some text analysis • Potential for much more impact Cons
- 46. Deep learning workflow Lots of labeled data Training set Validation set 80% 20% Learn deep neural net model Validate
- 47. Deep learning score card Pros • Enables learning of features rather than hand tuning • Impressive performance gains on - Computer vision - Speech recognition - Some text analysis • Potential for much more impact Cons • Computationally really expensive • Requires a lot of data for high accuracy • Extremely hard to tune - Choice of architecture - Parameter types - Hyperparameters - Learning algorithm - … • Computational + so many choices = incredibly hard to tune
- 48. 49 Can we do better? Input Learned hierarchy Output Lee et al. ‘Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations’ ICML 2009
- 49. Deep features: Deep learning + Transfer learning
- 50. 51 Transfer learning: Use data from one domain to help learn on another Lots of data: Learn neural net Great accuracy Some data: Neural net as feature extractor + Simple classifier Great accuracy on new problem Old idea, explored for deep learning by Donahue et al. ’14
- 51. 52 What’s learned in a neural net Neural net trained for Task 1 Very specific to Task 1More generic Can be used as feature extractor vs.
- 52. 53 Transfer learning in more detail… Neural net trained for Task 1 Very specific to Task 1More generic Can be used as feature extractor Keep weights fixed! For Task 2, learn only end part Use simple classifier e.g., logistic regression, SVMs Class?
- 53. 54 Using ImageNet-trained network as extractor for general features • Using classic AlexNet architechture pioneered by Alex Krizhevsky et. al in ImageNet Classification with Deep Convolutional Neural Networks • It turns out that a neural network trained on ~1 million images of about 1000 classes makes a surprisingly general feature extractor • First illustrated by Donahue et al in DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition 54
- 55. Transfer learning with deep features Training set Validation set 80% 20% Learn simple model Some labeled data Extract features with neural net trained on different task Validate Deploy in production
- 56. Demo 58
- 57. What else can we do with Deep Features? 59
- 59. Applications to text data
- 60. Simple text classification with bag of words aardvark 0 about 2 all 2 Africa 1 apple 0 anxious 0 ... gas 1 ... oil 1 … Zaire 0 Use simple classifier e.g., logistic regression, SVMs Class ? One “feature” per word
- 61. Word2Vec: Neural network for finding word representation Mikolov et al. ‘13 Skip-gram Model: From a word, predict nearby words in sentence dog A went for a walk Neural net Viewed as deep features
- 62. Word2Vec: Neural network for finding high dimensional representation per word Mikolov et al. ‘13 http://www.folgertkarsdorp.nl/word2vec-an-introduction/
- 63. 65 Related words placed nearby high dim space Projecting 300 dim space into 2 dim with PCA (Mikolov et al. ’13)
- 64. Blog corpus Haha Yea Hahaha Hahah Lisxc Umm Hehe laughingoutloud LOL Closest words in 300 dim Predicts gender of author with 79% accuracy
- 65. ML in production (Or how this is relevant to data scientists)
- 66. 2015: Production ML pipeline DATA YourWebServiceor IntelligentApp ML Algorithm Data cleaning & feature eng Offline eval & Parameter search Deploy model Data engineering Data intelligence Deployment Using deep learning Goal: Platform to help implement, manage, optimize entire pipeline
- 67. In real life
- 69. 71 Take Home Message Use simple classifier e.g., logistic regression, SVMs Class? Deep Features are remarkable!
- 70. 72 CONF.DATO.COM
- 71. 73 Dato Office Hours @ Galvanize SF • Bring your laptop & some data & we’ll help you get started • When: Thurs (tomorrow) 2:30p-5p followed by beers • Where: Galvanize – 44 Tehama St. (SOMA) in SF • Talk to me/email me: piotr@dato.com +
- 72. Get the software: dato.com/download Learn: dato.com/learn Learn more: blog.dato.com Join us: we’re hiring lots! Contact me: piotr@dato.com
- 73. 75 Go create something! [with Dato] Data Engineering Data Intelligence Deployment • Fast & scalable • Rich data type support • Visualization • App-oriented ML • Supporting utils • Extensibility • Batch & always-on • RESTful interface • Elastic & robust