From Research to Production with PyTorch
•
1 j'aime•688 vues
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2mBtB81. Jeff Smith covers some of the latest features from PyTorch - the TorchScript JIT compiler, distributed data parallel training, TensorBoard integration, new APIs, and more. He discusses some projects coming out of the PyTorch ecosystem like BoTorch, Ax, and PyTorch BigGraph. He digs into some of the use cases and industries where people are successfully taking PyTorch models to production. Filmed at qconnewyork.com. Jeff Smith is an engineering manager at Facebook AI where he supports the PyTorch team. He’s the author of Machine Learning Systems and Exploring Deep Learning for Language. While working at the intersection of functional programming, distributed systems, and machine learning, he coined the term reactive machine learning to describe an ideal ML architecture and associated set of techniques.
Recommandé
Recommandé
Contenu connexe
Plus de C4Media
Plus de C4Media (20)
Dernier
Dernier (20)
From Research to Production with PyTorch
- 1. FROM RESEARCH TO PRODUCTION WITH PYTORCH JEFF SMITH ENGINEERING MANAGER FACEBOOK AI
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ pytorch-torchscript-botorch/
- 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
- 4. AGENDA 01 WHAT IS PYTORCH 02 RESEARCH TO PRODUCTION 03 PRODUCTION PYTORCH 04 ECOSYSTEM 05 RESOURCES FOR DEVELOPERS
- 6. TRANSLATION SPARK AR OCULUS VR BLOOD DONATIONS
- 9. 1B+ PHONES RUNNING NEURAL NETS GLOBALLY
- 10. WHAT IS PYTORCH?
- 11. SIMPLICITY OVER COMPLEXITY HARDWARE ACCELERATED INFERENCE DISTRIBUTED TRAINING DYNAMIC NEURAL NETWORKS EAGER & GRAPH-BASED EXECUTION WHAT IS PYTORCH?
- 12. AA
- 13. AA DATASETS & DATA LOADERS TORCH.DATA MODELS, DATA TORCH.VISION AUTO DIFFERENTIATION TORCH.AUTOGRAD OPTIMIZERS TORCH.OPTIM TORCH SCRIPT DEPLOYMENT TORCH.JIT NEURAL NETWORKS TORCH.NN
- 15. TRAINING LOADING DATA, MISC THE FORWARD PASS DEFINING OUR NET import torch class Net(torch.nn.Module): def __init__(self): self.fc1 = torch.nn.Linear(8, 64) self.fc2 = torch.nn.Linear(64, 1)
- 16. TRAINING LOADING DATA, MISC THE FORWARD PASS DEFINING OUR NET import torch class Net(torch.nn.Module): def __init__(self): self.fc1 = torch.nn.Linear(8, 64) self.fc2 = torch.nn.Linear(64, 1)
- 17. TRAINING LOADING DATA, MISC THE FORWARD PASS DEFINING OUR NET def forward(self, x): x = torch.relu(self.fc1.forward(x)) x = torch.dropout(x, p=0.5) x = torch.sigmoid(self.fc2.forward(x)) return x
- 18. TRAINING LOADING DATA, MISC THE FORWARD PASS DEFINING OUR NET def forward(self, x): x = torch.relu(self.fc1.forward(x)) x = torch.dropout(x, p=0.5) x = torch.sigmoid(self.fc2.forward(x)) return x
- 19. TRAINING LOADING DATA, MISC THE FORWARD PASS DEFINING OUR NET net = Net() data_loader = torch.utils.data.DataLoader( torchvision.datasets.MNIST('./data')) optimizer = torch.optim.SGD(net.parameters())
- 20. TRAINING LOADING DATA, MISC THE FORWARD PASS DEFINING OUR NET net = Net() data_loader = torch.utils.data.DataLoader( torchvision.datasets.MNIST('./data')) optimizer = torch.optim.SGD(net.parameters())
- 21. TRAINING LOADING DATA, MISC THE FORWARD PASS DEFINING OUR NET for epoch in range(1, 11): for data, target in data_loader: optimizer.zero_grad() prediction = net.forward(data) loss = F.nll_loss(prediction, target) loss.backward() optimizer.step() if epoch % 2 == 0: torch.save(net, "net.pt")
- 24. 1 2 3 4 5 RESEARCH TO PRODUCTION
- 25. DETERMINE APPROACH 1 1 2 3 4 5 RESEARCH TO PRODUCTION
- 26. DETERMINE APPROACH 1 PREPARE DATA 2 1 2 3 4 5 RESEARCH TO PRODUCTION
- 27. DETERMINE APPROACH 1 PREPARE DATA 2 BUILD AND TRAIN MODEL 3 1 2 3 4 5 RESEARCH TO PRODUCTION
- 28. DETERMINE APPROACH 1 PREPARE DATA 2 BUILD AND TRAIN MODEL 3 TRANSFER MODEL TO PRODUCTION 4 1 2 3 4 5 RESEARCH TO PRODUCTION
- 29. DETERMINE APPROACH 1 PREPARE DATA 2 BUILD AND TRAIN MODEL 3 TRANSFER MODEL TO PRODUCTION 4 DEPLOY AND SCALE 5 1 2 3 4 5 RESEARCH TO PRODUCTION
- 30. DETERMINE APPROACH 1 PREPARE DATA 2 BUILD AND TRAIN MODEL 3 TRANSFER MODEL TO PRODUCTION 4 DEPLOY AND SCALE 5 1 2 3 4 5 RESEARCH TO PRODUCTION
- 31. DETERMINE APPROACH 1 PREPARE DATA 2 BUILD AND TRAIN MODEL 3 TRANSFER MODEL TO PRODUCTION 4 DEPLOY AND SCALE 5 1 2 3 4 5 RESEARCH TO PRODUCTION
- 32. DETERMINE APPROACH 1 PREPARE DATA 2 BUILD AND TRAIN MODEL 3 DEPLOY AND SCALE 5 RESEARCH TO PRODUCTION
- 33. DETERMINE APPROACH 1 PREPARE DATA 2 BUILD AND TRAIN MODEL 3 DEPLOY AND SCALE 4 RESEARCH TO PRODUCTION
- 37. HARDWARE EFFICIENCY OPTIMAL PERFORMANCE ON CPU/GPU/ASIC
- 39. SCALABILITY TRAINING ON HUGE DATASETS BILLIONS OF INFERENCES PER SECOND
- 41. CROSS-PLATFORM EMBEDDED OR MOBILE DEVICES CONSTRAINED SYSTEM ENVIRONMENTS
- 43. RELIABILITY TRAINING JOBS RUNNING FOR WEEKS 100S OF GPUS
- 44. TORCH.JIT TO CAPTURE AND OPTIMIZE PYTORCH CODE
- 45. EXPERIMENT EXTRACT TORCHSCRIPT OPTIMIZE AND DEPLOY TORCH.JIT TO CAPTURE AND OPTIMIZE PYTORCH CODE
- 47. EAGER MODE EXPERIMENTS TRAINING PROTOTYPING torch.jit.trace @torch.jit.script SCRIPT MODE COMPILER MOBILE SERVER LEVERAGING TORCH.JIT
- 48. DEMO
- 52. Real Time Designed to Scale Different Languages PYTEXT IN MESSENGER
- 53. PYTEXT RESEARCH TO PRODUCTION CYCLE
- 54. PYTEXT PARAMETER SWEEPING EVALUATION TRAINING MODEL AUTHORING NEW IDEA / PAPER PYTEXT RESEARCH TO PRODUCTION CYCLE
- 55. PYTEXT PARAMETER SWEEPING EVALUATION TRAINING MODEL AUTHORING NEW IDEA / PAPER PYTORCH MODEL PYTHON SERVICE SMALL-SCALE METRICS PYTEXT RESEARCH TO PRODUCTION CYCLE
- 56. PYTEXT PARAMETER SWEEPING EVALUATION TRAINING MODEL AUTHORING NEW IDEA / PAPER PYTORCH MODEL PYTHON SERVICE SMALL-SCALE METRICS PYTEXT PERFORMANCE TUNING EXPORT VALIDATION EXPORT TO TORCHSCRIPT PYTEXT RESEARCH TO PRODUCTION CYCLE
- 57. PYTEXT PARAMETER SWEEPING EVALUATION TRAINING MODEL AUTHORING NEW IDEA / PAPER PYTORCH MODEL PYTHON SERVICE SMALL-SCALE METRICS PYTEXT PERFORMANCE TUNING EXPORT VALIDATION EXPORT TO TORCHSCRIPT PYTORCH TORCHSCRIPT C++ INFERENCE SERVICE PYTEXT RESEARCH TO PRODUCTION CYCLE
- 60. LEVERAGING THE IMMUNE SYSTEM AND AI TO FIGHT CANCER
- 67. LOSS EPOCH BAYESOPT AT FACEBOOK | HYPERPARAMETER OPTIMIZATION
- 68. LOSS EPOCH Train Train Train Train BAYESOPT AT FACEBOOK | HYPERPARAMETER OPTIMIZATION
- 69. LOSS EPOCH LOW0.01 Train Train Train Train BAYESOPT AT FACEBOOK | HYPERPARAMETER OPTIMIZATION
- 70. LOSS EPOCH HIGH LOW 0.05 0.01 Train Train Train Train BAYESOPT AT FACEBOOK | HYPERPARAMETER OPTIMIZATION
- 71. LOSS EPOCH VERY HIGH HIGH LOW 0.1 0.05 0.01 Train Train Train Train BAYESOPT AT FACEBOOK | HYPERPARAMETER OPTIMIZATION
- 72. LOSS EPOCH GOOD VERY HIGH HIGH LOW 0.02 0.1 0.05 0.01 Train Train Train Train BAYESOPT AT FACEBOOK | HYPERPARAMETER OPTIMIZATION
- 74. PYTORCH UN-FRAMEWORK FOR BAYESIAN OPTIMIZATION BUILT ON PYTORCH PROBABILISTIC MODELING IN GPYTORCH BOTORCH
- 75. PYTORCH DOMAIN-AGNOSTIC ABSTRACTIONS DEPLOYMENT AUTOMATION UN-FRAMEWORK FOR BAYESIAN OPTIMIZATION BUILT ON PYTORCH PROBABILISTIC MODELING IN GPYTORCH BOTORCH AX
- 76. ADAPTIVE EXPERIMENTATION FOR NEWS FEED RANKING
- 77. EXPERIMENT CANDIDATES OFFLINE SIMULATION ONLINE A / B TEST AX / BOTORCH MULTITASK MODEL ADAPTIVE EXPERIMENTATION FOR NEWS FEED RANKING
- 80. GIVING BACK TO THE COMMUNITY
- 81. BOTORCH PYTEXT TRANSLATE HORIZON PYTEXT GIVING BACK TO THE COMMUNITY
- 84. DATA MODELS LIBRARIES FRAMEWORKS COMPILERS & OPTIMIZERS HARDWARE FULL STACK APPROACH
- 85. DATA MODELS LIBRARIES FRAMEWORKS COMPILERS & OPTIMIZERS HARDWARE HOUSE3D, CLEVER FAI3.5B, RESNEXT3D PYTEXT, HORIZON, DETECTRON, TRANSLATE PYTORCH 1.0 GLOW, TVM ZION, BIG BASIN, TIOGA PASS FULL STACK APPROACH
- 88. EASY ON-BOARDING IN THE CLOUD GOOGLE CLOUD PLATFORM AMAZON WEB SERVICES MICROSOFT AZURE
- 92. Thanks to Nick Felt (Google), William Chargin (Google), Mani Varadarajan (Google), Orion Reblitz-Richardson (FB), Tzu-Wei Huang
- 93. EDUCATION
- 94. PyTorch is now the second-fastest growing open source project on GitHub Source: venturebeat.com/2018/10/16/github-facebooks-pytorch-and-microsofts-azure-have-the-fastest-growing-open-source-projects/
- 97. PRIVACY IN AI - FREE COURSE FACEBOOK FUNDING 300 SCHOLARSHIPS TO CONTINUE EDUCATION FULL CURRICULUM OF DL COURSES TAUGHT WITH PYTORCH SOME OF THE CLASSES BASED ON PYTORCH:
- 99. OVER 1.5 MILLION MINUTES VIEWING TIME PER MONTH 100K MONTHLY VIDEO VIEWS GLOBAL COMMUNITY OF STUDENTS FROM THE US TO BENGALURU TO LAGOS 1
- 101. PYTORCH
- 102. PYTORCH EXPERIENCE INSTANTLY ON COLAB GET STARTED ON THE CLOUD INSTALL LOCALLY
- 103. JOIN THE PYTORCH DEVELOPER COMMUNITY FACEBOOK.COM/PYTORCH TWITTER.COM/PYTORCH PYTORCH.ORG
- 104. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ pytorch-torchscript-botorch/