FPGA Inference - DellEMC SURFsara
•Télécharger en tant que PPTX, PDF•
1 j'aime•511 vues
FPGAs are improving diagnoses and saving lives. View the slides from Dell EMC and SURFsara's Supercomputing 2018 presentation.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à FPGA Inference - DellEMC SURFsara
Similaire à FPGA Inference - DellEMC SURFsara (20)
Plus de Intel IT Center
Plus de Intel IT Center (20)
Dernier
Dernier (20)
FPGA Inference - DellEMC SURFsara
- 1. Superpowered to Low Power Deploying supercomputer-trained deep learning models for inference on Intel® FPGAs Lucas A. Wilson, PhD HPC & AI Engineering, Dell EMC @lucasawilson
- 2. 2 Dell EMC / Intel / SURFSara Collaboration Lucas A. Wilson, Vineet Gundecha, and Alex Filby Valeriu Codreanu and Damian Podareanu Vikram Saletore and Shawn Slockers
- 3. 3 Two Phases of Machine Learning Training Learn a suitable function approximation to correctly respond to the overwhelming majority of test cases Inference / Deployment Use learned function approximation to respond to new cases
- 4. Training Deep Neural Networks to Identify Thoracic Pathologies
- 5. 5 Superpowered Neural Network Training 386845 8319 16532 1742 1362 825 675 0 50000 100000 150000 200000 250000 300000 350000 400000 450000 DenseNet121, P=1, BZ=8 DenseNet121, P=64, BZ=64, GBZ=4096 VGG16, P=128, GBZ=8192 ResNet50, P=512, GBZ=4096 Resnet50, P=512, GBZ=8192 ResNet50, P=800, GBZ=8000 ResNet50, P=1024, GBZ=8192 TimetoSolution(seconds) 4.5 DAYS to reach a solution with DenseNet121 using 2 Intel® Xeon® Scalable Gold 6148 processors 11.25 MINUTES to reach a solution with ResNet50 using 512 Intel® Xeon® Scalable Gold 6148 processors! 573x FASTER total time to solution going from 1 to 256 Dell EMC PowerEdge C6420 nodes
- 6. 6 The Price of Performance 256x Dell EMC PowerEdge C6420 Nodes ~450W* Intel® Xeon® Gold 6148+6148F *Measured median usage per node 675 sec. Runtime to Train Model 21.6 KwH Power to Train Radiologist Model
- 7. 7 21.6 KwH of Electricity is Equivalent to… Emitting 20 lbs of CO2 by burning anthracite coal https://www.eia.gov/tools/faqs/faq.php?id=73&t=11 Keeping 360 60W Light Bulbs on for 1 hour Running a Hair Dryer for 18 hours Running a Student Cluster Challenge system for 7 hrs! *At maximum allowed power
- 9. 9 Why Deploy Models Elsewhere? Remote Deployment Edge Deployment Power/Thermal Constrained Deployment
- 10. 10 Deploying the trained model Dell EMC PowerEdge C6420 Intel® Xeon® Scalable Gold 6148+6148F Dell EMC PowerEdge R740 2x Intel® Xeon® Scalable Gold 6136 (300W) 4x Intel® PACs Dell EMC PowerEdge R640 2x Intel® Xeon® Scalable Gold 5118 (210W) 2x Intel® PACs
- 11. 11 Preparing the Model for Inference* *according to the instructions Remove Horovod nodes from TensorFlow graph tensorflow/optimize_for_inference.py Apply checkpoint weights and generate binary file tensorflow/freeze_graph.py INTEL® OPENVINO™ Provide binary file to Intel® distribution of OpenVINO™ toolkit
- 12. 12 Preparing the Model for Inference* *the way that actually worked INTEL® OPENVINO™ Save Checkpoint Files While Training Load weights into equivalent Keras model and generate TF checkpoint and protobuf Make sure to set training mode to False (0) Apply checkpoint weights and generate binary file tensorflow/freeze_graph.py Provide binary file to Intel® distribution of OpenVINO™ toolkit
- 13. 13 Quantization FP16 ← FP32 FP11 ← FP32 Reduce precision to improve throughput and memory utilization
- 14. 14 Flashing the Bitstream to Intel® PAC with Arria® 10 aocl program <target> <bitstream_dir>/2-0- 1_RC_FP11_ResNet50-101.aocx 4 targets available in test system: acl[0-3] Precision Topology Target device
- 15. 15 Executing the Model on FPGA/CPU using Intel® distribution of OpenVINO™ toolkit Image Preprocessing Convolution/Activation/BatchNorm Classification 150,528B (224x224x3) Image data Pre-classification model output 114,744B (28,686 x 32b)
- 16. Performance
- 17. 17 Power Consumption – Intel® PAC with Arria® 10 20 12 11 0 10 20 30 40 50 Power Consumption (W) Unprogrammed Programmed Streaming Data
- 18. 18 183 183 183 183 183 41 43 170 411 583 87 105 420 325 745 0 100 200 300 400 500 600 700 800 900 0 100 200 300 400 500 600 700 800 Intel® PAC w/ Arria® 10, batch 1, FP11 Intel® PAC w/ Arria® 10, batch 96, FP11 4x Intel® PAC w/ Arria® 10, batch 384, FP11 Intel® Xeon® Gold 6136 (24 cores), batch 96, FP32 Intel® Xeon® Gold 6136 (23 cores) + 4x Intel® PAC w/ Arria® 10, batch 480, FP32/11 Watts Imagespersecond Idle Power Used Power Images/second Power and Performance – Dell EMC PowerEdge R740 Inference performance tests using ResNet-50 topology. FP11 precision. Image size is 224x224x3. Power measured via racadm. Intel® distribution of OpenVINO™ toolkit version R3 2.3x Throughput by using 4 Intel® FPGAs with 2 Intel® Xeon® Gold 6136
- 19. 19 0.39 0.46 0.55 0.97 1.19 0 0.2 0.4 0.6 0.8 1 1.2 1.4 Intel® PAC w/ Arria® 10, batch 1, FP11 Intel® PAC w/ Arria® 10, batch 96, FP11 Intel® Xeon® Gold 6136 (24 cores), batch 96, FP32 Intel® Xeon® Gold 6136 (23 cores) + 4x Intel® PAC w/ Arria® 10, batch 480, FP32/11 4x Intel® PAC w/ Arria® 10, batch 384, FP11 Imagespersecondperwatt Power and Performance – Dell EMC PowerEdge R740 116% Efficiency Boost using 4x Intel® FPGAs vs 2x Intel® Xeon® Gold 6136 Inference performance tests using ResNet-50 topology. FP11 precision. Image size is 224x224x3. Power measured via racadm. Intel® distribution of OpenVINO™ toolkit version R3 76% Efficiency Boost using 4x Intel® FPGAs with 2x Intel® Xeon® Gold 6136
- 20. 20 128 128 128 128 128 41 43 86 116 202 75 105 210 135 345 0 50 100 150 200 250 300 350 400 Intel® PAC w/ Arria® 10, batch 1, FP11 Intel® PAC w/ Arria® 10, batch 96, FP11 2x Intel® PAC w/ Arria® 10, batch 192, FP11 Intel® Xeon® Gold 5118 (24 cores), batch 96, FP32 Intel® Xeon ® Gold 5118 (23 cores) + 2x Intel® PAC w/ Arria® 10, batch 288, FP32/11 Idle Power Used Power Images/second Power and Performance – Dell EMC PowerEdge R640 Inference performance tests using ResNet-50 topology. FP11 precision. Image size is 224x224x3. Power measured via racadm. Intel® distribution of OpenVINO™ toolkit version R3 2.6x Throughput by using 2 Intel® FPGAs with 2 Intel® Xeon® Gold 5118
- 21. 21 0.44 0.55 0.61 0.98 1.05 0 0.2 0.4 0.6 0.8 1 1.2 Intel® PAC w/Arria® 10, batch 1, FP11 Intel® Xeon Gold 5118 (24 cores), batch 96, FP32 Intel® PAC w/ Arria® 10, batch 96, FP11 2x Intel® PAC w/ Arria® 10, batch 192, FP11 Intel® Xeon ® Gold 5118 (23 cores) + 2x Intel® PAC w/ Arria® 10, batch 288, FP32/11 Imagespersecondperwatt Power and Performance – Dell EMC PowerEdge R640 91% Efficiency Boost adding 2x Intel® FPGAs to 2x Intel® Xeon® Gold 5118 78% Efficiency Boost using 2x Intel® FPGAs vs 2x Intel® Xeon® Gold 5118 Inference performance tests using ResNet-50 topology. FP11 precision. Image size is 224x224x3. Power measured via racadm. Intel® distribution of OpenVINO™ toolkit version R3
- 22. 22
Notes de l'éditeur
- In addition to raw throughput, time-to-solution was improved dramatically. Because only the sequential DenseNet model converged to an acceptable accuracy, comparison is made to that runtime. ResNet50 runs were able to exceed generalized accuracy of the sequential DenseNet model, while being highly scalable.
- There are many reasons why an organization would want to deploy AI models at a location other than the data center where the model was trained. It may make more sense to deploy the model to a datacenter closer to where the model will be used The model may be intended for deployment
- During data streaming, PAC uses between 9-11 additional watts (41-43W total).
- 4 card projection is aggregation