CUDA-Python and RAPIDS for blazing fast scientific computing

1. Abraham Stern, Ph.D., January 21, 2020 RAPIDS & CUDA PYTHON FOR BLAZING FAST SCIENTIFIC COMPUTING

3. NVIDIA ACCELERATED COMPUTING IS ACCELERATING Piz Daint Europe’s Fastest 5,704 GPUs| 21 PF ORNL Summit World’s Fastest 27,648 GPUs| 149 PF Total Pangea 3 Fastest Industrial 3,348 GPUs| 18 PF ABCI Japan’s Fastest 4,352 GPUs| 20 PF LLNL Sierra World’s 2nd Fastest 17,280 GPUs| 95 PF

4. 4 INTERSECTION OF HPC & AI TRANSFORMING SCIENCE HPC > Algorithms based on first principles theory > Proven models for accurate results AI > Neural networks that learn patterns from large data sets > Improve predictive accuracy and faster response time IDENTIFYING CHEMICAL COMPOUNDSEXASCALE WEATHER MODELING O&G FAULT INTERPRETATIONSPEEDING PATH TO FUSION ENERGY 90% Prediction Accuracy Publish in Nature April 2019 Tensor Cores Achieved 1.13 EF 2018 Gordon Bell Winner Orders Of Magnitude Speedup 3M New Compounds In 1 Day Time-to-solution Reduced From Weeks To 2 Hours

5. 5 AI FOR SCIENCE BRINGING HPC AND AI TOGETHER Combining simulation with the power of AI to Dramatically accelerate time to insight Design Space Exploration ICF + MERLIN — Fusion Inverse Problems LIGO — Gravitational Waves Faster Prediction ANI + MD – Chemistry Real-time Steering ITER – Fusion Energy

6. 6 2013 BEYOND MOORE’S LAW Progress Of Stack In 6 Years Base OS: CentOS 6.2 Resource Mgr: r304 CUDA: 5.0 Thrust: 1.5.3 2019 Accelerated Server With Fermi Accelerated Server with Volta NPP: 5.0 cuSPARSE: 5.0 cuRAND: 5.0 cuFFT: 5.0 cuBLAS: 5.0 Base OS: Ubuntu 16.04 Resource Mgr: r384 CUDA: 10.0 NPP: 10.0 cuSPARSE: 10.0 cuSOLVER: 10.0 cuRAND: 10.0 cuFFT: 10.0 cuBLAS: 10.0 Thrust: 1.9.0 GPU-Accelerated Computing CPU Moore’s Law 2013 2014 2015 2016 2017 2018 2019March RelativePerformance

7. 7 APPS & FRAMEWORKS CUDA-X & NVIDIA SDKs NVIDIA DATA CENTER PLATFORM Single Platform Drives Utilization and Productivity CUDA & CORE LIBRARIES - cuBLAS | NCCL DEEP LEARNING cuDNN HPC cuFFTOpenACC +600 Applications Amber NAMD CUSTOMER USE CASES Speech Translate Recommender SCIENTIFIC APPLICATIONS Molecular Simulations Weather Forecasting Seismic Mapping CONSUMER INTERNET & INDUSTRY APPLICATIONS ManufacturingHealthcare Finance MACHINE LEARNING cuMLcuDF cuGRAPH cuDNN CUTLASS TensorRT VIRTUAL GPU VIRTUAL GRAPHICS vDWS vPC Creative & Technical Knowledge Workers vAPPS TESLA GPUs & SYSTEMS EVERY MAJOR CLOUDTESLA GPU NVIDIA HGXNVIDIA DGX FAMILY EVERY OEM

8. 8 DEEP LEARNING GPU ACCELERATED LIBRARIES “Drop-in” Acceleration for Your Applications LINEAR ALGEBRA PARALLEL ALGORITHMS SIGNAL, IMAGE & VIDEO TensorR T nvGRAPH NCCL cuBLAS cuSPARSE cuRAND DeepStream SDK NVIDIA NPPcuFFT CUDA Math library cuSOLVER CODEC SDKcuDNN

9. 9 NVIDIA CUDA-X UPDATES Software To Deliver Acceleration For HPC & AI Apps; 500+ New Updates CUDA CUDA-X HPC & AI 40+ GPU Acceleration Libraries Linear Algebra Machine Learning & Deep Learning Computational Physics & Chemistry Computational Fluid Dynamics Life Sciences & Bioinformatics Structural Mechanics Weather & Climate Geoscience, Seismology & Imaging Numerical Analytics Electronic Design Automation Desktop Development Data Center Supercomputers GPU-Accelerated Cloud 600+ Apps Parallel Algorithms Signal Processing Deep Learning Machine Learning Visualization

10. 10 WHAT IS OPENACC main() { <serial code> #pragma acc kernels { <parallel code> } } Add Simple Compiler Directive Read more at www.openacc.org/about POWERFUL & PORTABLE Directives-based programming model for parallel computing Designed for performance and portability on CPUs and GPUs SIMPLE Open Specification Developed by OpenACC.org Consortium

11. 11 GAUSSIAN 16 Using OpenACC allowed us to continue development of our fundamental algorithms and software capabilities simultaneously with the GPU-related work. In the end, we could use the same code base for SMP, cluster/ network and GPU parallelism. PGI's compilers were essential to the success of our efforts. “ “ Mike Frisch, Ph.D. President and CEO Gaussian, Inc.

12. 12 VASP For VASP, OpenACC is the way forward for GPU acceleration. Performance is similar and in some cases better than CUDA C, and OpenACC dramatically decreases GPU development and maintenance efforts. We’re excited to collaborate with NVIDIA and PGI as an early adopter of CUDA Unified Memory. “ “ Prof. Georg Kresse Computational Materials Physics University of Vienna

13. 13 SCIENTIFIC COMPUTING WORKFLOW Running on the big machine is only one part of the workflow ● Data preprocessing ● Algorithm development or prototyping ● Test runs or small trial runs ● Post-run analysis

14. 14 REPERCUSSIONS OF WAITING ● Disrupts train of thought ● Slows the innovation/discovery cycle ● Kills momentum ● Hurts creativity ● Ultimately, hurts the science Running on the big machine is only one part of the workflow

15. 15 GPU ACCELERATED COMPUTING IN PYTHON ● Write GPU-accelerated code in Python ● High-level API with Numpy syntax ● Low-level API with CUDA syntax ● Automatic data movement ● Or manual data movement

16. 16 GPU ACCELERATED COMPUTING IN PYTHON WITH NUMBA 16 CUDA-provided variables for thread indices Map our thread indices to some work Decorator to just-in-time compile CUDA

17. 17 INTRODUCTION TO CUDA 17 GPUs do work in parallel GPU performWork<<<2, 4>>>()

18. 18 INTRODUCTION TO CUDA 18 GPU work is done in a thread GPU performWork<<<2, 4>>>()

19. 19 INTRODUCTION TO CUDA 19 Many threads run in parallel GPU performWork<<<2, 4>>>()

20. 20 INTRODUCTION TO CUDA 20 A collection of threads is a block GPU performWork<<<2, 4>>>()

21. 21 INTRODUCTION TO CUDA 21 There are many blocks GPU performWork<<<2, 4>>>()

22. 22 INTRODUCTION TO CUDA 22 A collection of blocks associated with a given kernel launch is a grid GPU performWork<<<2, 4>>>()

23. 23 INTRODUCTION TO CUDA 23 GPU functions are called kernels GPU performWork<<<2, 4>>>()

24. 24 INTRODUCTION TO CUDA 24 Kernels are launched with an execution configuration GPU performWork<<<2, 4>>>()

25. 25 INTRODUCTION TO CUDA 25 The execution configuration defines the number of blocks in the grid GPU performWork<<<2, 4>>>()

26. 26 INTRODUCTION TO CUDA 26 … as well as the number of threads in each block GPU performWork<<<2, 4>>>()

27. 27 INTRODUCTION TO CUDA 27 Every block in the grid contains the same number of threads GPU performWork<<<2, 4>>>()

28. 28 INTRODUCTION TO CUDA 28 Inside kernels definitions, CUDA-provided variables describe its executing thread, block, and grid GPU performWork<<<2, 4>>>()

29. 29 INTRODUCTION TO CUDA 29 gridDim.x is the number of blocks in the grid, in this case 2 GPU performWork<<<2, 4>>>() 2

30. 30 INTRODUCTION TO CUDA 30 blockIdx.x is the index of the current block within the grid, in this case 0 GPU performWork<<<2, 4>>>() 0 1

31. 31 INTRODUCTION TO CUDA 31 blockIdx.x is the index of the current block within the grid, in this case 1 GPU performWork<<<2, 4>>>() 0 1

32. 32 INTRODUCTION TO CUDA 32 Inside a kernel blockDim.x describes the number of threads in a block. In this case 4 GPU performWork<<<2, 4>>>() 4

33. 33 INTRODUCTION TO CUDA 33 All blocks in a grid contain the same number of threads GPU performWork<<<2, 4>>>() 4 4

34. 34 INTRODUCTION TO CUDA 34 Inside a kernel threadIdx.x describes the index of the thread within a block. In this case 0 GPU performWork<<<2, 4>>>() 0 1 2 3 0 1 2 3

39. 39 INTRODUCTION TO CUDA 39 Inside a kernel threadIdx.x describes the index of the thread within a block. In this case 1 GPU performWork<<<2, 4>>>() 0 1 2 3 0 1 2 3 performWork<<<2, 4>>>() 0 1 2 3 0 1 2 3

42. 42 42 performWork<<<2, 4>>>() GPU DATA GPUGPU 0 1 2 3 0 1 2 3 0 1 2 3 4 5 6 7 0 1 4 4 threadIdx.x + blockIdx.x * blockDim.x 0 0 0 dataIndex 0

50. 50 INTRODUCTION TO CUDA 50 We can also have a two- or three-dimensional thread block or grid performWork<<<2, (4, 4)>>>() 0 1 2 3 0 1 2 30 1 2 3 0 1 2 3 0 1 2 3 threadIdx.y threadIdx.x threadIdx.x

51. 51 GPU ACCELERATED COMPUTING IN PYTHON WITH NUMBA 51 CUDA-provided variables for thread indices Map our thread indices to some work Decorator to just-in-time compile CUDA

52. 52 52 GPU ACCELERATED COMPUTING IN PYTHON WITH NUMBA

53. 53 53 GPU ACCELERATED COMPUTING IN PYTHON WITH NUMBA Function prototype with decorator Work is automatically vectorized Kernel launch, data moved automatically

54. 54 RAPIDS — OPEN GPU DATA SCIENCE Software Stack Data Preparation VisualizationModel Training CUDA PYTHON APACHE ARROW DASK DEEP LEARNING FRAMEWORKS CUDNN RAPIDS CUMLCUDF CUGRAPH

55. 55 cuDF for GPU-accelerated DataFrames 55 cuDF

56. 56 Reading in Data 56 cuDF

57. 57 DataFrame from Pandas 57 cuDF

58. 58 DataFrames are PrettyPrinted 58 cuDF

59. 59 Integer Slicing for Row Selection 59 cuDF

60. 60 Column Selection by Column Name 60 cuDF

61. 61 Column Selection by Column Name 61 cuDF

62. 62 62 Slicing by Row and Column Using loc

63. 63 63 Slicing by Row and Column Using loc

64. 64 64 Compute a New Column “norm”

65. 65 65 Compute a New Column “norm”

66. 66 66 Construct a Boolean Series

67. 67 67 Select Data With Boolean Series

68. 68 68 Unique Elements

69. 69 69 User-Defined Functions

72. 72 Large set of vectorized operations are fast and convenient. Kernels can be used to define custom vector functions and expose CUDA parallelism. cuDF

73. 73 73 cuDF

74. 74 Simple supervised ML algorithm that classifies a test point (green) based on the labels of its nearest neighbors within a radius, K. If the radius is 5 (solid circle) the test point will be labeled as a red triangle because there are two red triangles within the circle. If the chosen radius is 7 (dashed circle) the test point will be labeled as a blue square. K nearest neighbors cuML

75. 75 Density-based clustering algorithm: given a set of points in some space, it groups together points that are closely packed together (points with many nearby neighbors), marking as outliers points that lie alone in low-density regions (whose nearest neighbors are too far away). DBSCAN is one of the most common clustering algorithms and also most cited in scientific literature. Density-based spatial clustering of applications with noise (DBSCAN) cuML

76. 76 Unsupervised algorithm that assigns divides unlabeled data into K clusters. It is an iterative algorithm that assigns each data point to a cluster based on its proximity to the centroids of the current clusters. This (re)assignment perturbs the centroids of each cluster. Iteration continues until assignments stabilize. K means clustering cuML

77. 77 A matrix factorization that expresses matrix M as a product: Truncated version is an approximate decomposition. Applications: Signal processing, data compression, image processing, satellite communication, wireless communication, least-squares fitting. Singular Value Decomposition (SVD) cuML

78. 78 An orthogonal transformation that identifies the basis which contains the highest variance. Applications: Compression, dimensionality reduction, signal processing, neuroscience, statistics, exploratory data analysis, and many disciplines of science and engineering. Principal Component Analysis (PCA) cuML

79. 79 Supervised decision tree ensemble algorithm that implements gradient boosting. Can be used for regression or classification. Extremely popular, especially on Kaggle. XGBoost

80. 80 CURRENTLY SUPPORTED ALGORITHMS ● Density-Based Spatial Clustering of Applications with Noise (DBSCAN) ● K-Means ● Principal Components Analysis (PCA) ● Truncated Singular Value Decomposition (tSVD) ● Uniform Manifold Approximation and Projection (UMAP) ● t-Distributed Stochastic Neighbor Embedding (TSNE) ● Regression (OLS) ● Linear Regression with Lasso or Ridge Regularization ● ElasticNet Regression ● Logistic Regression ● Stochastic Gradient Descent (SGD), (CD), (QN) (including L-BFGS and OWL-QN) solvers for linear models ● Random Forest (RF) Classification ● Random Forest (RF) Regression ● K-Nearest Neighbors (KNN) Classification ● K-Nearest Neighbors (KNN) Regression ● Support Vector Machine Classifier (SVC) ● Linear Kalman Filter ● Holt-Winters Exponential Smoothing ● Auto-regressive Integrated Moving Average (ARIMA) 80 v0.13.0a

CUDA-Python and RAPIDS for blazing fast scientific computing

CUDA-Python and RAPIDS for blazing fast scientific computing

Recommandé

Contenu connexe

Tendances

Tendances(20)

Similaire à CUDA-Python and RAPIDS for blazing fast scientific computing

Similaire à CUDA-Python and RAPIDS for blazing fast scientific computing(20)

Plus de inside-BigData.com

Plus de inside-BigData.com(20)

Dernier

Dernier(20)

CUDA-Python and RAPIDS for blazing fast scientific computing