Inside Microsoft's FPGA-Based Configurable Cloud
•
6 j'aime•3,446 vues
At Microsoft’s annual developers conference, Microsoft Azure CTO Mark Russinovich disclosed major advances in Microsoft’s hyperscale deployment of Intel field programmable gate arrays (FPGAs). These advances have resulted in the industry’s fastest public cloud network, and new technology for acceleration of Deep Neural Networks (DNNs) that replicate “thinking” in a manner that’s conceptually similar to that of the human brain. Watch the video: http://wp.me/p3RLHQ-gNu Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Inside Microsoft's FPGA-Based Configurable Cloud
Similaire à Inside Microsoft's FPGA-Based Configurable Cloud (20)
Plus de inside-BigData.com
Plus de inside-BigData.com (20)
Dernier
Dernier (20)
Inside Microsoft's FPGA-Based Configurable Cloud
- 2. A faster, more efficient, more intelligent cloud Data explosion: 2013 4.4 ZB - 2020 44 ZB ML, DNN, AI are driving requirements up faster Autonomous decision making Real-time insights into connected devices Interactive user experiences Cloud-scale services Searches and recommendations (Indexing the Internet!) The need for SCALE The need for LOW-LATENCY The need for THROUGHPUT 2013 2020 4.4 ZB 44 ZB Source: IDC 2014
- 3. FPGAs EVALUATION CPUs and FPGAs, ASICs under investigation EFFICIENCY TRAINING CPUs and GPUs, limited FPGAs, ASICs under investigation Control Unit (CU) Registers Arithmetic Logic Unit (ALU) + + + + + + + FLEXIBILITY CPUs GPUs ASICs
- 5. FPGA: spatial compute FPGA 1001010011101011101100111001111001110101 0110001 0100101001110101110110011100111100111010 1101110 1010011101011101100111001111001110101100 1011001 Data Instruction Instruction Instruction 100101001110101110110011100111100111 0101 Data Instruction Instruction Instruction CPU: temporal compute CPU Instruction
- 6. Catapult v0 Catapult v1 Scale v1 Catapult v2 2011 2012 2013 2014 2015 2016 … Ignite unveiling Production
- 8. WCS Gen4.1 Blade with NIC and Catapult FPGA Catapult v2 Mezzanine card
- 10. Azure Virtual Network Virtual network “Bring your own network” Segment with subnets and network security groups Control traffic flow with user defined routes Backend connectivity Point-to-site for dev/test VPN Gateways for secure site-to-site connectivity ExpressRoute for private enterprise grade connectivity Backend connectivity ExpressRoute VPN Gateways Users Internet Front-end access Dynamic/reserved public IP addresses Direct VM access, ACLs for security Load balancing DNS services: hosting, traffic management DDoS protection
- 11. Management Control Data Proprietary appliance Management plane Create a tenant Control plane Plumb tenant ACLs to switches Data plane Apply ACLs to flows Azure Resource Manager Controller Switch (Host) Management plane Data plane SDN Control plane Key to flexibility and scale is Host SDN
- 12. Acts as a virtual switch inside Hyper-V VMSwitch Provides core SDN functionality for Azure networking services, including: • Address Virtualization for VNET • VIP -> DIP Translation for SLB • ACLs, Metering, and Security Guards Uses programmable rule/flow tables to perform per-packet actions Available for Private Cloud in Microsoft Azure Stack VM Switch VFP VM VM ACLs, Metering, Security VNET SLB (NAT)
- 13. VMSwitch exposes a typed Match-Action-Table API to the controller Controllers define policy One table per policy Key insight: Let controller tell switch exactly what to do with which packets e.g. encap/decap, rather than trying to use existing abstractions (tunnels, …) Tenant Description VNet Description VNet Routing Policy ACLs NAT Endpoints Flow Action TO: 10.2/16 Encap to GW TO: 10.1.1.5 Encap to 10.5.1.7 TO: !10/8 NAT out of VNET Flow Action TO: 79.3.1.2 DNAT to 10.1.1.2 TO: !10/8 SNAT to 79.3.1.2 Flow Action TO: 10.1.1/24 Allow 10.4/16 Block TO: !10/8 Allow VNET LB NAT ACLS VFP Controller VM 1 10.1.1.2
- 14. Hosts are Scaling Up: 1G à 10G à 40G à 50G à 100G Reduces COGS of VMs (more VMs per host) and enables new workloads Need the performance of hardware to implement policy without CPU Need to support new scenarios: BYO IP, BYO Topology, BYO Appliance We are always pushing richer semantics to virtual networks Need the programmability of software to be agile and future-proof “How do we get the performance of hardware with programmability of software?
- 15. Use an FPGA for reconfigurable functions FPGAs are already used in Bing (Catapult) Roll out Hardware as we do software Programmed using Generic Flow Tables (GFT) Language for programming SDN to hardware Uses connections and structured actions as primitives Deployed on all new Azure compute servers since late 2015 SmartNIC can also do Crypto, QoS, storage acceleration, and more… Host SmartNIC FPGA ToR NIC ASIC SmartNIC CPU
- 16. VM VFP Southbound API GFT Offload API (NDIS) VMSwitch Northbound API GFT Table First Packet GFT Offload Engine 50G QoSCrypto RDMA GFT Transposition Engine REWRITE SLB Decap SLB NAT VNET ACL Metering ControllerControllerController Encap SmartNIC DNATDecap Allow Meter Rule Action * Meter Rule Action * Allow Rule Action * Rewrite Rule Action * DNAT Rule Action * Decap Flow Action 1.2.3.1->1.3.4.1, 62362->80 Decap, DNAT, Rewrite, Meter Flow Action 1.2.3.1->1.3.4.1, 62362->80 Decap, DNAT, Rewrite, Meter
- 17. SDN/Networking policy applied in software in the host FPGA acceleration used to apply all policies VM 1 VM 2 Virtual switch Physical server 1 Physical switch Virtual switch Physical server 2 Virtual Network VM 1 VM 2 Physical switch Virtual Network
- 18. The fastest cloud network Highest bandwidth VMs of any cloud DS15v2 & D15v2 VMs get 25Gbps Consistent low latency network performance Provides SR-IOV to the VM Up to 10x latency improvement Increased packets per second (PPS) Reduced jitter means more consistency in workloads Enables workloads requiring native performance to run in cloud VMs >2x improvement for many DB and OLTP applications
- 21. New 50GbE SmartNIC for Project Olympus (Announced at OCP 2017)
- 23. Deep neural networks (DNN) have led to breakthroughs in major AI problems Computer vision Language translation Speech recognition And more… But DNNs are challenging to serve in online services Latency, cost, and power-constrained Size and complexity of DNNs outpacing growth of CPUs DNN
- 25. Microsoft has the world’s largest cloud investment in FPGAs Multiple Exa-Ops of aggregate AI capacity We have built powerful DNN serving platform on our FPGA fabric FPGAs ideal for adapting to rapidly evolving ML CNNs, LSTMs, MLPs, reinforcement learning, feature extraction, decision trees, etc. Inference-optimized numerical precision Custom binarized, ternarized, tiny precision nets Sparsity, deep compression for larger, faster models Tens to hundreds of TOPS of effective inference throughput at low batch sizes Ultra-low latency serving on modern DNNs >10X better than CPUs and GPUs Scale to many FPGAs in single DNN service Performance Flexibility Scale
- 26. software FPGA 99.9% Query Latency versus Queries/sec HWvs.SWLatencyandLoad average software load 99.9% software latency 99.9% FPGA latency average FPGA query load
- 30. Traditional software (CPU) server plane QPI CPUCPU QSFP TOR40Gb/s Web search ranking
- 31. Web search ranking Traditional software (CPU) server plane QPICPU QSFP 40Gb/s ToR FPGA CPU 40Gb/s QSFP QSFP Hardware acceleration plane Interconnected FPGAs form a separate plane of computation Can be managed and used independently from the CPU Web search ranking Deep neural networks SDN offload SQL
- 32. Flexibility: many services need a large number of FPGAs, others underutilize theirs Deploy exactly as many instances as needed Many accelerators can handle load of multiple software clients Consolidate underutilized FPGA accelerators into fewer shared instances Increases efficiency & makes room for more accelerators Many services need to access multiple types of accelerators
- 33. F F F L0 L1 F F F L0 Pretrained DNN Model DNN Hardware Microservice DNN Engine Instr Decoder & Control Neural FU
- 34. CPU FPGA CPU FPGA
- 35. CPU FPGA CPU FPGA
- 36. Low-Level AI Representation (LLAIR) & Federated Runtime Customer DNN Model (TF, CNTK, etc) Hosted FPGA-powered Service in Azure FPGA0 FPGA1 Add500 1000-dim Vector 1000-dim Vector Split 500x500 Matrix MatMul500 500x500 Matrix MatMul500 MatMul500 MatMul500 500x500 Matrix Add500 Add500 Sigmoid500 Sigmoid500 Split Add500 500 500 Concat 500 500 500x500 Matrix
- 37. Host Ranking Service LTL Host FE FPGA Ranking Service LTL Host Free FPGA Ranking Service LTL Host DNN FPGA Ranking Service LTL Host FE FPGA Host LTL LTL
- 38. CPU compute layer Reconfigurable compute layer Converged network
- 39. We look forward to eventually making this available to you, a major step toward democratizing AI with the power of FPGA àOur technology will push the boundary of what is possible to deploy in the cloud Deeper convolutional neural networks for more accurate computer vision Higher dimensional recurrent neural networks toward human-like natural language processing State-of-the-art translation and speech recognition And much more… This technology is already powering services within Microsoft