3. 3
The Mission:
to Integrate and Harmonize the Infra and App Services
Making IT Infra Easy to Manage and Consume in the Form of Containers and VMs
Motivation Behind An Agile Hybrid Cloud Infra
IT
Administrator
We need scalable
infrastructure that’s
easy to manage and
easy for my users to
consume.
Developer /
DevOps / SRE
We need a place to
build my apps that’s
reliable, scalable, cost-
efficient, and can be
supported by our
internal IT team.
Kubernetes
(container)
Services
Network &
Security
Services
Storage
Services
Compute
Services
5. 5
Transform your infrastructure to build, run and manage modern applications (Spark included)
Introducing vSphere 7 with Kubernetes
Self-service Development
Developer
Kubernetes API
Container
service
Volume service
Network
service Registry service
vSphere 7: Developer Services
Code
Test
Deploy
Support
Agile IT Operations
Application
focused
management
VI Admin
Performance
Security
Availability
Cost
Diagnostics
vSphere 7: IT Operations
7. 7
Using Kubernetes to automate Kubernetes
Tanzu Kubernetes Grid (TKG)
Operator Desired State
Configuration
TKG Grid
Management
Cluster
AWS EC2 vSphere
• The Tanzu Kubernetes Grid
provides a consistent, upstream-
compatible implementation of
Kubernetes, that is tested,
signed, and supported by
VMware
• You can deploy Tanzu
Kubernetes Grid across your
vSphere and Amazon EC2
environments.
• The Tanzu Kubernetes Grid has
a native awareness of the multi-
cluster paradigm. You can
manage multiple clusters from a
single management location.
14. 14
Running Spark on a VMware Kubernetes platform
- Provides several advantages over Spark Standalone with no
performance degradation.
Spark on Kubernetes
Testing Spark on Kubernetes vs Spark Standalone
15. 15
First Round of Tests
The Spark Driver and Executor Components in Separate K8S Pods
16. 16
Spark on Kubernetes Performance Tests (1/2)
A 4-host Cluster that supports Spark Standalone as well as Spark on Kubernetes
4 2nd-Generation Intel® Xeon®
Scalable processor (“Cascade Lake”)
Servers, each with
2x Intel® Xeon® Platinum 8260
CPUs @ 2.4GHz
Total: 96 logical cores, 768 GB
Spark Standalone
1x Spark Master VM: 8 vCPU, 64 GB
16x Worker VMs: 16 vCPU, 120 GB
(Total 72 vCPU, 544 GB allocated
on 1st host)
Host 4
Host 1
Host 3
Host 2
Spark
Worker
Spark
Master
Spark
Driver
Spark
Worker
Spark
Worker
Spark
Worker
Spark
Worker
Spark
Worker
Spark
Worker
Spark
Worker
Spark
Worker
Spark
Worker
Spark
Worker
Spark
Worker
Spark
Worker
Spark
Worker
Spark
Worker
Spark
Worker
Need to install Spark on all nodes
17. 17
Spark on Kubernetes Performance Tests (2/2)
A 4-host Cluster which supports Spark Standalone as well as Spark on Kubernetes
4 2nd-Generation Intel® Xeon®
Scalable processor (“Cascade Lake”)
Servers, each with
2x Intel® Xeon® Platinum 8260
CPUs @ 2.4GHz
Total: 96 logical cores, 768 GB
Spark on Kubernetes
3x Master VM: 8 vCPU, 64 GB
1x HAProxy VM: 8 vCPU, 64 GB
16x Worker VMs: 16 vCPU, 120 GB
Host 4
Host 1
Host 3
Host 2
K8s
Worker
K8s Master
Spark
Driver
K8s
Worker
K8s
Worker
K8s
Worker
K8s
Worker
K8s
Master
K8s
Worker
K8s
Worker
K8s
Worker
K8s
Worker
K8s
Master
K8s
Worker
K8s
Worker
K8s
Worker
K8s
Worker
K8s
HAProxy
K8s
Worker
K8s
Worker
K8s
Worker
Install Spark only on K8s Master
1 VM = 1 K8s Worker Node
= 1 Spark Executor Pod
= 1 or more Spark Executors
18. 18
spark-submit can call a K8s master rather than a Spark master using k8s:// prefix with
URI
Recommended step: Create private namespace for Spark to enable Spark-specific
resource quotas and access control policies
kubectl create namespace spark
Required step: Create Spark service account and give it the ability to create pods
kubectl create serviceaccount spark --namespace=spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=spark:spark --namespace=spark
Spark Cluster mode (Spark driver runs in cluster) vs Client mode (driver runs in VM)
Submit Spark job to Kubernetes using spark-submit
Spark on Kubernetes: Performance Test Setup
19. 19
Performance scaled well for the Spark tests as the number of VMs increased from 4 (1 host server) to 16 (4 servers), with
Spark Standalone and Spark on Kubernetes within ~1% of each other. No performance degradation seen
Deep Learning Image Classification on Spark with BigDL - Test
Spark Standalone vs. Spark on Kubernetes Performance
28. 28
• A unified hybrid cloud platform that combines the agility of Kubernetes and the enterprise
capabilities of vSphere offers a compelling development and deployment system.
• Kubernetes delivers multiple simplified methods of deployment and execution of Spark and
other distributed applications.
• You can easily get started with Spark running on Kubernetes using the Bitnami Helm charts.
• We show comparable performance of Spark-based apps on VMs with and without Kubernetes.
• Kubernetes is becoming the method of choice for deploying both the training and inference parts
of ML.
• Machine and Deep Learning applications perform very well when deployed on Kubernetes on
vSphere.
• Tabular, structured datasets remain an important part of machine learning deployments.
Conclusions
29. 29
High Performance Virtualized Spark Clusters on Kubernetes for Machine Learning
https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/spark-k8s-vsphere67-
perf.pdf
Machine Learning on vSphere : http://blogs.vmware.com/apps/ml
Fast Virtualized Hadoop and Spark On All-Flash
Disks https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/bigdata-
vsphere65-perf.pdf
Virtualized Hadoop Performance with VMware vSphere 6 on High Performance Servers
http://www.vmware.com/resources/techresources/10452
Big Data Performance on vSphere 6
https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/bigdata-perf-vsphere6.pdf
Big Data Performance on VMware Cloud on AWS https://www.vmware.com/techpapers/2018/big-data-vmware-cloud-
aws-perf.html
References