Lyft is on the mission to improve people’s lives with the world’s best transportation. Starting 2019, Lyft has been running both Batch ETL and ML spark workloads primarily on Kubernetes with the Apache Spark on k8s operator. However, with the increasing scale of workloads in frequency and resource requirements, we started hitting numerous reliability issues related to IP allocation, container images, IAM role assignment, and Kubernetes Control Plane. To continue supporting growing Spark usage with Lyft, the team came up with a hybrid architecture optimized for containerized and non-containerized workload based on Kubernetes and YARN. In this talk, we will also cover a dynamic runtime controller that helps with per environment config overrides and easy switchover between resource managers.

1. Hybrid Spark
Architecture with
Yarn and Kubernetes
Catalin Toda (Sr Engineer @ Lyft)
Rohit Menon (Staff Engineer @ Lyft)

2. Agenda
▪ Spark @ Lyft
▪ Challenges with K8s
▪ Hybrid Model
▪ Spark Operator
▪ Image Hierarchy
▪ Spark Wrapper
▪ Progress & Future Plans

3. What is Spark Used For @ Lyft
• Primarily Python Shop with some Scala
• Running in AWS with S3 as permanent storage
• Interactive Development & ML with Jupyter and Spark
• ML Batch Use Cases
• Pricing
• ETA/Routing
• Mapping
• ETL Use Cases
• Event Ingestion
• GraphQL Offline Model
• Financial Datasets / SOX Complaint Datasets / Experimentation Offline Batch
Analysis and many more
• HiveQL to Spark-SQL Migration

4. 2018 Spark on Yarn
• Every major use case had its own ephemeral yarn
cluster
• Management overhead for infra team
• Custom Dependency management per cluster to pull
in python dependencies
• Tough to test/maintain cluster bootstrap scripts
• Custom IAM role/permission overhead

5. 2019 Spark On Kubernetes
• Lyft Infra supported Kubernetes deployment
• Google OSS spark-on-k8s operator availability
• Flyte (Container native scheduler) took off in Lyft for
ML use case
• Containerized workloads with easier python
dependency management
• Simpler support for per job IAM roles

6. Spark On Kubernetes Architecture

7. 2020 Spark on Kubernetes
• Maturing support for Spark on K8s
• Lyft Hadoop/Hive infrastructure as K8s deployment
• Auto-scaling handled for YARN cluster based on RM
load
• Spark ETL workloads move over from Yarn to Spark
on k8s
• Start hitting limits with Lyft k8s Infra setup
• Custom solutions required to support growing scale
• Group jobs to reduce spiky requests to k8s and AWS control plane
• Add new k8s clusters to support stronger isolation model

8. Current Spark Scale on K8s
- 1000 concurrent jobs
- 5 Kubernetes Clusters
- 20k executors at peak
- 1 AWS region
- 5 AWS Availability Zones

9. Challenges with K8s Model
• IPv4 Address Shortage
• Shortage across all 5 AZs
• Leads to driver and executors
startup delays
- IAM Wait Delays
- AWS IAM assignment could be
throttled
- IAM wait to assure assignment
increasing delays
- Infrastructure issues
- Etcd size tuning
- Impact of bad k8s node

10. Challenges with K8s Model
• Image Overheads
• Every project has their own image
• Registration of images for different
environments
• Startup delays caused by uncached image
• New nodes
• New image releases
• Release model
• Infra prepares a base image with Spark
latest changes
• Customers manage final release when the
image is tested
• Leads to maximum 1 month rollout time due
to several images to be updated

11. Challenges with K8s Model
- K8s scheduling
• Fixed per namespace quota
• Containers not admitted if over quota
• No priority between jobs in namespaces
- Control plane limits
- Maximum number of containers
- Short running containers are not typical K8s
workload
• Hive Deprecation
• Today hive scale = 5k jobs at peak
• Expected spark load to increase by 6x
• For interactive workload pod startup time
latency is high on k8s

12. 2021 Hybrid Model (YARN + K8s)
Separation by Workload Type
• Containerized (K8s)
• Pyspark + Custom Dependencies
• ML interactive
• Non-containerized (YARN)
• SQL Interactive
• SQL Batch
• Scala Workloads
• Simple Pyspark with no dependencies

13. Hybrid Model Architecture

14. Advantages of Hybrid Model
• YARN executors have low startup latency and can
handle spikes
• Easier Queue and Resource management
• Workloads without custom dependencies do not get
penalized with k8s infra overheads
• Mature support for dynamic allocation and external
shuffle service

15. Single Entry Point
• Spark-on-k8s-operator as single entry point for both YARN
and K8s
• Integrates easily open source
• Compatibility with Flyte
• No Lyft specific code
• Multi version branch by default
• Driver runs on k8s in client mode
• We plan to contribute this to OSS
• Current OSS design add overhead of spark submit pod
• Sets us infra team to move workloads seamlessly between
resource managers

16. Spark Wrapper Design
Stage 1
- Part of base image
- Downloads and runs stage2
Stage 2
- Manipulate configs
- Run spark driver
- Capture job logs and results
- Push application metrics

17. Spark Wrapper
• Custom image entry point
• Allows config management based on environments
• Allows switching between resource managers
• Metrics
• Push to events - queryable/dashboard using lyft stack
• Push to statsd for real time monitoring/alerting
• Integrates well with Lyft Infra
• Spark on k8s operator remains in sync with upstream
• Lyft specific logic that integrates with in-house tools
• Adds runtime controller to images

18. Image Hierarchy/Distribution
• One base image for spark per spark version
• Containerized spark extends base image
• Users can further extend containerized base image to add custom
dependencies
• ML base image
• Users maintain their own image
• Non-containerized use the base image directly
• Infra updates the image
• Consistent experience across use cases

19. Progress so far - Best of Both Worlds
- Spark driver startup < 1s
- Resource allocation managed in YARN
- K8s scale reduced by 20x
- IP addresses requirement reduced by 20x
- Per job IAM Roles using Web Identity provider

20. Progress so far - Best of Both Worlds
- No migration needed for containerized customers
- Python dependency management using an utility
library
- The latest version is synced in all environments
(adhoc, k8s, YARN)

21. Future Plans
- Consolidate Batch Compute on Spark (Hive -> Spark)
- Evaluate Data Lake technologies
- Continue to scale k8s and Spark Infrastructure

22. Conclusion
- YARN vs K8s
- Workload analysis is required before identifying the best solution
- For Lyft - existing YARN infrastructure helped choosing a hybrid model
- Fixing K8s model requires:
- K8s supports to IPv6 in the latest versions
- Scaling - Number of k8s clusters and a gateway to perform the routing between them
- Image - Design considerations/overheads with high number of images
- Quota - Investing in projects trying to solve this aspect
- Web Identity Provider - Custom Roles in K8s

23. Q & A
Contact Info:
ctoda@lyft.com
rmenon@lyft.com