In order to provide prompt results and efficiently deal with data-intensive workloads, Big Data applications execute their jobs on compute slots across large clusters. Also, for optimal performance, these applications should be as close as possible to the data they use. Data-aware scheduling is the way to achieve that optimization and can conveniently be set up using Kubernetes. We’ll present two different use cases: First, we’ll make use of how Big Data applications like Hadoop and Spark can use their native HDFS protocol for data-aware scheduling. Second, we’ll demonstrate an efficient way to write a data-aware scheduler for Kubernetes that satisfies not just your application’s requirements, but also keeps your admins happy. As a bonus, it’ll also allows us to run data-aware scheduling on applications other than Big Data. Event: Kubernetes Meetup Rhein-Neckar, 18.10.2017 Speaker: Johannes M. Scheuermann weiter Tech-Vorträge: https://www.inovex.de/de/content-pool/vortraege/ Tech-Artikel in unserem Blog: https://www.inovex.de/blog/

1. Data-aware scheduling
Spark on Kubernetes with HDFS
Johannes M. Scheuermann
Karlsruhe, 18.10.2017

2. Johannes M. Scheuermann
IT Engineering & Operations @ inovex
〉 Software-Defined Datacenters
〉 Infrastructure as Code
〉 Cloud technologies
〉 High Availability & Scalability
〉 Want‘s an IBM Z-Frame
〉 @johscheuer
2

3. Data-aware scheduling

4. • Why data-aware scheduling
• Data-ware for non Big-Data application
• Data-ware scheduler
• Big Data on Kubernetes
• Spark on Kubernetes
• HDFS on Kubernetes
Agenda

5. • I’m not a scheduling expert
• Concept is/was a PoC
• Share learnings/ideas
• Get feedback from the community
Spoiler(s) J

6. Data-locality

7. Why data-locality?

8. Data-aware scheduling for non Big-Data
• Databases
• (large) image processing
• Video encoding
• (Web)-Cache

9. • Distributed (parallel) POSIX file system
• Any workload with high performance (incl. throughput,
databases, small files)
• Can be deployed in containers, on kubelet hosts.
• Linearly scalable performance.
• Fully fault-tolerant, split-brain safe
Quobyte – What is Quobyte

10. Quobyte - Architecture

11. • Metadata servers make placement decisions against
policies
• on file level
• tiering, isolation, …
• keep stripes of files on disks of same machine => enable local read
• allow preferring writes to local storage servers => enable local
write
• Locality information can be retrieved per file
• that’s where the scheduler hooks in
Quobyte - Placement

12. Running multiple schedulers

13. • Specify wanted Data
• Lookup Data Placement
• Remapping if Storage runs in Containers
• Schedule Pod
Scheduling data-aware (file-based)

14. Scheduler Architecture (4000ft)

15. Scheduler Architecture (1000ft)

16. Scheduler Architecture (containerized)

17. Benchmarks

18. (Spark) Big-Data on Kubernetes

19. • https://github.com/apache-spark-on-k8s/spark
• Not the “faked” Spark on Kubernetes
• Still in development
• Still not in the official Apache Spark project
• Current: v2.2.0-0.4.0
• Alpha/Beta ?!
Spark on Kubernetes

20. Spark on Kubernetes
Spark Core
MesosYARNStandaloneKubernetes
StreamingMLlibSparkSQLGraphX

21. • Integrates with Kubernetes
• RBAC
• Resource Quotas
• Audit logging
• Etc.
• Only cluster-mode
Spark on Kubernetes

22. Spark on Kubernetes (cluster-mode)

23. Spark + HDFS on Kubernetes
Driver 1
Driver 2
Executor 1
Executor 2.1
HDFS NN
HDFS DN 1 HDFS DN 2
Pod
Network
Host
Network
Executor 2.2

24. Demo

25. • Rack-locality
• Node preferences
• Priority-based scheduling (K8s 1.8 alpha)
• NameNode HA
• Kerberos support
Missing pieces

26. Conclusions

27. • Good starting point
• Good integration
• Still some points open
• Work for better integration (more general)
• Play with it!
Conclusions

28. 28
We are hiring!
www.inovexperts.com

29. Q&A

30. • https://issues.apache.org/jira/browse/SPARK-18278
• https://www.youtube.com/watch?v=0xRHONrWwvU&
feature=youtu.be
• https://www.youtube.com/watch?v=DxCDxi08HWo&f
eature=youtu.be
Further reading

31. Johannes M. Scheuermann
inovex GmbH
Johannes.scheuermann@inovex.de
CC BY-NC-ND inovex.de +JohannesScheuermann
github.com/johscheu
er
@johscheuer youtube.com/inovexGmb
H