Toon is a leading brand in the European smart energy market, currently expanding internationally, providing energy usage insights, eco-friendly energy management and smart thermostat use for the connected home. As value added services become ever more relevant in this market, we have the need to ensure that we can easily and safely on-board new tenants into our data platform. In this talk we’re going to guide you across a less discussed side of using Spark in production – devops. We will speak about our journey from an on-premise cluster to a managed solution in the cloud. A lot of moving parts were involved: ETL flows, data sharing with 3rd parties and data migration to the new environment. Add to this the need to have a multi-tenant environment, revamp our toolset and deploy a live public facing service. It’s possible to find a lot of great examples of how Spark is used for data-science purposes. On the data engineering side, we need to deploy production services, ensure data is cleaned, secured and available, and keep the data-science teams happy. We’d like to share some of the options we took and some of the lessons learned from this (ongoing) transition.

1. Telmo Oliveira, Toon
Using Spark in the Cloud:
A Devops perspective

4. 4put your #assignedhashtag here by setting the footer in
view-header/footer

5. 5put your #assignedhashtag here by setting the footer in
view-header/footer

6. 6put your #assignedhashtag here by setting the footer in
view-header/footer

8. 8

9. 9

10. Requirements
10
• Seamless transition
• Ensure data anonymity
• Move fast, optimise later
• Ensure multi-tenancy
• As little disturbance as possible to the DS team

11. 11

12. 12

13. 13

14. 14
• Cluster timeouts
• Autoscaling
• Spot instances
• Well documented API

15. 15
Infrastructure as code
• Repeatability
• Fast deployment
• Resilience
• Documentation

16. 16

17. 17
Terraform
• S3 Buckets
• EC2 instances
• Network topology
• Log management
• RDS instances
• IAM roles/policies

18. 18
Terraform
• S3 Buckets
• EC2 instances
• Network topology
• Log management
• RDS instances
• IAM roles/policies

19. 19
Ansible
• User management
• Databases and ACLs
• Custom app deployment

20. 20
Ansible
• User management
• Databases and ACLs
• Custom app deployment

21. 21
Architecture
Overview

22. 22

23. Airflow
23

24. 24

25. 25

26. 26

27. 27

28. 28
• External Hive metastore
• Send logs to S3
• Authorisation
• i3.2xlarge nodes

29. Future plans
29
• Streaming
• Real time services
• Improve CI/CD

30. What’s all this for?
30

31. What’s all this for?
31

32. 32
Thanks to the team
Aemro Amare
Barend Garvelink
Bert Jan Katsman
Kliment Markovski
Miquel Monreal
Stanislava Potupchik

33. Questions?