About the Talk: Cloud native ecosystem is bringing a huge change in the way of DevOps in every cloud native organisation. Developers and operators in cloud native organisations are using tools and platforms like Kubernetes to achieve the agility promised by DevOps and microservices. The tools and best practices for stateless applications have been well established and the results can be seen in the agility of teams using these stateless applications. However, stateful applications pose new challenges to DevOps teams in achieving the agility as the best practices around persistent storage management are still emerging. In this talk, first we discuss the challenges faced by DevOps while dealing with persistent storage handling in stateful applications. Then we discuss the open source tools and best practices for DevOps teams to achieve data agility of cloud native applications.

1. Cloud Native Storage &
Data Agility for DevOps
@uma_mukkara
12 October 2018

2. ● MayaData delivers Data
Agility
● Sponsor of OpenEBS and
Litmus projects
● Large Bangalore based OSS
contributor
● https://mayaonline.io is SaaS
software used to manage
stateful workloads on
Kubernetes
Uma Mukkara
Co-founder & COO

3. ● Popular open source
storage project for stateful
applications on Kubernetes
● Team of 50+ hackers with
storage and DevOps
experience
● Leading project in CAS category
● 250+ contributors
● 5K+ github stars
● 1M+ docker pulls
slack.openebs.io

4. DevOps Micro-
services
Agility
Cloud Native
technologies
and practices
Docker Kubernetes
Istio OpenEBS
HelmPrometheus
Adopt
Enable
Achieve

5. Cloud
Native
Tech
Agility
Application
Agility
Data
Agility
Container Attached Storage
Enables

6. Cloud native technologies empower organizations to build and run
scalable applications in modern, dynamic environments such as
public, private, and hybrid clouds. Containers, service meshes,
microservices, immutable infrastructure, and declarative APIs
exemplify this approach.
These techniques enable loosely coupled systems that are
resilient, manageable, and observable. Combined with robust
automation, they allow engineers to make high-impact changes
frequently and predictably with minimal toil.
Definition from
What is “Cloud Native” ?

7. Data Agility Challenges
in Cloud Native Space

8. 1. HA within a cluster
• Microservices architecture separates every component. A
legacy application now becomes
• A stateless application and/or
• A stateful application with a database in it
• Stateless applications are mobile across nodes in a cluster
but Stateful applications need persistent data for easy
movement
Synchronous replication of cloud native data

9. 2. HA across clusters
• multi-cloud is now a reality
• Multi Cloud example
• Code is in Github
• Dev uses On-Prem
• CI/CD in Google
• Pre-Prod in Azure
• Production in AWS
Move data across clouds

10. 3. CI Pipelines
• State of the data moves to CI pipelines
• Developer needs access to the exact state of the data
when a pipeline fails
• Pipelines have to snapshot the data
• Pipelines have to clone the data and automate the
access to the developer
Data snapshots in CI pipelines

11. 4. E2E Tests
• For gaining credibility to push the stateful application into
production, building an E2E test suite is a must
• The stateful application that is containerized need to be
tested for all state transitions and changes in the
infrastructure
• E2E matrix for stateful applications is large and complex
App specific data resilience and availability

12. 5. Data Backup and Restore
• The traditional Backup and Restore systems do NOT work
in Cloud Native eco-systems
• Irrespective of the reliable infrastructure designs that
Cloud Native systems provide, DevOps still need to have
Backup and Restore strategy
• Converting the application into microservices architecture
is the one challenge but finding and
integrating/automating the Backup/Restore process is
another
Backup and DR for Cloud Native Data

13. Choose Cloud Native
technologies to resolve
Data Agility challenges

14. Cloud Native Tech for Data Agility
* These are applicable when you are using Kubernetes as Orchestration platform
Synchronous replication of cloud native data
Move data across clusters, clouds
Data snapshots in CI pipelines
App specific data resilience and availability
Backup and DR for Cloud Native Data
Data Agility Challenge
Choose any CAS like OpenEBS
Choose any CAS like OpenEBS
OpenEBS & Litmus
Litmus
Heptio ARK with OpenEBS Plugin
Cloud Native Technology

15. Cloud Native Tech for Data
Agility

16. What if storage for container native applications was itself
container native?
https://www.cncf.io/blog/2018/04/19/container-attached-storage-a-primer/
Blog
White Paper
https://openebs.io/assets/docs/WP-OpenEBS-0_7.pdf
Container Attached Storage or CAS - OpenEBS

17. Container Attached Storage or CAS - OpenEBS
Architecture

18. Container Attached Storage or CAS - OpenEBS

19. Chaos Engineering Framework - Litmus
Litmus- Chaos Engineering Framework for Stateful applications on Kubernetes
It is a K8S JOB
K8S
deployers
Provider
specific
deployers
Test cases (ansible
playbooks)Framework
contains Standard & customized
Workload containers
Tools to help
execute tests
(Chaos, logging etc)

20. Litmus - For CI Pipelines / E2E
Chaos Engineering Framework - Litmus

21. Backup and Restore framework - Heptio ARK+OpenEBS
Heptio ARK
OpenEBS Plugin
+
Cloud Native
incremental
data backup

22. Backup and Restore framework - Heptio ARK+OpenEBS

23. DevOps Data Agility
Summary

24. DevOps
Application
agility
Data
agility
Cloud Native tools and best practices
(sufficient expertise is already in the public domain)
Use CAS for data
storage and data
resiliency
Build
Snapshot/aware
CI pipelines
Build data aware
E2E suites
Choose Cloud
Native Backup
and DR tools

25. Data Agility Example
DevOps
Developer
DevOps
Architect
Dev Cluster
CI/CD Cluster
Production Cluster
1.Develop
Application YAML
Volume:
Name : Volume 1
Mount : /mnt/volume
PVC : OpenEBS-PVC
Persistent volumes are provisioned
automatically for developer.
Code is merged2. Code
is
merged
3. CI/CD Pipeline Build
Stage
Setup
Stage
Litmus CD
4. Deploy

26. Data Agility Example
DevOps
Developer
DevOps
Architect
Dev Cluster
CI/CD Cluster
Production Cluster
1.Develop
2. Code
is
merged
3. CI/CD Pipeline Build
Stage
Setup
Stage
Litmus
One test fails
Snapshot
Clone
Access clone and debug

27. Data Agility Example
DevOps
Developer
DevOps
Architect
Dev Cluster
CI/CD Cluster
Production Cluster
1.Develop
2. Code
is
merged
3. CI/CD Pipeline Build
Stage
Setup
Stage
Litmus CD
4. Deploy
5. Chaos
Engineering
using Litmus

28. Data Agility Example
DevOps
Developer
DevOps
Architect
Dev Cluster
CI/CD Cluster
Production Cluster
1.Develop
2. Code
is
merged
3. CI/CD Pipeline Build
Stage
Setup
Stage
Litmus CD
4. Deploy
6. Backup
Heptio Ark +
OpenEBS

29. Data Agility . Delivered

30. QUESTIONS?

31. We are Hiring DevOps Engineers
jobs@mayadata.io