Learn about ObjectRocket's adventures in Kubernetes. We'll cover why we chose Kubernetes for our DBaaS platform, the challenges we faced, and how we overcame them. A presentation for DevWeek Austin 2018.

1. DBaaS on Kubernetes
Peanut Butter and Jelly, or Oil and Water

2. About ObjectRocket
ObjectRocket helps you build better sites and apps, faster. You can concentrate on your mission without
having to worry about managing data stores.
2
Support
It’s the best hands-on support,
hands-down.
All of our customers get 24x7x365
support from database experts with
financially-backed SLAs.
ObjectRocket provides a
customized, high-touch, human-
centered approach to support.
Technology
Our Database-as-a-Service
platform is purpose-built from
the ground up with high
performance gear and flexibility
to architect a solution that fits
your specific workload.
Hassle-free hosting for
MongoDB®, Elasticsearch® +
KIbana®, and Redis®
Expertise
Our team of experts with decades
of experience understand how to
operate and maintain data stores
in complex production
environments.
MONGO®, MongoDB® and MongoDB® Design are registered trademarks of MongoDB, Inc. Redis® and the Redis® logo are trademarks of Salvatore Sanfilippo in the US and other countries. Elasticsearch® and Kibana®
are trademarks of Elasticsearch BV, registered in the US and in other countries.

3. Why we chose
Kubernetes for
our next DBaaS
platform
What You’ll See Today
The specific
challenges that
we faced
Our solutions to
those challenges
What we’re
doing next

4. Choosing a new DBaaS Platform

5. Our Original Hosting Platform
Built circa 2012 Heavily based on openVZ containers
Custom orchestration, hardware and
infrastructure management
Assumes a bare metal environment

6. What’s Changed
Built circa 2012 Heavily based on openVZ containers
custom orchestration, hardware and
infrastructure management
Assumes a bare metal environment
Not available
yet
6 years is a long time in
technology
Docker has become the de-
facto container format
Custom orchestration is not a
differentiator and awesome
standard tools exist
Cloud usage has grown and
not all need bare metal
performance

7. • Well-built, open source, modern orchestration
• First-class citizen in the clouds we want support
• People want to develop for it
• Operators make it easy… There are open source
options everywhere
• It’s 🔥🔥🔥🔥
Kubernetes + ObjectRocket: Like Peanut Butter and Jelly

8. Our Response

9. Kubernetes + ObjectRocket: Like Oil and Water
• Kubernetes not built for stateful applications and
databases are pretty damn stateful
• Most databases don’t tolerate disappearing resources
very well
• All operators aren’t easy… aaS offerings need a whole
lot more functionality

10. The Reality: This PB&J is not gonna make itself
• Stateful sets are a start, but there is a
whole lot more available in Kubernetes
now that makes it suitable for certain
databases.
• The types of databases we offer are
surprisingly tolerant of shifting resources.
• We can develop around it.
• There’s no operator easy button, but
there (so far) have always been solutions
to our trickiest scenarios.

11. Operators: TLDR

12. Kubernetes Operators
An Operator is a method of packaging, deploying and managing a Kubernetes application. A Kubernetes
application is an application that is both deployed on Kubernetes and managed using the Kubernetes
APIs and kubectl tooling.
Operators are a way to wrap business logic and manual operations around kubernetes features and
components.
Requests
Standard Resource
Requests
Custom Resource
requests
Actions Applied
Actions
CRD(s) Controller
Intercept events
Perform custom logic
Translate to standard resources
Register custom resources.

13. What Our Operator Will Manage
Service Features04
● Safely scale up/out/in/down
● Orchestrate support utilities
● Add-on dashboards and tooling
Database
Administration
03
● Cluster configuration and plugins
● Perform regular backups
● Apply minor and patch updates
● Running database utilities
Security02
● Database User CRUD
● Database User roles
● IP whitelist CRUD
● Cluster Certificates
The Basics01
● Safely create a full cluster
● Make sure the cluster is healthy
● Delete all resources when asked
● Handle multiple tenants

14. DBaaS Challenges in a ‘Stateless’ Landscape

15. Biggest Challenges
There are lots of small challenges, but the most interesting things we encountered
were:
• Safe Updates
• K8s Queuing behaviors
• Remote Execution
• State Management
• Supportability
• Velocity of the Kubernetes ecosystem

16. Safe Updates
The #1 concern for databases

17. The Challenge
How can we ensure that Elasticsearch cluster updates are performed safely?
• Version upgrades and some configuration changes require that nodes are restarted
• This should be accomplished without impacting database uptime and performance
1
1. Node disappears
2. Replicas on other nodes are promoted / New
replicas created on live nodes
3. When node returns to cluster, data reshuffles
4. When green, repeat for every node restart
2
3
The Wrong Way
2
1. Cluster allocation disabled
2. Node disappears
3. Replicas on other nodes promoted to primary,
but no data movement
4. When node returns to cluster, enable allocation
5. Elasticsearch verifies shards on returned node
6. When green, repeat for every node restart
The Right Way
5

18. • Unique network identifiers
• Persistent storage
• Ordered, graceful deployment and
scaling.
• Ordered, automated rolling updates.
Stateful Sets
StatefulSets are intended to be used with stateful applications and distributed
systems.

19. StatefulSet Rolling Update Strategies
OnDelete Strategy
.spec.updateStrategy.type = OnDelete
• Implements legacy (1.6 and prior) behavior.
• Controller will not automatically update each
pod in a StatefulSet.
• User manually deletes pods to cause the
controller to create new pods
• Must manage workflow to maintain uptime
RollingUpdate
.spec.updateStrategy.type = RollingUpdate
• Automated, rolling update of pods
• Controller will delete and recreate each
pod in the StatefulSet.
• Proceeds in the same order as pod
termination
• Waits until an updated pod is running and
ready prior to updating its predecessor.

20. Our Approach: Partitioned RollingUpdate
• RollingUpdate applies changes to pods in reverse order, from {N-1..0}.
• Kubernetes offers a couple of ways to safely tear down each pod:
• A grace period (time) is given before the pod is violently shutdown.
• preHooks: command/script to execute before the pod is violently shutdown.
• Partitioned RollingUpdate enables control over when each pod has the changes
applied.
• Only pods with an ordinal >= the partition value will be updated when the StatefulSet’s
.spec.template is updated.
• Pods with an ordinal < the partition will not be updated, even if they are deleted

21. Example Update Flow
Enable ES
Cluster
Allocation
Wait For
Pod Ready
State
Apply New
Pod
Definition
Increase
Partition
Size
Update
Received
Disable ES
Cluster
Allocation
Wait for ES
Cluster
Healthy
State

22. Remote Execution

23. The Challenge
Controlling the auth plugin that runs on the Elasticsearch containers
• User and role CRUD will need to be managed by the operator
• Apache Licensed Elasticsearch does not include an RBAC implementation
Apache 2.0 licensed Elasticsearch plugin that secures Elasticsearch by
providing authentication and authorization.
• Pre-built management utility
• Community Edition
• Internal user database for authentication
• Role based permissions for authorization
• Cluster and Index level permissions
• Live reloads of user database

24. How to Automate Updates?
Centralized Service k8s Job Remote Execution
Standalone service running,
listening on a queue and
executing its own copy of
sgadmin
The operator starts up a k8s
job which executes its own
copy of sgadmin in instance
namespace
The operator remotely
executes the copy of
sgadmin already existing on
the cluster

25. Winner: Remote Execution
Pros
● Security: Uses the k8s API server port; minimizes open ports on
the cluster
● Security: Leverages unique local admin certs we generate for each
cluster
● Simplicity: Uses the version of sgadmin installed on each cluster,
simplifying the support of multiple versions of the plugin
● Efficiency: No extra pods/run on demand means less wasted
resources
Cons
● Temporary multi-process container: Introduces another process
running in the Elasticsearch container breaking docker paradigms.
● Risk: client-go’s remotecommand library isn’t widely used

26. Re-queueing

27. Operator
The Challenge
Managing multiple, possibly conflicting requests
At the core of the Kubernetes design philosophy is an easy-to-understand idea: Bring the actual state
of an object in line with the specified state of an object. Just do that...forever…
Add
Modify
Delete
Cluster

28. Re-queueing
Some
Change Next

29. Re-queueing - Custom Rate-Limited
Slow Re-Queue
● Mounting PVC (EBS Volumes)
● Expanding volumes
● Removing nodes
● Cluster upgrades
Fast Re-Queue
● Rolling restarts
● Adding nodes
● Configuration updates

30. What’s Next?

31. Get free database resources on our
new Kubernetes-based platform on AWS in the region of your choice.
MongoDB | Elasticsearch | Redis
Your experience and feedback during the program will play a critical role in
helping us build the database platform you need.
https://objectrocket.com/alphaprogram
Next Generation Platform
Evaluation Program

32. Check Out Our Code
https://github.com/objectrocket

33. We’re Hiring!
• Developers
• Engineers
• Analysts
Apply at ObjectRocket.com/Careers
Meet ObjectRocket at the Hiring Mixer
4:30-6:30pm tonight