Almost all applications have some kind of state. Some data processing apps and databases have huge amounts of state. How do we navigate a cloud-based world of containers where stateless and functions-as-a-service is all the rage? As a long-time architect, designer, and developer of very stateful apps (databases and data processing apps), I’d like to take you on a journey through the modern cloud world and Kubernetes, offering helpful design patterns, considerations, tips, and where things are going. How is Kubernetes shaking up stateful app design?
3. @mathiasverraes
“There are only two hard problems in distributed
systems: 2. Exactly-once delivery 1. Guaranteed
order of messages 2. Exactly-once delivery”
4. What kind of state?
• Structured
• Semi-structured (logs, JSON, etc.)
• Graphs and networks
• Unstructured
• Config and passwords
• ML models and parameters
5. Characteristics of State
• Mutable vs Immutable
• Persistence (Temporary? Permanent? How permanent?)
• Availability
• Latency to retrieve and mutate
• Consistency
8. Container
Stateless - Where is my State
Requests/
Events
Memory
App
ReadOnly Disk Images
Temp local disk
Cloud Storage
— not persistent
9. “BUT WAIT… I thought stateless will solve all
my problems??”
10. Observations about Stateless
• A pattern that works for many scenarios
• All state pushed to other services - $$$
• Latency - stateless means every state change involves network
• Recovery - all local state must be recovered
• Many cloud data services are cloud specific (eg Dynamo, Kinesis) - multi-
cloud or moving clouds is huge amount of work
• Keeping state consistent across the cluster can be tricky
12. Container
Using Local State with Cloud Storage
Requests/
Events
Memory
App
ReadOnly Disk Images
Temp local disk
Cloud Storage (S3?)
— not persistent
Local
State
Local
State
13. Logs: Reasoning about State
“Stateless App”
Op1 Op2 Op3 Op4 Kubernetes
DB
DB2Event
Checkpoint
14. Logs: Reasoning about State
• A log of events and mutations are kept
• Checkpoints in the log represent snapshots of state
• Consistent state of system that can be recovered to
• Replaying the log allows predictable reconstruction of state and changes
• The foundation of all modern databases and data systems
19. Kubernetes Persistent Volumes
• Standard POSIX file semantics - just a mounted volume
• Yaml config of volume type, size, replication factor, desired speed, etc.
• Local Persistent Volumes - basically a HDD/SSD
• Networked, Replicated, not shared (single pod attachment only)
• AWS EBS, GCE PD, Azure Disk, Ceph, ScaleIO
• Can be very close to local disk in performance
• Replicated, Shared Network Storage (Multi pod attach)
• AWS EFS, CephFS, GlusterFS, NFS
21. Kubernetes StatefulSets: State Affinity
Pod 1
Memory
App
ReadOnly Disk Images
PV 1
Pod 2
Memory
App
ReadOnly Disk Images
PV 2
Pod 3
Memory
App
ReadOnly Disk Images
PV 3
22. Why Stateful Kubernetes?
• Run stateful services and databases yourself - to save $$
• You need local state persisted, or have a large amount of state
• Caching - lower latency
• ML models, iterative data transformations
• Want faster recovery for local state
• Need to work with local files (eg Lucene)
• Design for PVs, 1 abstraction - use on any cloud
23. Replicated DBs on Kubernetes
Leader Pod
Memory
PostGres L
ReadOnly Disk Images
PV 1
Follower Pod
Memory
PostGres F
ReadOnly Disk Images
PV 2
24. Kubernetes PVs vs S3
• Persistent, fast Local State
• S3/Remote Storage only for backups
• Kafka/persistent logging eliminated
or reduced
Pod 1
App
ReadOnly Disk Images
PV 1
Memory
Container
Kafka
App
ReadOnly Disk Images
Local DIsk
S3
Local
State
Memory
Local
State
PV
• Persistent logging (Kafka) and
cloud storage both essential
25. The Power of Replicated File Storage
• Don’t reinvent
distributed
coordination and
replication in every
data system/
database.
• Reuse a solid data
replication system.
27. Pod 1
Replicated Local State Using Kubernetes
App Shard 1
RocksDB Lucene
Replicated PV 1
SQLite
• Shard your app, each shard gets replicated storage
• Consistent snapshotting and state for diff parts of your app
Pod 2
App Shard 2
RocksDB Lucene
Replicated PV 2
SQLite
29. Shared K8s PV for Machine Learning
• Shared networked
Persistent Volume (FSx
for Lustre)
• Training job writes
files to FSx
• Kubernetes pods
serves models from
FSx
https://aws.amazon.com/blogs/storage/using-high-performance-persistent-storage-for-machine-learning-workloads-on-kubernetes/
30. PVs vs Cloud Data Services
Cloud Data Services Persistent Volumes
Replication and distribution handled by service/
database
Data in volume is replicated (if replicated PV used).
App needs to shard and handle coordination.
Each database/service has its own APIs Standard POSIX volume
Additional network latency of cloud services
Varies, but options for latency and performance
close to local drives
Each data service has its own consistency and failure-
handling characteristics
All data shared on the same PV has same
consistency & failure
31. Where State can Live
Type of State Cloud Service Local/Persistent Volume
Structured/SQL
MySQL, PostGres, RedShift,
etc. etc.
SQLite, H2, etc.
Key/Value Cassandra, Redis RocksDB, LMDB, MapDB, etc.
Semi-structured MongoDB, etc. etc.
Unstructured (binary, ML models, etc)
S3 Files on PV
Config K8s ConfigMap K8s ConfigMap
32. In Conclusion
• Super important:
• Where is your state
• What are its characteristics
• Think about state recovery and failure handling during design phase
• Replicated storage (PVs) is a very useful paradigm for data systems