Meet Spilo, Zalando’s HIGH-AVAILABLE POSTGRESQL CLUSTER - Feike Steenbergen
•
2 j'aime•2,107 vues
Spilo is a tool that provides high availability for PostgreSQL databases running on AWS. It uses Patroni and ETCD to handle replication, failover, and cluster state management. Teams at Zalando use Spilo to run over 150 PostgreSQL databases in a self-managed way on AWS, with each team responsible for their own databases. Spilo provides automation for deploying, replicating, and failing over PostgreSQL clusters on AWS, allowing for increased agility compared to managed database services.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (7)
Similaire à Meet Spilo, Zalando’s HIGH-AVAILABLE POSTGRESQL CLUSTER - Feike Steenbergen
Similaire à Meet Spilo, Zalando’s HIGH-AVAILABLE POSTGRESQL CLUSTER - Feike Steenbergen (20)
Plus de distributed matters
Plus de distributed matters (17)
Dernier
Dernier (20)
Meet Spilo, Zalando’s HIGH-AVAILABLE POSTGRESQL CLUSTER - Feike Steenbergen
- 1. Spilothe high-available PostgreSQL cluster Feike Steenbergen Zalando SE
- 2. • 15 EU countries • 3 fulfilment centers • 15+ million active customers • 2.2 billion € revenue 2014 Zalando
- 6. Our databases • > 150 production Postgresql databases • > 13.5 TB data • > 5 TB biggest DB • 400-1000+ write tps • > 2 DB failures/month
- 10. Purpose
- 11. Autonomy
- 12. Mastery
- 13. Cloud • 2013: ZCloud • 2014: project Pequod • 2015: Let’s just use AWS…
- 14. Amazon abbreviations • AWS - amazon web services • EC2 - elastic compute cloud • ELB - elastic load balancer • RDS - relational DB service • CF - Cloud Formation • ASG - Auto Scaling Group
- 15. AWS • One account per team • Microservices • REST/OAuth2 • Deployment with Docker
- 16. Autonomous teams on AWS REST INTERNET
- 17. Autonomous teams • Team decides which product to build • … and which technologies to use • REST/OAuth2 mandatory • Team is responsible for its infrastructure
- 18. • Developers should take care of infrastructure • ..including production databases • On AWS! Databases?
- 19. Isn’t it dangerous? DBAs running with scissors, by Gavin M. Roy: https://www.flickr.com/photos/gavinmroy/4638958958
- 22. • Detect the master failure • Elect a new master • Redirect clients Autofailover tasks
- 23. Autofailover issues • Discarded writes • Split-brain • False positives
- 24. RDS? • Support for PostgreSQL • Automatic failover • Most extensions • Automatic backups
- 25. RDS? • Vendor lock • No superuser • No untrusted languages • No logical decoding plugins • Costs
- 26. Spilo (სპილო)
- 27. Spilo does • Rapid deployment of PostgreSQL on AWS EC2 instances • Streaming replication with auto- failover
- 28. Spilo on AWS Spilo MASTER Spilo REPLICA Spilo REPLICA Master connection Application DB request ETCD cluster status update
- 29. Failover Spilo REPLICA Spilo REPLICA Master connection Application DB request ETCD cluster status update
- 30. Failover Spilo MASTER Spilo REPLICA Master connection Application DB request ETCD cluster status update NEW SPILO STARTS…
- 31. Failover Spilo MASTER Spilo REPLICA Master connection Application DB request ETCD cluster status update Spilo REPLICA
- 32. What is Spilo? c Patroni MASTER c Patroni REPLICA c Patroni REPLICA Auto-scaling group Auto-scaling group
- 33. Demo: Deploying Spilo • We use stups • First we define a template • We create a Cloud Formation Stack from this template
- 34. Patroni (პატრონი) • Handles new replicas and failover • Based on ideas and code of the Compose Governor • Open-source • Runs everywhere
- 35. Compose Governor idea ● Use etcd for failover decision ● Run etcd on every node ● Run 1 node with HAProxy + etcd
- 36. Distributed configuration systems • Fault tolerant • Reliably store small amounts of strongly-consistent data between distributed nodes • Good for storing the PostgreSQL cluster state
- 37. Distributed consensus LEADER CLIENT CLIENT CLIENT write request
- 38. Distributed consensus LEADER CLIENT CLIENT CLIENT write request LEADER
- 39. Cluster state in etcd $ export ETCD=172.17.0.2:4001 $ etcdctl -C $ETCD ls /service/ --recursive /service/dm /service/dm/optime /service/dm/optime/leader /service/dm/members /service/dm/members/postgresql_172_17_0_3 /service/dm/members/postgresql_172_17_0_4 /service/dm/members/postgresql_172_17_0_5 /service/dm/initialize /service/dm/leader
- 40. Leader key • Points to the member key • Has a TTL, autoexpires • Acts as an exclusive lock • Only the leader can become the master
- 41. Leader TTL $ http http://$ETCD/v2/keys/service/dm/leader ... { "action": "get", "node": { "createdIndex": 8, "expiration": "2015-11-20T09:56:43.59367 "key": "/service/dm/leader", "modifiedIndex": 85, "ttl": 22, "value": "postgresql_172_17_0_3" } }
- 42. Member key $ etcdctl -C $ETCD get /service/dm/members/postgresql_172_17_0_5 { "conn_url": "postgres://un:pw@172.17.0.5:5432/po "api_url": "http://172.17.0.5:8008/patroni", "tags": {}, "state": "running", "role": "replica", "xlog_location": 67109176 }
- 43. Patroni Patroni MASTER REPLICA MASTER LB PostgreSQL connection API HealthCheck (master) Connection and API URL
- 44. Patroni Patroni MASTER REPLICA MASTER LB PostgreSQL connection API HealthCheck (master) Connection and API URL
- 45. Patroni Patroni MASTER REPLICA MASTER LB PostgreSQL connection API HealthCheck (master) Connection and API URL
- 46. Patroni Patroni MASTER REPLICA MASTER LB PostgreSQL connection API HealthCheck (master) Connection and API URL
- 47. Connection and API URL Patroni Patroni MASTER REPLICA MASTER LB REPLICA LB API HealthCheck (slave) PostgreSQL connection API HealthCheck (master)
- 48. Initialize key $ etcdctl -C $ETCD get /service/dm/initialize 6219169399948550171 • PostgreSQL cluster system ID • Created by the first node that joins the cluster • Nodes with different system ID are not allowed to join
- 49. Patroni modules ETCD ZOOKEEPER ABSTRACT DCS PostgreSQL REST API High availability Asynchronous executor Callbacks
- 51. • Robust exception handling • Run long-running tasks (i.e. base backup in a separate thread) • ETCD + Zookeeper • Rest API Patroni improvements
- 52. Patroni improvements • Configurable replica imaging • Support for pg_rewind • patronictl • packaged: pip install patroni
- 53. Patroni improvements • Manual failover • Initialize from external cluster • Attach to already running PostgreSQL nodes • Tags (i.e. nofailover)
- 54. • Spilo: github.com/zalando/spilo spilo.readthedocs.org • Patroni: github.com/zalando/patroni patroni.readthedocs.org • Stups: github.com/zalando-stups/ stups.io • Feedback: @ekief Thank you!