The document discusses PostgreSQL high availability and scaling options. It covers horizontal scaling using load balancing and data partitioning across multiple servers. It also covers high availability techniques like master-slave replication, warm standby servers with point-in-time recovery, and using a heartbeat to prevent multiple servers from becoming a master. The document recommends an initial architecture with two servers using warm standby and point-in-time recovery with a heartbeat for high availability. It suggests scaling the application servers horizontally later on if more capacity is needed.
1. PostgreSQL
High Availability & Scaling
John Paulett
October 26, 2009
2. Overview
Scaling Overview
– Horizontal & Vertical Options
High Availability Overview
Other Options
Suggested Architecture
Hardware Discussion
10/26/2009 2
3. What are we trying to solve?
Survive server failure?
– Support an uptime SLA (e.g. 99.9999%)?
Application scaling?
– Support additional application demand
10/26/2009 3
4. What are we trying to solve?
Survive server failure?
– Support an uptime SLA (e.g. 99.9999%)?
Application scaling?
– Support additional application demand
→ Many options, each optimized for
different constraints
10/26/2009 4
6. How To Scale
Horizontal Scaling
– “Google” approach
– Distribute load across multiple servers
– Requires appropriate application architecture
Vertical Scaling
– “Big Iron” approach
– Single, massive machine (lots of fast processors,
RAM, & hard drives)
10/26/2009 6
7. Horizontal DB Scaling
Load Balancing
– Distribute operations to multiple servers
Partitioning
– Cut up the data (horizontal) or tables (vertical)
and put them on separate servers
– aka “sharding”
10/26/2009 7
8. Basic Problem when Load
Balancing
Difficult to maintain consistent state
between servers (remember ACID),
especially when dealing with writes
4 PostgreSQL Load Balancing Methods:
– Master-Slave Replication
– Statement-Based Replication Middleware
– Asynchronous Multimaster Replication
– Synchronous Multimaster Replication
10/26/2009 8
9. Master-Slave Replication
Master handles writes, slaves handle reads
Asynchronous replication
– Possible data loss on master failure
Slony-I
– Does not automatically propagate schema changes
– Does not offer single connection point
– Requires separate solution for master failures
10/26/2009 9
10. Statement-Based Replication
Middleware
Intercept SQL queries, send writes to all
servers, reads to any server
Possible issues using random(),
CURRENT_TIMESTAMP, & sequences
pgpool-II
– Connection Pooling, Replication, Load Balancing,
Parallel Queries, Failover
10/26/2009 10
12. Synchronous Multimaster
Replication
Writes & reads on any server
Not implemented in PostgreSQL, but
application code can mimic via two-phase
commit
10/26/2009 12
13. Load Balancing Issue
Scaling writes breaks down at a certain
point
10/26/2009 13
14. Partitioning
Requires heavy application modification
Performing queries across partitions is
problematic (not possible)
PL/Proxy can help
10/26/2009 14
15. Vertical DB Scaling
“Buying a bigger box is quick(ish). Redesigning
software is not.”
● Cal Henderson, Flickr
37 Signals Basecamp upgraded to 128 GB DB
server: “don’t need to pay the complexity tax
yet”
● David Heinemeier Hansson, Ruby on Rails
10/26/2009 15
16. Sites Running on Single DB
StackOverflow
– MS SQL, 48GB RAM, RAID 1 OS, RAID 10 for data
37Signals Basecamp
– MySQL, 128GB RAM. Dell R710 or Dell 2950
10/26/2009 16
18. High Availability
Application still up even after node failure
– (Also try to prevent failure with appropriate
hardware)
PostgreSQL High Availability Options
– pg-pool
– Shared Disk Failover
– File System Replication
– Warm Standby with Point-In-Time Recovery (PITR)
Often still need heartbeat application
10/26/2009 18
19. Shared Disk Failover
Use single disk array to hold database's
data files.
– Network Attached Storage (NAS)
– Network File System (NFS)
Disk array is central point of failure
Need heartbeat to bring 2nd server online
10/26/2009 19
20. File System Replication
File system is mirrored to another
computer
DRDB
– Linux filesystem replication
Need heartbeat to bring 2nd server online
10/26/2009 20
21. Point in Time Recovery
“Log shipping”
– Write Ahead Logs sent to and replayed on standby
– Included in PostgreSQL 8.0+
– Asynchronous - Potential loss of data
Warm Standby
– Standbys' hardware very similar to primary's
– Need heartbeat to bring 2nd server online
10/26/2009 21
22. Heartbeat
“STONITH” (Shoot the Other Node In The
Head)
– Prevent multiple nodes thinking they are the
master
Linux-HA
– Creates cluster, takes nodes out when they fail
10/26/2009 22
28. Suggested Architecture
2 nice machines
Point in Time Recovery with Heartbeat
Tune PostgreSQL
Monitor & improve slow queries
Add in Ehcache as we touch code
→ Leave horizontal scaling for another day
10/26/2009 28
30. Future Architecture
Scale up application servers horizontally
as needed
Improve DB Hardware
10/26/2009 30
31. Hardware Options
PostgreSQL typically constrained by RAM
& Disk IO, not processor
64-bit, as much memory as possible
Data Array
– RAID10 with 4 drives (not RAID 5), 15k RPM
Separate OS Drive / Array
10/26/2009 31
32. Dell R710
Processor: Xeon
4x 15k HD in RAID10
24GB (3x 8GB) RAM (up to 6x 16GB)
=$6,905
10/26/2009 32
33. Other Considerations
Should have Test environment mimic
Production
– Same database setup
– Provides environment for experimentation
Can host multiple DBs on single cluster
10/26/2009 33