1. Advanced WAL File Management
With OmniPITR
Robert Treat, JDCon 2011 #pgeast
/ Presentation
Thursday, March 31, 2011
2. Who Am I?
• Robert Treat
• OmniTI
• Database Management and Consulting
• We’re Hiring!
• Postgres
• Major Contributor, Web, Advocacy, Random
• xzilla.net
• @robtreat2
Thursday, March 31, 2011
3. Who Am I?
• Robert Treat
• OmniTI
• Database Management and Consulting
• We’re Hiring!
• Postgres
• Major Contributor, Web, Advocacy, Random
• xzilla.net
• @robtreat2
Thursday, March 31, 2011
9. What is PITR?
•Postgres emits WAL files
Thursday, March 31, 2011
10. What is PITR?
•Postgres emits WAL files
•Send the WAL file to another server
Thursday, March 31, 2011
11. What is PITR?
•Postgres emits WAL files
•Send the WAL file to another server
•The other server can replay the WAL
Thursday, March 31, 2011
12. What is PITR?
Use combination of
data files and WAL
to make more postgrezes
Thursday, March 31, 2011
13. What is PITR?
•Postgres emits WAL files
Use combination of
data files and WAL
to make more postgrezes
Thursday, March 31, 2011
14. What is PITR?
•Postgres emits WAL files
•Send the WAL file to another server
Use combination of
data files and WAL
to make more postgrezes
Thursday, March 31, 2011
15. What is PITR?
•Postgres emits WAL files
•Send the WAL file to another server
•The other server can replay the WAL
Use combination of
data files and WAL
to make more postgrezes
Thursday, March 31, 2011
48. OmniPITR-Archive Example
archive_command =
'/opt/OMNIpitr/bin/omnipitr-archive
-l "/var/log/omnipitr/omnipitr-^Y-^m-^d.log"
-s /var/lib/pgsql/omnipitr/state/
-t /var/tmp/
-dr gzip=db4:/mnt/db/prod/walarchive/
-dr db4:/mnt/db/prod/db4-walarchive/
-db /var/lib/pgsql/omnipitr/backup.xlogs
"%p"'
state-directory to handle errors when sending wal to
multiple destinations
Thursday, March 31, 2011
49. OmniPITR-Archive Example
archive_command =
'/opt/OMNIpitr/bin/omnipitr-archive
-l "/var/log/omnipitr/omnipitr-^Y-^m-^d.log"
-s /var/lib/pgsql/omnipitr/state/
-t /var/tmp/
-dr gzip=db4:/mnt/db/prod/walarchive/
-dr db4:/mnt/db/prod/db4-walarchive/
-db /var/lib/pgsql/omnipitr/backup.xlogs
"%p"'
Where To Create Temp Files
Thursday, March 31, 2011
50. OmniPITR-Archive Example
archive_command =
'/opt/OMNIpitr/bin/omnipitr-archive
-l "/var/log/omnipitr/omnipitr-^Y-^m-^d.log"
-s /var/lib/pgsql/omnipitr/state/
-t /var/tmp/
-dr gzip=dbx:/mnt/db/prod/walstorage/
-dr db4:/mnt/db/prod/db4-walarchive/
-db /var/lib/pgsql/omnipitr/backup.xlogs
"%p"'
send to remote server (dbx) compressed (gzip)
Thursday, March 31, 2011
51. OmniPITR-Archive Example
archive_command =
'/opt/OMNIpitr/bin/omnipitr-archive
-l "/var/log/omnipitr/omnipitr-^Y-^m-^d.log"
-s /var/lib/pgsql/omnipitr/state/
-t /var/tmp/
-dr gzip=db4:/mnt/db/prod/walarchive/
-dr db4:/mnt/db/prod/db4-walarchive/
-db /var/lib/pgsql/omnipitr/backup.xlogs
"%p"'
send to remote server (db4) uncompressed
Thursday, March 31, 2011
52. OmniPITR-Archive Example
archive_command =
'/opt/OMNIpitr/bin/omnipitr-archive
-l "/var/log/omnipitr/omnipitr-^Y-^m-^d.log"
-s /var/lib/pgsql/omnipitr/state/
-t /var/tmp/
-dr gzip=db4:/mnt/db/prod/walarchive/
-dr db4:/mnt/db/prod/db4-walarchive/
-db /var/lib/pgsql/omnipitr/backup.xlogs
"%p"'
Where to store xlogs when building a backup
Thursday, March 31, 2011
53. OmniPITR-Archive Example
archive_command =
'/opt/OMNIpitr/bin/omnipitr-archive
-l "/var/log/omnipitr/omnipitr-^Y-^m-^d.log"
-s /var/lib/pgsql/omnipitr/state/
-t /var/tmp/
-dr gzip=db4:/mnt/db/prod/walarchive/
-dr db4:/mnt/db/prod/db4-walarchive/
-db /var/lib/pgsql/omnipitr/backup.xlogs
"%p"'
The xlog file :-)
Thursday, March 31, 2011
55. OmniPITR-Restore
•Lots of options (though not as many)
Thursday, March 31, 2011
56. OmniPITR-Restore
•Lots of options (though not as many)
•compressed files?
Thursday, March 31, 2011
57. OmniPITR-Restore
•Lots of options (though not as many)
•compressed files?
•custom log file
Thursday, March 31, 2011
58. OmniPITR-Restore
•Lots of options (though not as many)
•compressed files?
•custom log file
•special paths
Thursday, March 31, 2011
59. OmniPITR-Restore Example
restore_command = '
/opt/OMNIpitr/bin/omnipitr-restore
-l /var/lib/pgsql/data/pg_log/omnipitr-^Y-^m-^d.log
-s /var/lib/pgsql/wal_archive/
-p /var/lib/pgsql/wal_archives.pause
-v
-r
%f %p'
location of omnipitr-restore program
Thursday, March 31, 2011
60. OmniPITR-Restore Example
restore_command = '
/opt/OMNIpitr/bin/omnipitr-restore
-l /var/lib/pgsql/data/pg_log/omnipitr-^Y-^m-^d.log
-s /var/lib/pgsql/wal_archive/
-p /var/lib/pgsql/wal_archives.pause
-v
-r
%f %p'
custom log file format
Thursday, March 31, 2011
61. OmniPITR-Restore Example
restore_command = '
/opt/OMNIpitr/bin/omnipitr-restore
-l /var/lib/pgsql/data/pg_log/omnipitr-^Y-^m-^d.log
-s /var/lib/pgsql/wal_archive/
-p /var/lib/pgsql/wal_archives.pause
-v
-r
%f %p'
Source of WAL Files to Use For Restore
Thursday, March 31, 2011
62. OmniPITR-Restore Example
restore_command = '
/opt/OMNIpitr/bin/omnipitr-restore
-l /var/lib/pgsql/data/pg_log/omnipitr-^Y-^m-^d.log
-s /var/lib/pgsql/wal_archive/
-p /var/lib/pgsql/wal_archives.pause
-v
-r
%f %p'
pause xlog removal if this file exists
(foreshadow: used for making backups)
Thursday, March 31, 2011
64. OmniPITR-Restore Example
restore_command = '
/opt/OMNIpitr/bin/omnipitr-restore
-l /var/lib/pgsql/data/pg_log/omnipitr-^Y-^m-^d.log
-s /var/lib/pgsql/wal_archive/
-p /var/lib/pgsql/wal_archives.pause
-v
-r
%f %p'
Remove Xlogs Files Which Are No Longer Needed
Thursday, March 31, 2011
65. OmniPITR-Restore Example
restore_command = '
/opt/OMNIpitr/bin/omnipitr-restore
-l /var/lib/pgsql/data/pg_log/omnipitr-^Y-^m-^d.log
-s /var/lib/pgsql/wal_archive/
-p /var/lib/pgsql/wal_archives.pause
-v
-r
%f %p'
standard macros for “basename of segment” and
“destination of recovery”
Thursday, March 31, 2011
82. OmniPITR-Slave-Backup
•On MySQL, you could dump on slave
for a long time
Thursday, March 31, 2011
83. OmniPITR-Slave-Backup
•On MySQL, you could dump on slave
for a long time
•On ZFS, we use snapshots to make
slaves on the backups
Thursday, March 31, 2011
84. OmniPITR-Slave-Backup
•On MySQL, you could dump on slave
for a long time
•On ZFS, we use snapshots to make
slaves on the backups
•While “we” prefer to use Postgres,
some of us prefer to use Linux
Thursday, March 31, 2011
85. OmniPITR-Slave-Backup
•On MySQL, you could dump on slave
for a long time
•On ZFS, we use snapshots to make
slaves on the backups
•While “we” prefer to use Postgres,
some of us prefer to use Linux
•But, you still don’t want to pay the
price for backups on master
Thursday, March 31, 2011
86. OmniPITR-Slave-Backup Example
/opt/OMNIpitr/bin/omnipitr-backup-slave
-v
-l "/var/log/omnipitr/omnipitr-slave-backup-^Y-^m-^d.log"
-s /mnt/db/prod/db4-walarchive
-p /var/lib/pgsql/omnipitr/pause.removal
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz
-dl gzip=/mnt/db/prod/backups/
-pp /usr/pgsql-9.0/bin/pg_controldata
Location of omni-pitr-slave command
Thursday, March 31, 2011
88. OmniPITR-Slave-Backup Example
/opt/OMNIpitr/bin/omnipitr-backup-slave
-v
-l "/var/log/omnipitr/omnipitr-slave-backup-^Y-^m-^d.log"
-s /mnt/db/prod/db4-walarchive
-p /var/lib/pgsql/omnipitr/pause.removal
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz
-dl gzip=/mnt/db/prod/backups/
-pp /usr/pgsql-9.0/bin/pg_controldata
custom log location / format
Thursday, March 31, 2011
89. OmniPITR-Slave-Backup Example
/opt/OMNIpitr/bin/omnipitr-backup-slave
-v
-l "/var/log/omnipitr/omnipitr-slave-backup-^Y-^m-^d.log"
-s /mnt/db/prod/db4-walarchive
-p /var/lib/pgsql/omnipitr/pause.removal
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz
-dl gzip=/mnt/db/prod/backups/
-pp /usr/pgsql-9.0/bin/pg_controldata
source of wal files
Thursday, March 31, 2011
90. OmniPITR-Slave-Backup Example
/opt/OMNIpitr/bin/omnipitr-backup-slave
-v
-l "/var/log/omnipitr/omnipitr-slave-backup-^Y-^m-^d.log"
-s /mnt/db/prod/db4-walarchive
-p /var/lib/pgsql/omnipitr/pause.removal
-D /pg_data/90data/
-t /var/tmp/
-gp /usr/bin/pigz
-dl gzip=/mnt/db/prod/backups/
-pp /usr/pgsql-9.0/bin/pg_controldata
file to pause removal of xlogs no longer needed for
restore but still needed by backup
Thursday, March 31, 2011
97. Production Use?
•Versions 8.2, 8.3, 8.4, 9.0
Thursday, March 31, 2011
98. Production Use?
•Versions 8.2, 8.3, 8.4, 9.0
•Linux, Solaris
Thursday, March 31, 2011
99. Production Use?
•Versions 8.2, 8.3, 8.4, 9.0
•Linux, Solaris
•TB+ Databases, with multiple
tablespaces
Thursday, March 31, 2011
100. Production Use?
•Versions 8.2, 8.3, 8.4, 9.0
•Linux, Solaris
•TB+ Databases, with multiple
tablespaces
•Thousands of txn/sec
Thursday, March 31, 2011
101. Production Use?
•Versions 8.2, 8.3, 8.4, 9.0
•Linux, Solaris
•TB+ Databases, with multiple
tablespaces
•Thousands of txn/sec
•Cross Datacenter
Thursday, March 31, 2011
102. Where’s The Code?
https://labs.omniti.com/labs/pgtreats
svn co https://labs.omniti.com/pgtreats/trunk/omnipitr/
Thursday, March 31, 2011
103. Where’s The Code?
•Currently in “PGTreats” Repo
https://labs.omniti.com/labs/pgtreats
svn co https://labs.omniti.com/pgtreats/trunk/omnipitr/
Thursday, March 31, 2011
104. Where’s The Code?
•Currently in “PGTreats” Repo
•Available Via SVN Pull
https://labs.omniti.com/labs/pgtreats
svn co https://labs.omniti.com/pgtreats/trunk/omnipitr/
Thursday, March 31, 2011
105. Where’s The Code?
•Currently in “PGTreats” Repo
•Available Via SVN Pull
•“BSD” Licensed
https://labs.omniti.com/labs/pgtreats
svn co https://labs.omniti.com/pgtreats/trunk/omnipitr/
Thursday, March 31, 2011
107. TODO
•Better Monitoring
Thursday, March 31, 2011
108. TODO
•Better Monitoring
•Parallel Multi-Destination
Thursday, March 31, 2011
109. TODO
•Better Monitoring
•Parallel Multi-Destination
•Multiple-Source Restores
Thursday, March 31, 2011
110. BUGS
Failover of a streaming replication
based slave created from a slave
based backup file against a stream
replication based slave doesn’t
work
Thursday, March 31, 2011
111. BUGS
Failover of a streaming replication
based slave created from a slave
based backup file against a stream
replication based slave doesn’t
work, sometimes,
Thursday, March 31, 2011
112. BUGS
Failover of a streaming replication
based slave created from a slave
based backup file against a stream
replication based slave doesn’t
work, sometimes, when under load
Thursday, March 31, 2011
113. BUGS
Failover of a streaming replication
based slave created from a slave
based backup file against a stream
replication based slave doesn’t
work, sometimes, when under load
We think it’s a flaw in Postgres,
but hard to say for sure
Thursday, March 31, 2011
114. THE END
Thanks!
pgeast, pgus
Slides
http://www.xzilla.net/
Thursday, March 31, 2011