This document provides an overview of Bucardo, an open source tool for replicating and synchronizing PostgreSQL databases. Bucardo uses triggers and asynchronous notifications to replicate data changes between a master and slave database. It allows custom filtering and processing of replication events. The document discusses Bucardo's architecture, installation, configuration, administration, and limitations.

1. Bucardo:
Replication with
tiny little goats
Selena Deckelmann
selena@endpoint.com
en
dp
oi
nt
.c
om
http://www.flickr.com/photos/kevincollins/

2. Hi!
★ Software Engineer
http://endpoint.com
★ User Group Organizer
http://pugs.postgresql.org/pdxpug
★ Conference Organizer
http://opensourcebridge.org
en
dp
oi
nt
.c
om

3. SEAPUG?
• subscribe:
email to: majordomo@postgresql.org
message body: subscribe seapug
• Talk to me after!
en
dp
oi
nt
.c
om

4. en
dp
http://www.telegraph.co.uk/science/science-news/4409958/Extinct-ibex-is-
oi
nt
.c
resurrected-by-cloning.html
om

5. What can it do?
en
dp
oi
nt
.c
http://www.flickr.com/photos/8602783@N06/2294028540
om

6. Master-slave
• Scaling/Redundancy
• UPGRADES (8.x -> 8.4)
• Reporting databases
• Data warehousing
• Now handles SEQUENCES!
en
dp
oi
nt
.c
om

7. Master-master
• Write two two databases!
• Sync type: SWAP
• Conflict resolution through hooks!
en
dp
oi
nt
.c
om

8. Custom code
• Respond to DML changes
• Filter
• Run arbitrary code!
en
dp
oi
nt
.c
om

9. STORY TIME!
item
ski
item
ski
item
ski
item
item
ski
item
ski
item onhand
ski
item
skis 10
ski
item
ski
item
ski
item
ski
on hand = count(item)
en
dp
oi
nt
.c
om

10. prod db
reporting db
new!
item NOTIFY bucardo
ski
sync B new!
item
syn ski
cC
on hand
cA
Y
TIF +1
syn
NO
custom
code
count(items)
and other stuff
en
dp
oi
nt
.c
om
goat courtesy of: http://www.flickr.com/photos/nuritwilde/394603312

11. Strengths
• Drop-in, no changes to Postgres
• Control daemon/DB can run anywhere
• EXTREME async
• No locking of slaves
• Command-line controls rock!
en
dp
oi
nt
.c
om

12. The Fine Print
• No smooth DDL handling
• Not for failover (aka high availability)
• bloat: pg_listener, q, bucardo_delta,
bucardo_track
• No locking of slaves
• No Windows support
• One developer
en
dp
oi
nt
.c
om
http://www.flickr.com/photos/ncarey/135901120/

13. http://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling
en
dp
oi
nt
.c
om

14. Installing
en
dp
oi
nt
.c
om
http://www.flickr.com/photos/kendrick/361173694

15. Bucardo Requirements
• Postgres (8.3 or higher recommended)
• Perl & some packages:
• DBD::Pg 2.0
• DBI 1.51
• DBIx::Safe 1.2.4
• ExtUtils::MakeMaker 6.32
• pl/perlu and pl/pgsql
en
dp
oi
nt
.c
om

16. Simple set up
MASTER setup:
bucardo_ctl add db MyDB name=master
bucardo_ctl add all tables
bucardo_ctl add all sequences
SLAVE setup:
bucardo_ctl add db MyDB name=slave port=6543
bucardo_ctl add herd all_tables
psql# INSERT INTO herdmap (herd, goat)
SELECT ‘all_tables’, id FROM goat
WHERE (reltype = ‘table’
OR reltype = ‘sequence’)
en
AND db = ‘master’;
dp
oi
nt
.c
om

17. Setting up, cont.
TEST:
bucardo_ctl validate all
SYNC:
bucardo_ctl add sync delta
type=pushdelta source=all_tables
targetdb=slave
(installs triggers!)
en
dp
oi
nt
.c
om

18. Setting up
• Initiate a one-time-copy: onetimecopy=[1|2]
• Set up ongoing syncs: DEFAULT (can set
timeouts)
• MANY configuration options
en
dp
oi
nt
.c
om

19. Admin stuff
• .bucardorc (set things like: dbport)
• Only need one ‘bucardo’ database
• Doesn’t need to be your prod databases
• System uses the ‘bucardo’ schema for ease
of admin, removal
en
dp
oi
nt
.c
om

20. Smooth upgrades!
bucardo_ctl upgrade
• Stop bucardo (bucardo_ctl stop)
• applies DDL changes, functions
• see also UPGRADE file
en
dp
oi
nt
.c
om

21. Stuff to watch out for
• search_path for ‘bucardo’ user
• PL/Perlu & pl/pgsql must be installed
• Uses prepared statements, turn off with:
bucardo_ctl update db MyDB
server_side_prepares=0
en
dp
oi
nt
.c
om

22. Internals
en
dp
oi
nt
http://www.flickr.com/photos/chelseagirlphotos
.c
om

23. Internals
bucardo=# d
List of relations
Schema | Name | Type | Owner
---------+-------------------------------+----------+---------
bucardo | audit_pid | table | bucardo
bucardo | audit_pid_id_seq | sequence | bucardo
bucardo | bucardo_config | table | bucardo
bucardo | bucardo_custom_trigger | table | bucardo
bucardo | bucardo_custom_trigger_id_seq | sequence | bucardo
bucardo | bucardo_log_message | table | bucardo
bucardo | bucardo_rate | table | bucardo
bucardo | customcode | table | bucardo
bucardo | customcode_id_seq | sequence | bucardo
bucardo | customcode_map | table | bucardo
bucardo | db | table | bucardo
bucardo | db_connlog | table | bucardo
bucardo | dbgroup | table | bucardo
bucardo | dbmap | table | bucardo
bucardo | goat | table | bucardo
bucardo | goat_id_seq | sequence | bucardo
bucardo | herd | table | bucardo
bucardo | herdmap | table | bucardo
bucardo | q | table | bucardo
bucardo | sync | table | bucardo
bucardo | upgrade_log | table | bucardo
en
(21 rows)
dp
oi
nt
.c
om

24. goats
Table "bucardo.goat"
Column | Type |
----------------------+--------------------------+
id | integer |
db | text |
schemaname | text |
tablename | text |
reltype | text |
pkey | text |
qpkey | text |
pkeytype | text |
has_delta | boolean |
ping | boolean |
...
en
dp
oi
nt
.c
om

25. goats
# bucardo_ctl list tables
Table: archive.course_logging DB: odw_master PK: none
Table: archive.messages DB: odw_master PK: none
Table: archive.student_assignments DB: odw_master PK: none
Table: archive.student_courses DB: odw_master PK: none
Table: logging.activity DB: odw_master PK: activity_name (varchar)
Table: logging.course_logging DB: odw_master PK: course_logging_id (int4)
Table: public.ecash_transaction DB: odw_master PK: trans_id (int4)
Table: public.feedback DB: odw_master PK: date (timestamp)
Table: public.holidays DB: odw_master PK: customernum|month|day|
year (int4|int4|int4|int4)
...
en
dp
oi
nt
.c
om

26. bucardo_ctl list sequences
Sequence 63: logging.logging_seq
Sequence 64: public.trans_id_seq
Sequence 65: public.num_seq
Sequence 66: public.upload_seq
en
dp
oi
nt
.c
om

27. bucardo database
bucardo=# d
List of relations
Schema | Name | Type | Owner
---------+-------------------------------+----------+---------
...
bucardo | db | table | bucardo
bucardo | db_connlog | table | bucardo
bucardo | dbgroup | table | bucardo
bucardo | dbmap | table | bucardo
bucardo | goat | table | bucardo
bucardo | goat_id_seq | sequence | bucardo
bucardo | herd | table | bucardo
bucardo | herdmap | table | bucardo
bucardo | q | table | bucardo
bucardo | sync | table | bucardo
bucardo | upgrade_log | table | bucardo
(21 rows)
en
dp
oi
nt
.c
om

28. Internals
# bucardo_ctl status
Days back: 3 User: bucardo Database: bucardo Port: 8000
PID of Bucardo MCP: 6876
Name Type State PID Last_good Time I/U/D Last_bad Time
========+=====+=====+====+=========+=====+=====+========+====
a_delta | P |idle |6886|1m4s |0s |0/0/0|unknown |
b_delta | P |idle |6885|1m4s |0s |0/0/0|9h32m47s|0s
c_delta | P |idle |6887|1m4s |0s |0/0/0|unknown |
d_delta | P |idle |6888|54s |0s |0/0/0|2m6s |3s
e_delta | P |idle |6890|1m4s |0s |0/0/0|9h32m47s|0s
f_delta | P |idle |6889|1m4s |0s |0/0/0|9h32m47s|0s
en
dp
oi
nt
.c
om

29. # bucardo_ctl status a_delta
Days back: 3 User: bucardo Database: bucardo Port: 8000
======================================================================
Sync name: a_delta
Current state: idle (PID = 6888)
Type: pushdelta
Source herd/database: all_a / a_master
Target database: a_slave
Tables in sync: 26
Last good: 3m 34s (time to run: 0s)
Last good time: Oct 16, 2009 17:52:20 Target: a_slave
Ins/Upd/Del: 20 / 0 / 50
Last bad: 4m 47s (time to run: 3s)
Last bad time: Oct 16, 2009 17:51:08 Target: a_slave
Latest bad reason: ?
PID file: /var/run/bucardo/bucardo.ctl.sync.a_delta.pid
PID file created: Fri Oct 16 17:52:09 2009
Status: active
Limitdbs: 0
Priority: 0
Checktime: none
Overdue time: 00:00:00
Expired time: 00:00:00
Stayalive: yes Kidsalive: yes
Rebuild index: 0 Do_listen: yes
Ping: yes Makedelta: no
en
dp
Onetimecopy: 0
oi
nt
.c
om

30. How Bucardo replicates
en
dp
oi
nt
.c
om

31. Triggers
INSERT, UPDATE, DELETE
bucardo_add_delta
IN
SE
RT
bucardo_delta
bucardo_triggerkick_[sync]
NOTIFY!
en
dp
oi
nt
.c om

32. NOTIFY / LISTEN
• Simple interprocess communication
psql# LISTEN cupcakes;
psql# NOTIFY cupcakes;
Asynchronous notification "cupcakes"
received from server process with PID
1337.
• Any Postgres process can listen in
en
dp
oi
nt
.c
om

33. Actual Triggers!
Table "public.test"
Column | Type | Modifiers
--------+-----------------------+-----------
test | character varying(15) | not null
Indexes:
"test_pkey" PRIMARY KEY, btree ("test")
Triggers:
bucardo_add_delta AFTER INSERT OR DELETE OR
UPDATE ON test FOR EACH ROW EXECUTE PROCEDURE
bucardo_add_delta_role()
bucardo_triggerkick_a_delta AFTER INSERT OR
DELETE OR UPDATE ON test FOR EACH STATEMENT
EXECUTE PROCEDURE bucardo_triggerkick_a_delta()
en
dp
oi
nt
.c
om

34. On your master DB
Tables:
• bucardo_delta table has: ID, txn_time
• bucardo_track table: who’s done what?
en
dp
oi
nt
.c
om

35. Processes!
• MCP, CTL, KID
• MCP: master control process
• CTL: controller (track, kick off/kill KIDs)
• KID: sync processes
en
dp
oi
nt
.c
om

36. syncs
• A set of tables (herd) that are replicated
together
en
dp
oi
nt
.c
om

37. Initiating syncs
• NOTIFY vs. timeout vs. kick
• NOTIFY - from master db to MCP
• timeout - controller detects
• kick - manual/from MCP to controller
en
dp
oi
nt
.c
om

38. Bucardo
NOTIFY NOTIFY
NOTIFY MCP CTL KID
TRIGGER
INSERT INTO “q”
SELECT from “q”
UPDATE “q”
Flow of control REPLICATION
en
EVENT!
dp
oi
nt
.c
om

39. Administrativa
• Totally ok to kill KIDs
• CTL cleans up after you
• May need to kill open database handles (for
unfinished COPYs, etc)
en
dp
oi
nt
.c
om

40. Sync types
• Fullcopy. Timeout/kick. TRUNCATE or
DELETE, then COPY
• Pushdelta. Trigger. One way master-slave.
• Swap. Trigger. Two-way. Conflict resolution.
en
dp
oi
nt
.c
om

41. Bucardo anti-trigger
• canonical way to disable triggers:
ALTER TABLE DISABLE TRIGGER ALL;
:(
• pre 8.3 way: UPDATE pg_class..
• SET session_replication_role = ‘replica’;
• Applies to just the current session
• Thanks, Jan!
en
dp
oi
nt
.c
om

42. Bucardo hooks
• LIVE DATA CHANGES!
• pass in a hashref (ROW or ROWINFO)
• Returned value - bitmapped (a/b or both!)
• Affect the replication event, side effects
• Example: cache invalidation
en
dp
oi
nt
.c
om

43. Bucardo Hooks
• customcode & customcode_map
• Conflict handlers (SWAP syncs)
• Exception - fix and try again
• Pre and post trigger
en
dp
oi
nt
.c
om

44. Development
• bucardo.org
• mailing lists: http://bucardo.org/wiki/
Bucardo_mailing_list
• Release early, often.
• Please submit patches! bug reports! tests!
en
dp
oi
nt
.c
om

45. Questions?
.com
nt
oi
dp
en

46. Thanks!
http://www.slideshare.net/selenamarie/bucardo
selena@endpoint.com
@selenamarie on Twitter
rss: http://www.chesnok.com/daily
en
dp
oi
nt
.c
om