1. guarding your data
Andrew Gilmartin
senior developer
agilmartin@crossref.org

2. don’t loose the data
• Oracle holds the data in Boston
Oracle × 2 + tape + off-site tape
• Oracle holds the data in Denver
Oracle × 2
• Denver less than 20 minutes out of sync with Boston

3. syncing Boston & Denver
• Oracle's
Oracle Streams …
• Other's
Tungsten Replicator …
• Our's

4. the tool’s context
• use existing tooling
• use a small set of dependent parts
• use already monitored parts
• Oracle
• file systems
• networks
• …

5. the tool’s necessities
• reliable
• recoverable
• chatty
• alarming

6. Oracle’s archive-log
1 archive-log represent the changes to the database
since the last archive-log
2 discover created archive-log via Boston Oracle’s
tracing-log
3 transfer archive-log to Denver
4 register archive-log with Denver’s Oracle,
5 discover archive-log applied via Denver Oracle’s
tracing-log

7. a bash script
• script & configuration are short at ~200 & ~10 lines
respectively
• same script & configuration in Boston & in Denver
• Boston based script relies on Denver based script and
vise versa
• a known set of functions

8. significant event markers
• events triggered by tracing-log files
• events are recorded & coordinated via marker files
• events are created, transferred, registered, & applied
• markers are distinctively named, empty files in Boston
arch_1_640475_754538743.arc.created
archive-log’s name!
marker!

9. were what happens
• Boston continuously watches for archive-log creations
invokes “transfer & register” process in Denver
• Denver continuously watches for archive-log
applications
invokes “applied” process in Boston
• Boston periodically investigate process’s status
• Boston & Denver periodically clean up

10. some implementation notes
• humans are are good at • check command exit
recovery
status
• use the operating system • retry commands at
... ssh keys
... directory as database
• use configuration files
rather than command
line options
• Use command timeouts
lengthening intervals
… 5s 30s 300s …
• use bash's -E option

11. some resources
• spark.sh & spark.conf
• ask Chuck for our tool, ckosher@crossref.org
• Giuseppe Maxia’s
“Script it! Make professional DBA tools out of nothing”
https://www.percona.com/live/mysql-conference-2013/
sessions/script-it-make-professional-dba-tools-outnothing