Net-a-Porter has embarked on the mission of separating database refactoring from code deployment. The solution we've come up with is "refactoring as a service" and a soon-to-be released Perl module which drives it. I'll explain how Liquibase, our Git repositories, Puppet and Jenkins all fit together to make database refactoring easy and deployment safe and roll-backable. I'll also tell you how I just discovered Sqitch as a possible replacement for Liquibase. ... and all within 20 minutes!

1. +
Database Change Management* as a service
* Also known as ‘patching’ or ‘refactoring’

2. +
About me
(Andrew Solomon)
Was …
an academic teaching Perl
Am now doing …
 Perl Web Development at Net-A-Porter
 (learning) Devops at Net-A-Porter
 Teaching at Geekuni

3. +
Motivation
Data is inconvenient
 Next app release coming
 App has new features
 Needs different DB structure
 Can’t just trash and install DB like an app

4. +
Principles
For safe refactoring
 Backward compatibility
 Old and new app versions are ok with the refactored DB
 DB is versioned
 You can quickly determine which patches have been applied
 App won’t be run if the DB is not compatible
 Write a rollback for every DB change
http://www.amazon.co.uk/Continuous-Delivery-Deployment-
Automation-Addison-Wesley/dp/0321601912

5. +
The Net-a-Porter Way
Old school

6. +
The Net-a-Porter Way
Old school
 Put refactor code into My-App’s code base
 Approach A: Run when installing the My-App RPM
 Approach B: Run when starting My-App
 Log the patches run in the database

7. +
The Net-a-Porter Way
Old school
 Put refactor.pl into My-App’s code base
 PROBLEM: A slightly different refactor.pl for each app
 PROBLEM: refactor.pl has extra CPAN dependencies
 Approach A: Run when installing the My-App RPM
 Approach B: Run when starting My-App
 Log the patches run in the database

8. +
The Net-a-Porter Way
Old school
 Put refactor.pl into My-App’s code base
 PROBLEM: A slightly different refactor.pl for each app
 PROBLEM: refactor.pl has extra CPAN dependencies
 Approach A: Run when installing the My-App RPM
 Approach B: Run when starting My-App
 PROBLEM: Multiple My-App servers per DB
 Log the patches run in the database

9. +
The Service Way

10. +
Available Refactoring Tools
Flyway
Liquibase
DB Deploy

11. +
Winner: Liquibase
Big user base
Bought by Datical
Has branching and
More features than Flyway
DB Deploy uses Apache ANT

12. Overview

13. Liquibase

14. +
Liquibase
How it works
Input:
 An XML file listing SQL files to run
 Special comments in the SQL files for tagging
Output:
 ‘databasechangelog’ tables
 Which patches have been run
 ‘databasechangeloglock’
 Ensure only one refactoring process!

15. +
Liquibase
Example XML
<databaseChangeLog …>
<include file="1.0.0/000-create_schema.sql”
relativeToChangelogFile="true"/>
<include file="1.0.0/010-populate.sql”
relativeToChangelogFile="true"/>
<changeSet author="a.solomon" id="tag-1.0.0">
<tagDatabase tag="1.0.0"/>
</changeSet>
</databaseChangeLog>

16. +
Liquibase
Example SQL
--liquibase formatted sql
--changeset a.solomon:1.0.1-000
BEGIN;
CREATE TABLE public.foobar (
id INTEGER PRIMARY KEY,
name TEXT UNIQUE NOT NULL
);
ALTER table public.foobar OWNER to magpie;
--ROLLBACK DROP TABLE public.foobar;
COMMIT;

17. +
Liquibase
Example Update
/usr/bin/liquibase
--changeLogFile=changeset.xml
--url="jdbc:postgresql://db1.myapp.com:5432/myapp-db1"
--username=liquibase –password=foobar update

18. +
Liquibase
Example Rollback
/usr/bin/liquibase
--changeLogFile=changeset.xml
--url="jdbc:postgresql://db1.myapp.com:5432/myapp-db1"
--username=liquibase –password=foobar rollback 1.0.0

19. +
Liquibase
Example databasechangelog
author | filename | md5sum | tag
a.solomon | 1.0.1/000-create_schema.sql | 7:49c8 | 1.0.1

20. +
Liquibase::Git
What is it?
 Perl module and script (soon on CPAN?)
 A perl script
 Clones git repo of the app
 Looks in a directory of patches
 Calls liquibase

21. Liquibase::Git

22. +
Liquibase::Git
Example
$ liquibase-git --username liquibase
--password foobar
--db mydb-db1
--hostname db1.myapp.com
--git-repo https://github.com/nap/myapp.git
--git-changeset-dir db/db1
--git-identifier master
--db-type postgresql
--changeset-file changeset.xml

23. +
NAP::Liquibase
Getting the Puppet to help
Most of Liquibase::Git’s parameters are retrieved from the Puppet
DB
Just reads a new file from My-App – the list of DBs to refactor
# in the puppet hieradata
$ cat myapp.com.yaml
liquibase:
databases:
- db: 'myapp-db1'
- db: 'myapp-db2’
# ask mco for myapp’s dbhost
$ nap-liquibase --db-host db1.myapp.com
--app-git-repo https:://github.com/nap/myapp.git
--app-git-identifier master

24. NAP::Liquibase

25. Jenkins Deploy-oid
mco ping … -F system_db=liquibase

26. +
Appendix
… managing DBs outside the Puppet estate
Cases where the box is being managed differently with puppet…
Jenkins Continuous Integration Test Server
Developer’s Box
 Option 1: Emulate Liquibase on an empty DB – 5 lines
 Option 2: Call liquibase-git to update a DB dump

27. +
Conclusion
Don’t write new DB refactoring
code for each app
Don’t assume you’ll only have one
app server per DB
Don’t deploy DB refactoring code
on the app’s host
Treat it as a
service

28. +
But wait!!! There’s more…
 Ilmari told me about it
 It’s a Perl project of David E.
Wheeler / theory
 Based on sane principles:
http://sqitch.org/
Sqitch is to Liquibase
Is what
Git is to CVS
 The switch to sqitch will be almost
invisible at the architecture level
 Only difference is that the developers
get LOCAL application of the patches
on the dev box with a simple
command-line interface

29. +
Questions?
… or grab me at the pub.