Streamline your database changes by versioning your database instances and your database schema, running database instances locally and implementing database changes using migration scripts based on database refactoring patterns.

1. Database Schema Evolution
Lars Thorup
ZeaLake Software Consulting
August, 2013

2. Who is Lars Thorup?
● Software developer/architect
● C#, JavaScript and C++
● Test Driven Development
● Coach: Teaching agile and
automated testing
● Advisor: Assesses software
projects and companies
● Founder and CEO of
BestBrains and ZeaLake

3. Elevator pitch
● Don't do this
● Do this instead
$ importData
Error: Invalid column name 'PartnerId'
$ ...???
$ importData
Error: Database should be version 47 but is 44
$ migrate
Migrated database from version 44 to 47
$ importData
Success!

4. Keeping database and app in sync

5. Demo
$ SampleData.exe migrate 32
Latest version is 73
Current version is 30
Target version is 32
Migrating up to version 31 using 31_section_track_up.sql
Migrating up to version 32 using 32_action_kind_up.sql
$

6. Got problems?
● Tests fail because data was changed by someone else?
● Upgrades fail because some script was missing?
● Upgrades and downgrades can't be tested reliably?
● Creating new test and development environments is a
hassle?
● Who made this schema change and why?
● Applications fail because code and database are not in sync?

7. We have solutions!
● Never share a development database
● Automate database schema deployment
● Version your database schema and database instances
● Base script, up and down scripts
● Check db version on app start
● Tools: FlyWay, LiquiBase, home grown
● Database refactorings
● Migration conflicts on commit
● Data builders

8. Never share a development database
● Run development database locally
● MySQL
● Use free versions of commercial database products where
available
● Oracle XE
● SQL Server Express
● Use compatible open source database products if feasible
● PostgreSQL instead of Netezza

9. Automate database schema deployment
● Recreate database from scratch
● Development boxes and CI servers
● $ mvn dbschema
● Upgrade database to newer version
● Test and production servers
● $ migrate
● Rollback database to older version
● Production servers
● $ migrate down 44

10. Version your schema and instances
● Schema described by .sql files
● Keep those .sql files under version control
● Include a SCHEMA_INFO table in the schema
● Record the database version in SCHEMA_INFO

11. Base script, up and down scripts
● 1-base-up.sql
● If you are not starting from scratch
● Contains a dump of the current schema
● Including the SCHEMA_INFO table
● 51-customer-unique-{up,down}.sql
● up: migrates the schema from version 50 to version 51
● down: migrates the schema from version 51 to version 50
● Tip: establish a naming convention
● List all scripts in a central file
● migrations.txt

12. Up and down scripts
● 51-customer-unique-up.sql
alter table Customer
add constraint NameUnique
unique nonclustered (Name) on [primary]
● 51-customer-unique-down.sql
alter table Customer
drop constraint NameUnique

13. Check db version on app start
● Application code
var expectedVersion = Config.ExpectedDatabaseVersion;
var actualVersion = session.GetDBVersion();
if (expectedVersion != actualVersion)
{
throw new ApplicationException(string.Format(
"The database needs migration
(its version should be {0} but is {1})",
expectedVersion, actualVersion));
}

14. Tools: FlyWay
● Open-Source, Java-based
● Plain SQL scripts

15. Tools: LiquiBase
● Open-Source, Java-based
● XML or formatted SQL scripts

16. Database refactorings
● Catalog of database refactorings
● Split Table, Merge Column, etc
● 18-replace-roles-with-privileges-up.sql
alter table Participation
add Privileges int constraint DF_privileges
default 1 not null
update Participation set Privileges=1
update Participation set Privileges=3
where UserId in
(select Id from User where Email='admin')
drop table UserRole

17. Migration conflicts on commit
● Commit might conflict on migrations.txt
● Merge migrations.txt, keep both
● Renumber your migration
● Test
● Commit

18. Test data builder
[Test]
public void GetResponseMedia()
{
// given
var stub = new StubBuilder
{
Questions = new [] {
new QuestionBuilder { Name = "MEDIA" },
},
Participants = new[] {
new ParticipantBuilder { Name = "Lars", Votes = new [] {
new VoteBuilder { Question = "MEDIA", Responses =
new ResponseBuilder(new byte [] {1, 2, 3}) },
}},
},
}.Build();
var voteController = new VoteController(stub.Session);
// when
var result = voteController.GetResponseMedia(vote.Id, true) as MediaResult;
// then
Assert.That(result.Download, Is.True);
Assert.That(result.MediaLength, Is.EqualTo(3));
Assert.That(TfResponse.ReadAllBytes(result.MediaStream), Is.EqualTo(new byte[] {1, 2, 3}));
}

19. Read more
● http://techportal.inviqa.com/2011/01/11/database-version-
control/
● http://databaserefactoring.com/
● http://flywaydb.org/