Presentation used at PortoData (28/07/2016) about manageing database changes in application projects. Demoed Flyway migrations for script packages, and a strategy for using DBUp and localDb for integration testing at the data layer
2. Delivering changes for applications and databases
@EdPiairo
https://pt.linkedin.com/in/jesuspiairo
eduardopiairo@outlook.com
http://www.eduardopiairo.com/
Eduardo Piairo, DBA & DevOps @ Celfinet
3. Delivering changes for applications and databases
@MytyMyky
https://pt.linkedin.com/in/miguelalho-982331
alho@miguelalho.com
http://www.miguelalho.pt/
Miguel Alho, Software Engineer @ Faro Europe
8. DLM – Database Lifecycle Management
Data management and data migration
Data monitoring
Data recovery
9. Typical problems we want to fix
• Databases are out of pace with application development
• Lack of traceability of database changes (changes history)
• Manual databases processes prevent the CI and CD utilization
• Lack of testability
• Database setup time for new environments
• Bugs in production
10. Typical problems we want to fix
Databases becoming a
bottleneck in an agile
delivery process
13. Why database automation?
• Enable control over database development
• Increase speed of response to change
• Keep a versioned “history” of database states and changes
• Reliability of the release process
• Increase release frequency though repeatability of processes
• Reduce time spent fixing bugs - automated tests
• Remove/reduce human intervention in the release process
15. 1st step: Source Control
• First step in your database deployment pipeline
• Traceability through change history
• SQL as documentation
• Shared code-base and shared process
• Enforceable standards to reduce conflicts
17. Migrations-based approach vs State-based approach
• State based solutions
• Script represents the current database state
• Your source of truth is how the database should be
• Migrations based solutions
• Script represents a migration
• Migration represents how to transition to the next database version
• Your source of truth is how the database should change
18. Flyway
• Flyway – “Database Migrations made Easy”
http://flywaydb.org/
• Open source database migration tool
• Flyway command-line enabled
Simplicity: easy to setup, no need to install
Zero dependencies (java + jdbc)
Scripts are written in SQL
19. Flyway – How does it work?
• Flyway commands:
• migrate, clean, info, validate, baseline, repair
• Metadata table:
• Used to track the state of the database
• If the database is empty, flyway won’t find it and will create it
• Flyway scans the migrations directory and check migrations against the metadata table
• Migrations are sorted based on their version number and applied in order
20. Version Control System
Two components:
Database scripts repository
Flyway – database migrations
Rule 1: Script version (timestamp)
Rule 2: Operation type
Rule 3: Object type
Rule 4: Object name
Example: V20160728.1100__Create_TB_MyTable.sql
23. Database and Application change alignment
● Database schemas should evolve along with application evolution
○ deploy app => deploy changes / migrations
● (Integration) Test app and db changes together
○ should pass tests while developing,
○ should pass tests while promoting to new environments
● Ideally, one application per database
● Pipeline deployment only!
24. Integration test at the database layer
● Use TDD techniques
● Limit database integration tests
to data layer
○ faster tests!
○ validate sql and mappings
● Always test with the latest
schema
DatabaseApplication
DatabaseApplication
28. Challenges
• Challenge #1: managing multiple branches
• Manage merge conflicts
• The default branch is the main/master
branch and all the merges for this branch
must be done with caution
29. Challenges
• Challenge #2: Script expiration date
• Should the migration script have an
expiration date?
• Sometimes, in the next day, the migration
script no longer makes sense
30. Challenges
• Challenge #3: migration scripts stack
• How can I manage the dependencies
between different databases?
31. Challenges
• Challenge #4: developer sandbox
• How to supply a disposable development
environment where the database is
included?