The easy part of implementing chef is getting applications and OS configuration within the system. But how do you deal with databases? We'll talk through the patterns that are out there and give some examples of how NCR Hospitality is approaching the problem.
https://youtu.be/DYSvbcFC2ck
5. 2008: Darkness
Automation and CI not
enough
Influence Deployment
and Delivery
Blog at hedge-
ops.com
Married to a
photographer
5
About me :: Michael Hedgpeth
15. Our new process will be consistent
BUT we are in debt with our old process
We’ve made schema consistency a separate initiative
REDGATE is a great start to this effort
6/2/2015 15
Schema Consistency Is Separate
Who I am: and I’m Michael Hedgpeth from NCR
Brief Problem Overview:
Databases are a key part of our infrastructure, but nobody talks about them!
I’d like to lay a foundation that will get us to start talking about it
Outline:
Context
Requirements
Example
Strategy
Context :: NCR
We invented the cash register 131 years ago
We are reinventing ourselves into a hardware-driven software company
I work within the Hospitality division of NCR which mainly focuses on providing technology solutions for restaurants globally.
Our biggest challenge is:
High availability with a side of fries –
What happens when we’re ringing someone up and they spill a coke on the terminal?
Each transaction is vital - failover
Rapidly change out physical components quickly
Deal with an unsafe world
Tell My Story:
Remember 2008 when the world was going to end?
We wanted to automate build and test – did so with TeamCity as our core
It wasn’t enough
Biggest constraint: deployment and delivery
Blog plug
Annie Plug
In 2014 we wanted to automate configuration management of our hosted products in the DC and engaged Chef for a division-wide automation initiative
Story: Daniel’s first question (Windows, PostgreSQL, hadoop)
Snowflakes: frustrated me because it felt like we were supposed to figure it out on our own
felt like Karate Kid
After doing the investigation, though, the database question has
Simple solutions that apply to many different situations
It’s more than just snowflakes
So when I decided to go to ChefConf, I wrote the organizer that this is a session that I wanted to attend, and if they weren’t providing it, I would give it.
When people ask the database question, they’re talking about one of four things
How do we manage virtual images for creating new database nodes?
We already solve this in other ways
We could use chef for this, but it’s not important to us right now
How do we do backup and restore? – this is already solved by Red Gate Backup
How do we do schema management? – this should be already solved, but we’ll need to focus on this some more
How do we configure the database itself once it is installed? – this should be already solved but we’ll need to focus on this some more
So we’ll talk about the last two
A majority of the session will be about schema management
Before we do, let’s talk about what we aren’t talking about first.
We want to create a repeatable schema management solution going forward. But what do you do about the past bad behavior that has created schema inconsistency?
We need to deal with that separately. We’ll use a tool like REDGATE to ferret out differences in schemas and solve them. We’re not going to use Chef to discover differences in schemas, because that’s not what Chef is great at.
Schema consistency issues, when they exist, have become separate but complementary initiatives to chef.
Our strategy for schema management is simple:
Let’s not reinvent the wheel. If a team has a schema management approach that fits our requirements, we let them keep that approach. If not, we’ll make minor improvements to the approach to fit the requirements I’ll share in a few moments.
We really want to make existing solutions Chef friendly. And this has proven to be remarkably simple.
Our requirements are simple:
We need for all actions related to schema management to be persisted within the database itself.
This lets you check a database and see what has happened
It lets you restore a backup and then apply an update seamlessly
It’s just easier
We need to run something that knows if a change is needed
Just like chef – it applies to databases too
We need to execute those changes
We need to run it within a recipe in an idempotent way
This book is awesome
It breaks down reading into composable pieces
And reproduces it over and over again
And then my son gets a reading rainbow cake
That’s pretty much what we’re going to do
Except the cake is awesome automation
Let’s look at a sample application that fits these requirements
Available now on github and the chef supermarket
Also look at my blog for links: hedge-ops.com
A Version table within the database keeps track of what has run against the database
We measure everything just as we do with software in the database
It’s natural to persist stuff in the database!
Create a scripts folder with sql files where the name is equal to the version number.
The application can initialize this table within the database, and a resource within the cookbook ensures that the table is there and ready to go.
We need to run something that knows if a change is needed
We need to execute those changes
We need to run it within a recipe in an idempotent way
And we’re going to use John Keiser’s awesome resource cookbook!
We need to run it within a recipe in an idempotent way
We need to run it within a recipe in an idempotent way
We need to run it within a recipe in an idempotent way
Before you put it in production, know these things:
It’s only for SQL Server
It’s not transactional, so failures will leave you in a bad state
Use it as a starting point for ideas
Before we finish the topic of schema management, let’s talk about a couple of challenges we’ve faced when planning out schema management for all of our products.
Story: flagship product, thousands of databases on hundreds of nodes – all upgraded at once – with Scott
Analytics will tell us, but we need real time – we use these tools for certain things but need a tighter time resolution
What to do if the internal team does a lot of changes but wants to limit the changes available to external teams? Have a major/minor version! (image – internal/external, maybe a doorway)
Analytics will tell us, but we need real time – we use these tools for certain things but need a tigher time resolution
Story: Mike manages dozens of schema changes a month in every release. How does he manage it?
Configuration Management boils down to database engine configuration, which is limited
At first we think that database people will be the most resistant to change
That hasn’t been my experience
They are my biggest allies, and here’s how I’ve made it this way
We’re going to make the other nodes in the system better, which will make the database easier to manage.
It’s easier to create a preproduction environment that models production. A lot of that comes down to cost of time, and we are lowering that through automation
The servers in the infrastructure make a transition from pets that we love adore and know the names of, to cattle. This gives them more flexibility to changing their solution based on changing technology
The servers in the infrastructure make a transition from pets that we love adore and know the names of, to cattle. This gives them more flexibility to changing their solution based on changing technology
I got frustrated about snowflakes
But now I know what it was about:
Chef is a framework that combines existing solutions with new capabilities
I don’t have to wait around for the right answer
I can create our right answer
With applications and with the database
Let’s keep talking about this; I know there’s more to discuss.