Skybase system is a DevOps platform designed to be used for deployment and maintenance of Services inside all locations of an organization including Dev, QA, Prod and different clouds and geographic regions and data centers.
1. SkyBase – DevOps Platform.
(or Finding your way in Hybrid
Clouds)
Vlad Kuusk
Lead CloudOps Engineer
(May 2014)
2. what is this presentation about?
About science fiction? – No
About software project?
– Maybe, but mostly about finding a way
to simplify the complex situation and
create a software to bring order out of
chaos.
3. Introduction
• Lithium is moving into Hybrid Cloud and
adopting DevOps model
• A lot of tools and approaches, but one still
needs to glue them together
• Thus project SkyBase was born.
4. What is going on at Lithium?
• We are SaaS company with 400+ enterprise
customers
• Moving into hybrid cloud
• Establishing CI and CD processes
• Adopting DevOps model
• Releasing independent services using SOA
While running at 100 mph with main business
5. What we are dealing with?
• The goal is to find a common ground between
services and teams! --- but it’s like herding cats.
6. Role of Ops in DevOps
To support multiple DevOps teams we need to
create a base infrastructure (platform), so these
teams can maintain services they develop.
Among other things, this platform has to contain
a deployment pipeline.
How this Deployment Pipeline might look like?
8. How to implement this pipeline in our
case?
• Currently two teams working on the project from Dev and
Ops sides So establishing of CI and CD processes
could/should proceed in parallel.
• Define the standard target location for app deployment ->
simplify promotion job Dev QA Prod.
• Decouple deployment mechanism from promotion job
quickly adapt to changes in infrastructure and business
workflows
• Two new concepts: “ArtiBall” and “Planet”
9. WTF is ARTIBALL ?
• Decoupling CI and CD with ArtiBall* ( Tarball of Release Artifacts)
• Artiball contains everything needed to create a service, including Infrastructure-as-a-Code
– Application code (package)
– Application configs (for all universes)
– Chef cookbooks to install servers
– Deployment Templates (last but not the least)
* - term Artiball was borrowed from Chef Conference in SF, Apr 2014
12. Pipeline is a sequence of deployments
into different planets
• Business decides the order. But it is independent from the
deployment engine.
13. Basic Single Deployment steps
1. Prepare Support Services (DNS, Yum, S3,...)
2. Prepare Chef Server (CookBooks, Roles, DataBags)
3. Launch Instances and point them to correct Chef
Server
4. Verify Functionality (Smoke Test) of New Deployment
5. Allow Traffic to the New Deployment
15. Four Truths about
Deployment Pipeline
As Hodja Nasreddin says: there is no wrong way to ride a pack mule.
1. There is no single universal way to create deployment pipeline
1. Tools give too much flexibility to users causing confusion.
1. It is possible to create Deployment Pipeline to be useful for any company
1. Skybase is the way to do it.
16. Design Principles and Technologies
• Minimalistic user interface – less options but
balanced with the ease of changing standards
(flat files)
• Chef for server configuration
• Salt for remote execution and orchestration
• Written in Python
• Managing templates for AWS Cloudformation
and Openstack Heat
20. Conclusion
• Decouple CI and CD using Artiballs
• Define standard planet to simplify configuration
• Identify fundamental steps of deployment and
allow flexibility in each
• Use Skybase to glue it all together
21. Future of Skybase platform
• ChatOps – HipChat + Skybot (based on Hubot)
• Connectors to external RESTful API services (Jira,
Monitoring and Cost Analytics)
• Workflow Service (sequence of actions)
• Open Source coming soon
Decisions about: Which CM: Chef, Puppet, Salt, Ansible, ...?
Hybrid Cloud Management System -Rightscale, Scalr, ..., or create our own
How Continuous Integration and Delivery pipeline should look like?
After trying different tools and approaches we discovered that with all those tools out there we were still missing something.
Here I’d like to tell the story of our journey and present our rationale for the choices we made and describe the solution, which we are currently developing
I hope that this talk will be helpful for those of you who are in the same position as we are and who feel the same pain as we are.
.
all at the same time is a very tricky situation
To give an idea what we are dealing with here.
1. Multiple Apps (2 major and some other standalone services)
2. Each app is customized and deployed for multiple customers
3. Different versions of App are potentially running for different customers
4. Different services deployed separately but depend on and interact with each other
5. Multiple Teams develop and deploy their services into production
It’s not just a simple move from Dev-QA-Prod for a single app.
The role of Ops team is to provide common tools for the other DevOps teams to deploy and manage their services.
This is our pipeline.
It’s more or less standard pipeline, where during each phase different groups of servers are used (orange blocks).
These groups are usually called environments: Dev, QA, Stage, Prod.
Later we’ll see that word environment might be confusing in situations with multiple locations and clouds.
Usually pipelines are promoting new release automatically up to QA environment. After that people are usually using the approval gates to UA and Prod.
We were focusing in several areas.
We came up with two main concepts: Arti-Ball and Planet.
What are these?
Decoupling of CI and CD is achieved by creating an ArtiBall , so CI creates AB, then CD uses AB.
From our experience, when discussing CD pipeline and even simply describing location of the server or service
term “EVIRONMENT” is often ambiguous and confusing. And even in the same conversation different people mean different things.
Is it DEV/QA/PROD or it’s a colocation cage, or is it Chef Server Environment and so on.
Let’s consider this matrix and define planet as Universe-Provider-Region. We can think of planet as a VPC.
These 3 parameters is enough, because there is no need to have 2 VPCs in the same region.
1. “Planet” is based on a set of subnets grouped by location and purpose. ( Dev-AWS-us-west-1, Prod-AWS-eu-west-2)
2, “Planet” contains supporting infrastructure to provision, install and maintain Our Services.
Usually One Our Service runs in one planet.
One Deployment is always putting one ArtiBall into one Planet
Multi-Region deployments will spend across two regions.
After we introduced Planets the implementation of deployment pipeline simplified.
Any Pipeline can be configured as sequence of single planet deployments
Any deployment system has to include these steps.
Now let’s talk about implementation of concepts described earlier.
But first this...
Let’s talk about our approach but first an old joke:
Once two men came to Hodja Nasreddin and asked him to resolve their dispute.
Hodja agrreed to do this.
First man says: “I think that this thing has to be done from right to left” – Hodja says: “You are absolutely right!”
Second man says: “I thing that same thing has to be done from left to right: - Hodja says: “And you are absolutely right!”
Both men left with joy.
A bystander, who saw this, says to Hodja: “Hodja, it is impossible that both men can be both right at the same time!”
So Deployment Pipeline is a such thing.
“And you are absolutely right!” – says Hodja Nasreddin.
CI process dumps AB to an S3 bucket
Process AB
Process target planet data
Update Support services
Update Chef server
Create and apply CF templates for the stack
Stack is created and Chef client pointed to Chef server
Chef Server completes the install
Client
WebApp with REST API
Worker (executes a request)
SkyBase Clases perform actions on AB and Planet)
Decoupling of the CI and CD allows to develop these processes independently at different pace – This is accomplished by using ArtiBalls ( Release Bundles )
Defining a target for deployment as a planet allows greatly simplify management of stack deployment templates ( e.g. Cloud Formation or Heat templates) thus accelerating transition of developers into DevOps.
There several fundamental steps in the deployment, and with correct architecture it possible to change the mechanisms of each step without affecting other – which allows to create a functioning system while some parts of the design are still in discussion or if the new system has to coexist with legacy mechanisms
Skybase by definition is a central location, which has access to all elements of infrastructure and can easily be extended with new functionality, and as such it presents a convenient DevOps platform for all
External services like monitoring, cost analytics, ...
Openstack “Mistrel”
