Every customer-facing project aims to achieve high uptime. We, as developers, write thorough unit and integration tests to catch as many bugs as possible. We even set up staging environments to simulate production as well as we can. Yet, downtime still happens and not only once we had to revert a change. Blue-Green deployment is a technique that allows us to ship code with high confidence, minimizing the risk of errors. We will go through the why and what of Blue-Green, as well as show how you can start doing it using Spinnaker.

1. Safe deployments with
Blue-Green and Spinnaker
Platform Bucharest
23.02.2017

2. Hi
Mihnea Dobrescu-Balaur
@mihneadb
https://www.mihneadb.net

3. Deployment
• getting product out to clients
• agility matters - first to market
• speed vs risk

4. Rolling deployment
• classic/easiest way of doing things
• on every server: git pull; restart
• you are probably doing some variation of this

5. http://searchitoperations.techtarget.com/definition/rolling-deployment

6. Rolling deployment (cont.)
• Advantages:
• simple
• no extra hardware
• can do partial deploys

7. Rolling deployment (cont.)
• Disadvantages:
• might drop requests
• slow rollback
• finding the "last good state"
• testing in staging only

8. Blue-Green deployment
• two identical production environments
• only one serves requests
• deploy to the new one; test; route traffic

9. http://fbrnc.net/blog/2016/05/green-blue-deployments-with-aws-lambda-and-cloudformation

10. Blue-Green deployment
(cont.)
• Advantages:
• testing in production
• no downtime
• quick rollback
• predictable release time

11. Blue-Green deployment
(cont.)
• Disadvantages
• cost - 2x servers
• databases require extra work
• development time to implement

12. Fixing the database problem

13. Optimizations
• connection draining
• canary
• sticky sessions
• automated tests on green
• reproducible deploys using AMIs/Docker images

14. Real world
• using it at Hootsuite since summer 2015
• 1K deploys in prod with no issues*
• has not affected our velocity
• deploy to prod is a mundane task
* trivial rollback in case we had any

15. Real world (cont.)
• Tech stack:
• Docker
• Asgard
• Jenkins
• ELB
• Hubot

16. Our deployment pipeline
http://code.hootsuite.com/blue-green-deployments/

17. Our deployment pipeline
(cont.)
• cluster check
• conflict check
• merge tests
• live tests
• git merge
• manual canary release
• manual blue-green switch

18. Spinnaker
• successor of Asgard (thanks, Netflix)
• custom build and deploy pipelines
• multiple cloud providers supported

19. Why Spinnaker?
• easy trigger integration with Jenkins
• handles all the "cloud" stuff
• server groups
• load balancers
• security groups
• useful templates built-in

20. https://cloud.google.com/solutions/spinnaker-on-compute-engine

21. Spinnaker stack
• Jenkins
• Packer
• .deb repo
• Spinnaker core

22. Deploying .debs
• not everybody used to this
• lifesaver: fpm
• running your stuff - upstart

23. Demo!
http://www.spinnaker.io/docs/from-source-to-prod