This presentation shows how we at Onegini do Continuous Delivery. It takes you through the challenges we encountered and the choices we made.

1. Continuous Delivery
How it’s done at a SAAS start-up
Presenter: Stein Welberg

2. About Me
Studied at Twente University
Waterpolo Player
Identity & Access management expert
CTO @ Onegini

3. Why am I here?
Sell Onegini
Explain about Continuous Delivery
Inspire you to think about CD
Steal engineers ;-)

4. What is Onegini
SaaS Platform: onegini.me
SSO solution to give external users access to your organisation
Focus on ease of use
Focus on security

5. Continuous Integration
"Continuous Integration is a software development practice where
members of a team integrate their work frequently, usually each
person integrates at least daily - leading to multiple integrations per
day.
Each integration is verified by an automated build (including test) to
detect integration errors as quickly as possible. Many teams find that
this approach leads to significantly reduced integration problems and
allows a team to develop cohesive software more rapidly."
- Martin Fowler

6. Continuous Delivery
Key concepts
Release early, release often
Small releases
Development close to business
Test the sh*t out of your application
Not just the code, also infra setup, db schema, etc!
Have software that can always go live!
Treat anything as software
Automate, automate, automate… automate!!!
Full deployment pipeline
Summary: Continuous Delivery is Continuous Integration on steriods

7. Continuous Delivery @ Onegini
A new release into production every 4 weeks
Devs could release anytime, business can’t keep up with us ;-)
Fully automated Deployments (infrastructure as code)
For all Business critical applications
Automated (integration / acceptance) testing
Critical issues are hardly ever found by our testers
Not all corner cases are tested automatically

8. Our road to Continuous Delivery
The challenges
Ruby (on Rails) and Java combined. Could not easily drop a war into e.g.
Google App engine or Azure.
Plans to use a graph / nosql database in the near future
Run (and be able to customize) and roll own virtual machine
Completely automated build environment testable and repeatable
No (manual) installation or installation manual ;-)
No SSH to the servers
No manual database changes
Time needed to update the production cluster must be less than one hour

9. Choices we made
Cloud Provider: Amazon
Advantages
IAAS: so you can host anything you want
We didn’t know exactly how many machines we needed or what kind of software stack was
necessary to complete the job
A lot of tooling exists
Great API’s
Relatively cheap when you are small
Free tier
Disadvantages
Small machines are really slow
Speed to cost ratio is quite bad

10. Choices we made
Rolling entire machines
Advantages
Machines stay fresh
Every machine is exactly the same (does not matter which environment)
Up- and downscaling is very easy and can be automated
Creating a machine can be automated and tested
Disadvantages
Need to manually run OS updates if you don’t roll machines often enough
No state inside the machine
Database needs to be backwards compatible
Property files need to be present on the machines (can be extracted)

11. Choices we made
AWS RDS Database
Advantages
MySQL or other DB type (Oracle, MS SQL) is available
Easy to setup
Disadvantages
Only marginal customisability. Heavy performance tuning is not possible.
Write performance is not great
Backup and restore results in a different hostname

12. Choices we made
Installation automation: Chef Solo
Advantages
Cookbooks in ruby
Standalone
No need to manually login to a server
Disadvantages
Updating cookbooks does break our build sometimes
Dependant of the community for updates of cookbooks
We have forked some cookbooks because they were broken or out of date
Troubleshooting can be time consuming

13. Choices we made
Base image
Advantages
Saves time: not necessary to install / run initial OS updates on the clones
Install software that is necessary on the entire stack
Disadvantages
If the base AMI contains an error all images do!
Base AMI
based on
Application
Application
AMI's
AMI's
uses
Application
Instance

14. Choices we made
Asgard
Open-source tool by Netflix written in Groovy
Web-based AWS cloud management and deployment
Advantages
It provides the “rolling entire machines” approach
Uses the Amazon API’s
Provides it’s own API’s for automation
Amazon does not provide a UI to create Auto scaling groups
Disadvantages
Can only be used on Amazon AWS

15. Asgard
Cloud deployment model

16. Asgard
Cloud deployment model
App 1
App 2
App n

17. Asgard
Fast rollback

18. Asgard
Fast rollback

19. Asgard
Fast rollback

20. Asgard
Fast rollback

21. Asgard
Fast rollback

22. Asgard
Fast rollback

23. Build Architecture Overview
local
Dev
commit
test
trigger
Build pipeline
Jenkins CI
+
detect
create
Asgard
test
create
use
+
Subversion
AMI
Inst ances

24. Demo
Asgard
Chef Cookbook
Deploying a new version to test

25. Lessons learned
Needed to clean up old Amazon AMI’s, volumes, launch
configurations, etc
A lot of custom ruby scripts
AMI builder
Base AMI builder
Cleanup
Custom way to prevent building the entire pipeline in jenkins
Automating to starting a new machine and connecting to it via SSH is
quite ugly..
Amazon reuses IP addresses, this sometimes results in a fingerprint
conflict which causes a pipeline failure

26. The future
Amazon VPC setup
Network segmentation
In and egress filtering
Migration to another data center
Host data in a data center that is not owned by an American organisation