A DevOps experiment to make a Jira ticket describing a feature into a deployed application reachable at jira-ticket.serv.sg with a twist: the AWS infrastructure is dynamically created and destroyed once the feature is approved/rejected by the product team. We use Slack, Jenkins, Ansible, Packer, Terraform, AWS, Jinja2 Cli, github

1. How we build features at
Serv.sg
in a Pandan Leaf...
November 25th 2015
Nicolas Mas
Nicolas@serv.sg
Nicolas.Mas@gmail.com
https://www.linkedin.com/in/nicolasmas
https://github.com/nicolasmas

2. Acknowledgment
“Alone we can do so little; together we can
do so much”
Sandeep Isac Marco Yang Alex Hayden
Alexander Bulat Andrei Ping Yang

3. What this talk won’t do
⬗ Entertain you with funny
pictures (sorry). This will be
the only one.
⬗ Describe the tools being
used, unless there is a
specific request (let’s keep
the focus).
⬗ Guarantee that Murphy’s law
won’t impact the demo.

4. So what is it going to be?
⬗ What do we do at Serv.sg?
⬗ Our (current) technology stack
⬗ Feature build (+ DEMO)
⬗ What is the problem we solved?
⬗ Limits
⬗ Enhancements
⬗ B Tracks: if we really have time
Disclaimer: I am a DevOps padawan, what is described tonight is probably
not the best solution ever. Any suggestion is welcome~

5. What we do at Serv.sg
✓ Real time Mobile workforce location tracking
✓ Automated scheduling of service appointments
✓ Automated email / SMS reminders of upcoming appointments and contracts.
✓ Manage customer data and all history of service appointments
✓ Manage Services offered, including recurring contracts and packages, etc.
✓ Advanced reports on employee utilization as well as business

6. Technologies (Stack… for now)
Key
backend
technology
Data
storage
Key
frontend
technology

7. Technologies (DevOps)
For now… (bis)

8. Overview
JIRA-456
JIRA-456.serv.sg
JIRA-456.db.serv.sg
/serv-branch-build
Branch JIRA-456
JOB JIRA-456
On “New Branch”
JOB Monitor Branches

9. Overview
On “Branch Deleted”
JIRA-456
JIRA-456.serv.sg
JIRA-456.db.serv.sg
/serv-branch-build
Branch JIRA-456
JOB JIRA-456
JOB Monitor Branches

10. It all starts in Jira
⬗ The developer takes ownership of, let’s say the item “JIRA-456” during a
sprint.
⬘ Actually, he does not do it in JIRA, but from github by branching out the
codebase.
⬘ The Branch HAS TO BE named after the jira item, e.g. JIRA-456

11. It continues on Slack
⬗ The moment the branch is created, we can actually build the
infrastructure:
⬘ In slack: “/serv-branch-build”
⬗ Slack is asking a Jenkins job to monitor the github repository for
new/deleted branches

12. What happens in the jenkins monitoring job
⬗ The Jenkins job will do a few things:
⬘ 1. It will connect to github and get all the branches
⬘ 2. It will detect our new branch
⬘ 3. A custom jenkins job will be created for that branch
⬘ Terraform will be invocated
⬗ It will also keep you updated of the progress in Slack

13. The Infra is built...
...But not yet provisioned !!
Note: we use a pre-baked OS image for
the instances

14. Building the branch
⬗ In Jenkins

15. Provisioning the branch
⬗ Ansible playbook
⬘ Site.yml dynamically set
⬗ Roles
⬘ Hostname set
⬘ Supervisord
⬘ Yum additional packages
⬘ Web conf
⬘ DB conf
⬘ DB Load
⬘ (...)

16. End Result

17. Destroying the build
⬗ The features has been validated or ditched, the branch is either merged
or destroyed on github
Note: If we need a few back and forth between the product and the dev for the
feature, it’s just a matter of running the jenkins branch job to deploy again

18. Back in Slack
⬗ “/serv-branch-build”
⬗ AWS quick check.. the instances are terminated

19. BTW, What problem are we solving?
⬗ Deploying by feature is not new. Immutable infra seems to be less
common. The two combined? Anyone doing it?
⬗ From a business/product perspective
⬘ More responsive to bug fixing, features deliveries
⬘ No confusion on what has been deployed
⬘ The branch can be hotfixed => one step closer to CD
⬗ From a technical perspective
⬘ Leveraging the cloud infrastructure, programmatically
⬘ No environment recycling or reconfiguring with the related uncertainty
⬘ As close to production as we can
⬘ A good way to craft baked OS with the latest update and test it

20. Limits/Challenges
⬗ Hard to keep track of the full cycle, from the OS baking to the feature
deployed:
⬘ Packer: need to manually describe the target provisioned OS. Hard to test,
need to manage the produced artifacts
⬘ Ansible: need to manually write the roles, hard to test and a pain to
maintain
⬘ Terraform: need to manually describe the infra, hard to test, painful to
maintain and evolve. Some bugs with route 53
⬘ Dependency on Jenkins
⬘ A fair amount of templating (jinja2 style) with a risk induced by the dynamic
aspect of the build (hostnames, EIP...)
⬗ Many pieces in movement, dependency
⬗ Need to be in sync with the product team to check and act upon the
feature as soon as it is available. We don’t want a stack running for ages..

21. Enhancement
⬗ Use a tool to orchestrate the branch + infra build aspect (Spinnaker.io?
StackStorm.io?) or build a simple micro service to do it
⬗ Choose the branch to build in slack instead of all the branches with the
pattern “JIRA-XYZ”
⬗ Improve the provisioning and better manage the ansible playbook
(Tower.io?)
⬗ How is this compatible with our next architecture steps? (micro services)

22. Bonus Track - Did you know ?
“The mother of all demos” http://en.wikipedia.
org/wiki/The_Mother_of_All_Demos
⬗ Douglas Engelbart's December 9, 1968
⬗ 90-minute live public demonstration of the online system, NLS, they had
been working on since 1962
⬗ 1000 attendees and a demo of almost all features we use today
⬘ Mouse
⬘ hypertext, object addressing and dynamic file linking
⬘ shared-screen collaboration involving two persons at different sites
communicating over a network with audio and video interface.
http://web.stanford.
edu/dept/SUL/library/extra4/sloan/mousesite/1968Demo.html

23. https://www.youtube.com/playlist?list=PL76DBC8D6718B8FD3
Watch it !!

24. Bye Bye !