How ecommerce site support company is doing DevOps

1. DevOps in real time
Andriy Samilyak
OpsWay
CEO

2. 20 online projects
One 24х7 support team

3. Me v0.1
● 7 yrs in IT
● since 2008 – Smile Ukraine CTO
● since 2013 – leading OpsWay
● coaching people on DevOps

4. Problem №1 – life in 24x7
Часы
%
new
requests

5. Problem №2 - Geography

6. Problem №3 - Developers
“When I see ... developers – all I see is risk”
Chuck Rossi, Facebook

7. Plan
● Processes
– Deployment cycle
– Working with many teams
● Tools
– Chef for automatic configuration
– Continuous integration
– Monitoring
– Backups
– Team communication

8. Opscode Chef
● Full control over server configuration
● Continuous delivery
● Access control
● Making testing simplier
● Controlling what and when is monitored
● Configuration as application vs documentation

9. Why Chef?
● Facebook, Amazon
● Ruby vs XML
● Simple start
● Not expensive and reliable hosted server
● Historically it was a decision against Puppet
(no holy wars!)
vs

10. Opscode Chef - beginning

11. Autoscaling projects

12. Single rules for server naming
ansam@vostok:~$ knife sshx projectx-live-node
Multiple instances found!
1. i-cca165f0 since 2013-08-06 10:11:16 UTC ip 54.253.26.66
2. i-6ca71550 since 2013-08-07 19:06:05 UTC ip 54.253.66.83
3. i-6ea71552 since 2013-08-07 19:06:05 UTC ip 54.253.44.185
4. i-6fa71553 since 2013-08-07 19:06:05 UTC ip 54.252.44.245
Choose <1-4>: [1] 2
Instance i-6ca71550 ip 54.253.66.83
Connecting to 54.253.66.83 as user ubuntu on port 22
проект среда роль

13. Controll access by SSH

14. Templates
Apache virtual host template

15. Continuous integration
What else can you choose?:
– TeamCity
– Bamboo
– Travis CI
– tool Х
Why Jenkins?

16. Deployment

17. Access control

18. Simple deployment cycle
Development
Vagrant box
Test server Live
Many features simultaneously developed

19. Branch switch

20. Working with other teams
master
develop
fork for DevTeamX - develop

21. Bakery
Jenkins
jobs
Live Amazon AMI
Sandbox
about 4 hours
5 min

22. Creating a Sandbox

23. Sandbox
● DNS via Route53
● Can make demo to client
● Deployed from к github repo/branch
● New job in Jenkins for deployment – easily
redeployed
● EC2 instance
● Almost complete copy of live project

24. Workflow 1.1
Scheduled, once a day
Live
Test
Sandbox1 Sandbox2

25. Code is live – is it working?

26. Monitoring
Configure
monitoring
depending on
server type
● Message routing
● Scheduling plugins
● Cyclic DB
● Data processing
● Charting and plotting
● results DB
● search
● alerts
● Analysis and charting

27. Chef-controlled monitoring

31. ● Take all records from access.log from all servers of ProjectX
● Include records with User-Agent = “Apache*” and URL like /aaaa/bbb-123.html
● Calculate avg time of such requests for the last 7 days

32. Our dashboards

33. Synthetic tests

35. Redmine
SMS
email
Splunk alerts

36. Chef + Amazon EBS = Backup profit
EBS
Server configuration
(Chef)
/var/www

37. Backups

38. Backups: snapshots

40. Andriy Samilyak
samilyak@gmail.com
skype: samilyaka
http://opsway.com