When a brand new project is started from scratch, you are free to choose between any and all technologies, architecture, infrastructure and so on. But do you consider if these decisions will still be viable after a year? Sometimes you have to make crucial changes and even migrate infrastructure between clouds.

1. Audrius Žiemelis
www.devbridge.com/people
ServiceBridge: Azure to AWS
January, 2016

2. What is ServiceBridge?
ServiceBridge field service management software is designed to assist
franchise businesses with work order scheduling and distribution, eliminate
paperwork and engage customers to increase sales. We’ve had great success
with landscape maintenance, carpet cleaning, pool cleaning, pest control, and
disaster clean up service franchises.

3. Facts
• Company count: 556 (646)
• User count: 1751 (2129)
• Customer count: 1 467 283 (2 051 145)
• Job count: 5 995 674 (8 557 062)
• Database size:
Shard1 - 34Gb (43Gb)
Shard2 - 31Gb (37Gb)
SB - 12Gb (18Gb)
• ElasticSearch size: 32Gb (10Gb)

4. ServiceBridge components (Azure)

5. • SQL Profiler (only locally)
• Logging improvements to ElasticSearch
• NewRelic
Monitoring tools

6. Attempts to keep wheels running...
Infrastructure:
• Distributed Cache implementation
• Database 2nd shard implementation (by CompanyId)
• ElasticSearch – 3 smaller nodes with more RAM instead of 2 big
• MSSQL Index maintenance
Software:
• Bulk inserts / updates (NHibernate.IStatelessSession)
• DynamicUpdate anotation
• new TransactionScope() – by default Serializable!
• NHibernate IsDirty check

7. The last drops of patience
• DTU – what is 250 or 500?
• Azure SQL: P1 → P2 → P4 → P6 → P11…
• Azure SQL Price: $465 → $930 → $1860 → $3720 → $7001
• Performance is still bottleneck
• Cost cost cost!

8. Text
Text

9. Text
Text

10. IAM

11. VPC

12. EC2

13. Directory Service

14. CloudTrail

15. CloudWatch

16. SNS

17. ElastiCache

18. S3

19. Lambda

20. CodeDeploy

21. Trusted Advisor

22. CloudFormation

24. Infrastructure
● 2 x Web Servers (SB360) (c4.xlarge, 4 vCPU, 7.5 GB Memory)
● 2 x API Servers (c4.xlarge, 4 vCPU, 7.5 GB Memory)
● 2 x Web Servers (c4.large, 2 vCPU, 3.75 GB Memory)
● 2 x SQL Servers (r3.2xlarge, 8 vCPU, 61 GB Memory)
● 1 x SQL Servers (r3.xlarge, 4 vCPU, 30.5 GB Memory)
● 3 x ElasticSearch servers (for search), (r3.large, 2 vCPU, 15.25 GB Memory)
● 2 x ElasticSearch servers (for logs), (r3.large, 2 vCPU, 15.25 GB Memory)
● 2 x NAT Servers (m3.medium, 1 vCPU, 3.75 GB Memory)
● 1 x Redis Cache cluster with 2 nodes (m3.medium, 1 vCPU, 2.78 GB Memory)
● 3 x Public load balancers (Web and API Servers)
● 2 x Internal load balancers (SB360 and API Servers)
● 2 x Internal load balancers (ElasticSearch Servers)

25. Load
SB360:
● Web: ~1200 rpm
● Non-Web: ~1200 rpm
● DB: ~3000 rpm
● Redis: ~1100 rpm
API:
● API: ~1300 rpm
● Non-Web: ~8000 rpm
● DB: ~17000 rpm

26. What has changed?
• Communication with „Cloud“:
- Active Directory
- VPN
• NAT Server:
- Outbound traffic goes through NAT server
- Only two public IPs
• Web servers
- 3 couples of web servers (2 x Web, 2 x API, 2 x Everything else)
- DFS (IIS configuration is synchronized between servers)

27. What has changed?
• SQL Server:
- EC2 instead of SQL as a Service (RDS)
- Web edition
- Backups to S3 (Bucket lifecycle rules)
- Accessible only from VPC
• Redis Cache
- No built-in security
- Accessible only from VPC
- Security controlled via Security Groups

28. What has changed?
• ElasticSearch:
- 5 Linux servers (2 x logs, 3 x search)
- Isolated Staging and Production environments
- Accessible only from VPC
- Backup strategy: snapshots
- Heap memory size configuration
• Reverse engineering VPC:
- CloudFormation
- Separate Linux EC2 instance to “script” VPC (shutdown)

29. Deployments
• CCNET to Team City
• Single build for all solutions: ~23 mins vs > 60 mins
• Deployment packages are stored on S3
• Lambda service triggers CodeDeploy service
• CodeDeploy deploys packages to EC2 instances one at a time
• Configuration may vary:
- One at a time
- Half at a time
- All at once

30. Migration day
• Source code and configuration:
In advance
• Web sites stopped using app_offline.htm
• Database migration:
From Azure SQL using .bacpac
From Azure VM using .bak
• ElasticSearch:
Using snapshot / restore functionality

31. Migration day issues
• App_Offline.htm design (Arnold Schwarzenegger vs. Pretty lady)
• What pizzas to order for lunch?
• Minor issues (outdated configuration, third party components security)
• Main functionality restored in 2 hours 30 mins
• Regarding to boss – it was the most complicated and smoothest migration in
his career!

32. 5 months since migration…
• 0 night calls
• 100% uptime
• Average response time is stable on peak time
• $11k/month → $6k/month

33. Questions?