Looking to the cloud to serve Bitbucket (formerly Stash) to a horde of hungry developers? Netflix has the recipe for you! Take one part fresh open source platforms, two parts juicy deployment pipelines, and add a dash of gooey-rich custom bakes. Join this session to find out how Netflix used this unique combination of technologies to move Stash from their data center to the cloud and what they burned (yes, burned) along the way.

1. ED BUKOSKI • SENIOR SOFTWARE ENGINEER • NETFLIX • @EBUKOSKI
Bitbucket in the AWS
Cloud At Netflix

2. • Big cool statistic
• 2,56
9
• Add-Ons in Marketplace

3. • Big cool statistic
• 2,56
9
• Add-Ons in Marketplace

4. • Big cool statistic
• 2,56
9
• Add-Ons in Marketplace

5. • Big cool statistic
• 2,56
9
• Add-Ons in Marketplace

6. • Big cool statistic
• 2,56
9
• Add-Ons in Marketplace

7. BITBUCKET IN AWS
LESSONS LEARNED
Q&A
Agenda

8. Pre-Cloud
Traditional DC
N-Tier Web Applications
Relational Databases
Cloud
Cloud infrastructure - AWS
Java, Linux, Apache
No SQLHigh Availability
Scale
Performance
Bitbucket At Netflix

9. Bitbucket At Netflix
Total LOC
1,610,528,526
Total Commits
5,554,379
Builds / Day
7,132
Jenkins Slaves
335
Developers
950

10. Bitbucket in AWS

11. Cloud Infrastructure
Metrics, Monitoring, Backups
Load Testing
Bitbucket in AWS

12. Cloud Infrastructure
Bitbucket in AWS

13. Cloud Infrastructure
Amazon Machine Image (AMI)
Elastic Compute Cloud (EC2) Instance
Elastic Block Storage Volume (EBS)
Relational Database (RDS)
Elastic Load Balancer (ELB)

14. Cloud Infrastructure
1. Stock Atlassian AMI
nginx
Stash
PostgreSQL
setups & configs
Amazon Machine Image (AMI)

15. 2. Custom AMI
Cloud Infrastructure
Apache/Tomcat
How to Make?
Organizational
Amazon Machine Image (AMI)
Metrics Sidecars

16. Cloud Infrastructure
EC2 Instance
c3.8xlarge, 32 vcpu, 60 GB RAM
2x320 SSD ephermal

17. Cloud Infrastructure
EC2 Instance
c3.8xlarge, 32 vcpu, 60 GB RAM
2x320 SSD ephermal
EBS Volume
General Purpose GP2, 1 TB
Stash Home

18. Cloud Infrastructure
EC2 Instance
c3.8xlarge, 32 vcpu, 60 GB RAM
2x320 SSD ephermal
EBS Volume
General Purpose GP2, 1 TB
Stash Home
RDS
db.m3.xlarge, 4 vcpu, 16 GB RAM
100 GB storage

19. Cloud Infrastructure
EC2 Instance
c3.8xlarge, 32 vcpu, 60 GB RAM
2x320 SSD ephermal
EBS Volume
General Purpose GP2, 1 TB
Stash Home
RDS
db.m3.xlarge, 4 vcpu, 16 GB RAM
100 GB storage
ELB
DNS

20. Cloud Infrastructure
Auto Scaling Group?
EC2 Instance
c3.8xlarge, 32 vcpu, 60 GB RAM
2x320 SSD ephermal
EBS Volume
General Purpose GP2, 1 TB
Stash Home
RDS
db.m3.xlarge, 4 vcpu, 16 GB RAM
100 GB storage
ELB
DNS

21. Cloud Infrastructure
Bitbucket in AWS
Metrics, Monitoring, Backups

22. Metrics, Monitoring, Backups
2. Custom AMI
Amazon Machine Image (AMI)
Metrics
Sidecars

23. CPU, memory, sessions, JDBC
Completely stand-alone WAR
Built-in charting
Easy, drop in metrics
Easy, drop in metrics
Completely stand-alone WAR
System (CPU, etc), http sessions
Built-in charting

24. Prana
Sidecar “platform”
Standalone JVM
Ship and index log files
Visualize with Kibana
Resp times, error rates
Logstash,Elasticsearch,
Kibana

25. Bitbucket DIY Backup Scripts
1. EBS Snapshots
2. Database backups*
RDS Database
*EBS Instance only
custom scripting
maint mode — short < 30s
Metrics, Monitoring, Backups

27. Cloud Infrastructure
Metrics, Monitoring, Backups
Bitbucket in AWS
Load Testing

28. Load Testing
load testing goals
client side measurements
1. Load testing scripts

30. 2. Bake an AMI
Aminator
3. Create Launch Configs and ASG
4. Scale up ASG to generate load
Load Testing

31. Scale up ASG

32. Server
Metrics

33. Click here to add page title
Client
Metrics

34. Cloud Infrastructure
Metrics, Monitoring, Backups
Load Testing
Bitbucket in AWS

35. Lessons Learned

36. Lessons Learned
RDS Manual Snapshot Limit
max 50 snapshots
snapshot error -> kept stash in maint mode

37. Lessons Learned
Janitor cleans unused
AWS infrastructure
new rule to clean old
RDS snapshots
RDS Manual Snapshot Limit
backup script more resilient to errors
Janitor Monkey

38. Volume mount dupe disaster
snapshot, attach to new instance
two Stashs connected to same database
prod
test Stash migrated Stash prod database
tables mismatched with code
prod immediately failed hard
Lessons Learned
3.5
context: populate test with prod data
test
3.8
connected to prod database
started test instance (prod configs!)

39. 3.5ish Volume mount dupe disaster
prod
Lessons Learned
3.5
test
3.8
shutdown test
ad-hoc SQL to stabilize database
did not restore database from backups
analyze liquibase code -> roll-back script
upgrade: roll-back then roll-forward

40. Bitbucket in AWS - Takeaways
Embrace cloud infrastructure
Include monitoring and metrics
Learn from our mistakes
External resources
netflix.github.io

41. Thank you!
ED BUKOSKI • SENIOR SOFTWARE ENGINEER • NETFLIX • @EBUKOSKI