1. Atlassian delivered radical architectural changes by migrating 60,000 virtual Linux containers and 120,000 application instances to a microservices architecture on their platform without major incidents. 2. They achieved this through careful planning including gradual rollouts, shadowing new services, and making data migrations idempotent. 3. End-to-end ownership of the changes by dedicated teams also helped ensure flexibility and minimized incidents during the migration.

1. How to Deliver
Radical Architectural Change
SIDNEY SHEK • ARCHITECT •
ATLASSIAN • @SIDNEYSHEK
…without the customer noticing!
ROBIN FERNANDES • TEAM LEAD •
ATLASSIAN • @REWBS

2. Source: BMW S1000 RR. Dinner for RR. www.youtube.com/watch?v=-cM9S2AzU28

3. 22TB attachments moved
65m objects
Zero major incidents
Done during business hours

4. 60,000 +Virtualised Linux containers

5. 120,000 +Monolithic application
instances

7. A solid platform for all
BlobStore

8. Why
microservices?

9. Scale and Agility

10. Getting the
Architecture Right

11. Replicas everywhere

12. Logging and
metrics
Seamless
deployment
Resource
management
Monitoring and
alerts
+
service
descriptor
.json }{service artifact
{
...
scaling: {
min: 5
}
loadBalancer: {
timeout: 600
}
...
}
Heavy lifting by Atlassian PaaS

13. Pure Functional
Architecture

15. BlobStore
server
S3 bucket
Uploading to BlobStore

16. Stored at b10b
(SHA-1 of content)
PUT att-1
BlobStore
server
S3 bucket
Uploading to BlobStore

17. Stored at b10b
(SHA-1 of content)
PUT att-1
ETAG b10b
BlobStore
server
S3 bucket
Uploading to BlobStore

18. Dynamo table
Stored at b10b
(SHA-1 of content)
PUT att-1
ETAG b10b
BlobStore
server
S3 bucket
Logical Key Sequence Operation
att-1 1 Insert b10b
Uploading to BlobStore

19. Dynamo table
Stored at b10b
(SHA-1 of content)
BlobStore
server
S3 bucket
Logical Key Sequence Operation
att-1 1 Insert b10b
Deleting from BlobStore
DELETE att-1
ETAG b10b

20. Dynamo table
Stored at b10b
(SHA-1 of content)
BlobStore
server
S3 bucket
Logical Key Sequence Operation
att-1 1 Insert b10b
att-1 2 Delete
Deleting from BlobStore
DELETE att-1
ETAG b10b
204 “OK”

21. Dynamo table
Stored at b10b
(SHA-1 of content)
BlobStore
server
S3 bucket
Logical Key Sequence Operation
att-1 1 Insert b10b
att-1 2 Delete
Deleting from BlobStore
DELETE att-1
ETAG b10b
204 “OK”

22. Dynamo table
Stored at b10b
(SHA-1 of content)
BlobStore
server
S3 bucket
Logical Key Sequence Operation
att-1 1 Insert b10b
att-1 2 Delete
att-1 3 Insert b10b
Restoring in BlobStore
DELETE att-1
ETAG b10b

23. Building the
Software Right

24. Strongly typed
functional
progamming

25. Use proper types.
For everything.

26. Proper types for
property-based
testing

27. ‘Example’ based unit test = 1 manually written test case
Property-based test = 100s auto-generated examples

28. Advanced types to
simplify testing of
non-blocking code

29. Production code runs asynchronously
new S3Storage[Future](bucketName)

30. Test code runs synchronously
new S3Storage[Id](bucketName)

31. Confidence
through continuous
deployment

32. Merge to master
in Stash

33. Merge to master
in Stash

34. Merge to master
in Stash
Automated staged
deployments
to prod < 2 hours

35. Smooth
Roll-Out
&
Data Migration
© Mathieu Jarry

36. Estimate Up Front

37. Shadow First

38. Gradually Increase the Criticality of the New Service
Disk BlobStore
1
Disk BlobStore
2
Disk BlobStore
3
JIRA JIRA JIRA

39. Roll out to Cohorts

40. Rolling Out Progressively
Data Centre 1
AWS
Customer instance in
OpenVZ container
(approx. 1500 per rack)
Rack 1
…
Rack N
…
…
…
Data Centre 2
Rack 1
…
Rack N
…
…
…
Data Centre 3
Rack 1
…
Rack N
…
…
…

41. Step 1: A Manageable Number of Instances
Data Centre 1
AWS
Rack 1
…
Rack N
…
…
…
Data Centre 2
Rack 1
…
Rack N
…
…
…
Data Centre 3
Rack 1
…
Rack N
…
…
…

42. Step 2: Full Racks
Data Centre 1
AWS
Rack 1
…
Rack N
…
…
…
Data Centre 2
Rack 1
…
Rack N
…
…
…
Data Centre 3
Rack 1
…
Rack N
…
…
…… …

43. Step 3: All Instances
Data Centre 1
AWS
Rack 1
…
Rack N…
Data Centre 2
Rack 1 Rack N…
Data Centre 3
Rack 1 Rack N…
… … … … … … … …

44. Repeat For Next Mode!
Data Centre 1
AWS
Rack 1
…
Rack N…
Data Centre 2
Rack 1 Rack N…
Data Centre 3
Rack 1 Rack N…
… … … … … … … …

45. Repeat For Next Mode!
Data Centre 1
AWS
Rack 1
…
Rack N…
Data Centre 2
Rack 1 Rack N…
Data Centre 3
Rack 1 Rack N…
… … … … … … … …… …

46. Repeat For Next Mode!
Data Centre 1
AWS
Rack 1
…
Rack N…
Data Centre 2
Rack 1 Rack N…
Data Centre 3
Rack 1 Rack N…
… …… ……………… …

47. Know Your
Monitoring

48. Have an “Off”
Switch

49. Make your data migration
idempotent.

50. Organisation
&
Ownership

51. Micros (PaaS)
DevOps, Atlassian Style
One team owns the change, end-to-end.
JIRA
JIRA Attachment Manager
BlobStore Plugin
BlobStore
Client
BlobStore
Server
BlobStore Migration Plugin
Support
Scripts
Monitoring
Tooling
Roll-out
Tooling
Roll-out
Process
Consumer Service
Operational
Tooling
Processes
Monitoring
Process
Incident
Response
Owned by the
BlobStore team
Data
Deletion

52. 1. A microservices architecture requires investment
in infrastructure.
2. Safety needs to be built into the architecture
and software from the onset
3. Plan for gradual roll-outs & data migrations
4. End-to-end ownership means architectural
flexlibility and fewer / less severe incidents
5. Don’t underestimate the integration points –
technical and people.
Key Takeaways

53. aws-scala lib:
http://bitbucket.org/atlassian/aws-scala
event sourcing lib:
http://bitbucket.org/atlassianlabs/eventsrc

54. How to deliver radical architectural change
without the customer noticing
Submit your feedback:
go.atlassian.com/acarchitectural