The Angry Birds Global Merchandize Shop runs and scales 100% automatically in the AWS Infrastructure and is designed to easily handle 20x traffic peaks during marketing and new release peaks. Learn about the infrastructure and the applied strategies for Continuous Delivery, Autoscaling and Zero Downtime Rollouts with the Amazon Cloud. Speaker: Daniel Pötzinger, CTO AOE Media

1. Freitag, 3. Mai 13

2. Angrybirds fly in the
cloud
Freitag, 3. Mai 13

3. Angrybirds fly in the
cloud
Presented by
Daniel Pötzinger
AWS Summit 2013
Freitag, 3. Mai 13

4. Freitag, 3. Mai 13

5. danpoetz
Freitag, 3. Mai 13

6. Freitag, 3. Mai 13

7. Freitag, 3. Mai 13

8. Freitag, 3. Mai 13

9. 152,2 Million €
68,5 Million € Consumer Products
Freitag, 3. Mai 13

10. Freitag, 3. Mai 13

11. Freitag, 3. Mai 13

12. PC games
Plush toys, shirts, ...
Freitag, 3. Mai 13

13. > 24.000.000 likes > 500.000 followers
Social Media
Impact
Freitag, 3. Mai 13

14. Irregular Traffic Behaviour
Freitag, 3. Mai 13

15. Challenges
Freitag, 3. Mai 13

16. Challenges
Frequent development
and releasing of new
features
Freitag, 3. Mai 13

17. Challenges
Frequent development
and releasing of new
features
Scalable and stable
Operation
Freitag, 3. Mai 13

18. Freitag, 3. Mai 13

19. The solution?
Freitag, 3. Mai 13

20. Freitag, 3. Mai 13

21. Continuous
Deployment
CloudDevOps
Freitag, 3. Mai 13

22. Continuous
Deployment
CloudDevOps
Freitag, 3. Mai 13

23. Continuous
Deployment
CloudDevOps
Freitag, 3. Mai 13

24. Why Amazon Web Services ?
Freitag, 3. Mai 13

25. ✓Scales quickly on demand
✓Pay only what for what
you use
Freitag, 3. Mai 13

26. 8 regions with 2-4 availability zones
A lot of „edge servers“ for content delivery
Freitag, 3. Mai 13

27. Think cleanly about your operation infrastructure
Freitag, 3. Mai 13

28. Setting up a new environment should always
be faster than fixing an existing one
Freitag, 3. Mai 13

29. What you should do
Freitag, 3. Mai 13

30. What you should do
Automate the setup of new servers
Freitag, 3. Mai 13

31. What you should do
Automate the setup of new servers
Automate failover
Freitag, 3. Mai 13

32. What you should do
Automate the setup of new servers
Automate failover
Automate scaling
Freitag, 3. Mai 13

33. What you should do
Automate the setup of new servers
Automate failover
Automate scaling
Don‘t do work that others did better
Freitag, 3. Mai 13

34. Powerfull AWS Services for Angrybirds
Freitag, 3. Mai 13

35. Powerfull AWS Services for Angrybirds
EC2
Freitag, 3. Mai 13

36. Powerfull AWS Services for Angrybirds
S3EC2
Freitag, 3. Mai 13

37. Powerfull AWS Services for Angrybirds
S3EC2 RDS
Freitag, 3. Mai 13

38. Powerfull AWS Services for Angrybirds
CloudFrontS3EC2 RDS
Freitag, 3. Mai 13

39. Powerfull AWS Services for Angrybirds
CloudFront ElastiCacheS3EC2 RDS
Freitag, 3. Mai 13

40. Powerfull AWS Services for Angrybirds
CloudFront ElastiCacheS3EC2 RDS ELB
Freitag, 3. Mai 13

41. Powerfull AWS Services for Angrybirds
CloudFront ElastiCacheS3EC2 RDS ELB R53
Freitag, 3. Mai 13

42. Powerfull AWS Services for Angrybirds
Amazon Web Services & Amazon Console
CloudFront ElastiCacheS3EC2 RDS ELB R53
Freitag, 3. Mai 13

43. Powerfull AWS Services for Angrybirds
Rightscale Services
Amazon Web Services & Amazon Console
CloudFront ElastiCacheS3EC2 RDS ELB R53
Freitag, 3. Mai 13

44. Freitag, 3. Mai 13

45. Freitag, 3. Mai 13

46. Cloud
Architecture
Freitag, 3. Mai 13

47. Freitag, 3. Mai 13

48. Varnish Array
Freitag, 3. Mai 13

49. Varnish Array
EC2 EC2
Freitag, 3. Mai 13

50. Varnish Array
Frontend Array
EC2 EC2 EC2
EC2 EC2
Freitag, 3. Mai 13

51. Varnish Array
Frontend Array Backend Array
EC2 EC2EC2 EC2 EC2
EC2 EC2
Freitag, 3. Mai 13

52. Varnish Array
Frontend Array Backend Array
RDS
EC2 EC2EC2 EC2 EC2
EC2 EC2
Freitag, 3. Mai 13

53. RDS
Varnish Array
Frontend Array Backend Array
RDS
EC2 EC2EC2 EC2 EC2
EC2 EC2
Freitag, 3. Mai 13

54. RDS
Varnish Array
Frontend Array Backend Array
RDS
S3
Assets Storage
EC2 EC2EC2 EC2 EC2
EC2 EC2
Freitag, 3. Mai 13

55. RDS
Varnish Array
Frontend Array Backend Array
RDS
CloudFront
CDN for Assets
S3
Assets Storage
EC2 EC2EC2 EC2 EC2
EC2 EC2
Freitag, 3. Mai 13

56. RDS
Varnish Array
Frontend Array Backend Array Worker Array
RDS
CloudFront
CDN for Assets
S3
Assets Storage
EC2 EC2
EC2
EC2 EC2 EC2
EC2 EC2
Freitag, 3. Mai 13

57. RDS
Varnish Array
Frontend Array Backend Array Worker Array
ELB
RDS
CloudFront
CDN for Assets
S3
Assets Storage
EC2 EC2
EC2
EC2 EC2 EC2
EC2 EC2
Freitag, 3. Mai 13

58. RDS
„Deployment Definition“
+ Autoscaling
Varnish Array
Frontend Array Backend Array Worker Array
ELB
RDS
CloudFront
CDN for Assets
S3
Assets Storage
EC2 EC2
EC2
EC2 EC2 EC2
EC2 EC2
Freitag, 3. Mai 13

59. RDS
„Deployment Definition“
+ Autoscaling
Varnish Array
Frontend Array Backend Array Worker Array
ELB
RDS
CloudFront
CDN for Assets
S3
Assets Storage
EC2 EC2
EC2
EC2 EC2 EC2
EC2 EC2
Monitoring
Freitag, 3. Mai 13

60. RDS
„Deployment Definition“
+ Autoscaling
Varnish Array
Frontend Array Backend Array Worker Array
ELB
RDS
CloudFront
CDN for Assets
S3
Assets Storage
EC2 EC2
EC2
EC2 EC2 EC2
EC2 EC2
MonitoringAPI
Freitag, 3. Mai 13

61. RDS
„Deployment Definition“
+ Autoscaling
Varnish Array
Frontend Array Backend Array Worker Array
CloudFront
CDN for Skin
ELB
RDS
CloudFront
CDN for Assets
S3
Assets Storage
EC2 EC2
EC2
EC2 EC2 EC2
EC2 EC2
MonitoringAPI
Freitag, 3. Mai 13

62. RDS
„Deployment Definition“
+ Autoscaling
Varnish Array
Frontend Array Backend Array Worker Array
CloudFront
CDN for Skin
ELB
R53
DNS
RDS
CloudFront
CDN for Assets
S3
Assets Storage
EC2 EC2
EC2
EC2 EC2 EC2
EC2 EC2
MonitoringAPI
Freitag, 3. Mai 13

63. RDS
„Deployment Definition“
+ Autoscaling
Varnish Array
Frontend Array Backend Array Worker Array
CloudFront
CDN for Skin
ELB
R53
DNS
RDS
ElastiCache
Cache Backend
CloudFront
CDN for Assets
S3
Assets Storage
EC2 EC2
EC2
EC2 EC2 EC2
EC2 EC2
MonitoringAPI
Freitag, 3. Mai 13

64. RDS
„Deployment Definition“
+ Autoscaling
Varnish Array
Frontend Array Backend Array Worker Array
CloudFront
CDN for Skin
ELB
R53
DNS
RDS
S3
Deployment
ElastiCache
Cache Backend
CloudFront
CDN for Assets
S3
Assets Storage
EC2 EC2
EC2
EC2 EC2 EC2
EC2 EC2
MonitoringAPI
Freitag, 3. Mai 13

65. Scaling is about finding the
bottleneck
Freitag, 3. Mai 13

66. CPU
Network
Storage (DB)Traffic
Possible Bottlenecks
Offloading, Optimisation,
Caching, add more CPU (scale)...
Offloading, Optimisation,
Scale Storage UP (H/V),
Replace Storage...
Offloading (CDN, Proxy)
Optimize Size of the Site
Scale Network IS...
Freitag, 3. Mai 13

67. HTTP Caching
Traffic
Freitag, 3. Mai 13

68. HTTP Caching
Application
Traffic
Freitag, 3. Mai 13

69. HTTP Caching
Application
Offload expensive requests
Traffic
Freitag, 3. Mai 13

70. HTTP Caching
Application
Offload expensive requests
Traffic
Browser
Traffic
Freitag, 3. Mai 13

71. HTTP Caching
Application
Offload expensive requests
Traffic
CDN
Browser
Traffic
Freitag, 3. Mai 13

72. HTTP Caching
Application
Varnish
(Reverse Proxy) Traffic
Offload expensive requests
Traffic
CDN
Browser
Traffic
Freitag, 3. Mai 13

73. HTTP Caching
Application
Varnish
(Reverse Proxy) Traffic
Offload expensive requests
Traffic
CDN
Browser
Traffic
Purging
Freitag, 3. Mai 13

74. 10.000+Possible Pages / Seconde
from ELB &Varnish
Freitag, 3. Mai 13

75. Autoscaling
Traffic
Freitag, 3. Mai 13

76. Autoscaling
Traffic
FrontendArray
EC2EC2EC2
Freitag, 3. Mai 13

77. Autoscaling
Traffic
FrontendArray
EC2EC2EC2
Freitag, 3. Mai 13

78. Autoscaling
Traffic
FrontendArray
EC2EC2EC2
EC2EC2EC2
Freitag, 3. Mai 13

79. Autoscaling
•Groups with policys
•Min, Max
•Votingrules for
scaling up & down
•Replace broken
instances
Traffic
FrontendArray
EC2EC2EC2
EC2EC2EC2
Freitag, 3. Mai 13

80. 10.000possible Orders / Hour
Freitag, 3. Mai 13

81. Be prepared for changes
Freitag, 3. Mai 13

82. Freitag, 3. Mai 13

83. Freitag, 3. Mai 13

84. Continuous
Deployment
Freitag, 3. Mai 13

85. Freitag, 3. Mai 13

86. Freitag, 3. Mai 13

87. Don‘t be scared of
releasing to
production
Freitag, 3. Mai 13

88. Late night bugfixing?
Freitag, 3. Mai 13

89. That’s how a deployment should look like
Freitag, 3. Mai 13

90. And you?
Freitag, 3. Mai 13

91. And you?
• Do you use continuous integration?
Freitag, 3. Mai 13

92. And you?
• Do you use continuous integration?
• Do you use automated acceptance tests?
Freitag, 3. Mai 13

93. And you?
• Do you use continuous integration?
• Do you use automated acceptance tests?
• Do you keep them green?
Freitag, 3. Mai 13

94. And you?
• Do you use continuous integration?
• Do you use automated acceptance tests?
• Do you keep them green?
• Do you use continuous deployment?
Freitag, 3. Mai 13

95. Continuous Deployment
Freitag, 3. Mai 13

96. Continuous Deployment
We want on-demand deployment
Freitag, 3. Mai 13

97. Continuous Deployment
We want on-demand deployment
of tested features and fixes
Freitag, 3. Mai 13

98. Continuous Deployment
We want on-demand deployment
of tested features and fixes
in a quick and reliable way
Freitag, 3. Mai 13

99. We reach this by
Freitag, 3. Mai 13

100. Automating everything
We reach this by
Freitag, 3. Mai 13

101. Automating everything
We reach this by
Build
Freitag, 3. Mai 13

102. Automating everything
We reach this by
Build
Testing
Freitag, 3. Mai 13

103. Automating everything
We reach this by
Build
Testing
Deployment
Freitag, 3. Mai 13

104. ..and
Freitag, 3. Mai 13

105. doing it frequently
..and
Freitag, 3. Mai 13

106. doing it frequently
..and
flickr deploys 3
times a day...
Freitag, 3. Mai 13

107. Freitag, 3. Mai 13

108. Freitag, 3. Mai 13

109. The Deployment
Pipeline
Freitag, 3. Mai 13

110. Freitag, 3. Mai 13

111. Freitag, 3. Mai 13

112. Commit
and Build
Build Downstream and Test Strategy
SVN
Backup
Storage
Angry Birds Deployment Pipeline
Feedback to Developers
Freitag, 3. Mai 13

113. Commit
and Build
Build Downstream and Test Strategy
SVN
Backup
Storage
Unit Tests
Angry Birds Deployment Pipeline
Feedback to Developers
Freitag, 3. Mai 13

114. Commit
and Build
Build Downstream and Test Strategy
SVN
Backup
Storage
Unit Tests
Angry Birds Deployment Pipeline
Feedback to Developers
Freitag, 3. Mai 13

115. Commit
and Build
Build Downstream and Test Strategy
SVN
Backup
Storage
Unit Tests
Install on
“latest”
Angry Birds Deployment Pipeline
Feedback to Developers
Freitag, 3. Mai 13

116. Commit
and Build
Build Downstream and Test Strategy
SVN
Backup
Storage
Unit Tests
Install on
“latest”
Angry Birds Deployment Pipeline
Feedback to Developers
Freitag, 3. Mai 13

117. Selenium
Acceptance Tests
Commit
and Build
Build Downstream and Test Strategy
SVN
Backup
Storage
Unit Tests
Install on
“latest”
Angry Birds Deployment Pipeline
Feedback to Developers
Freitag, 3. Mai 13

118. Selenium
Acceptance Tests
Commit
and Build
Build Downstream and Test Strategy
SVN
Backup
Storage
Unit Tests
Install on
“latest”
Angry Birds Deployment Pipeline
Feedback to Developers
Freitag, 3. Mai 13

119. Selenium
Acceptance Tests
Commit
and Build
Build Downstream and Test Strategy
SVN
Backup
Storage
Unit Tests
Install on
“latest”
Performance
Tests
Angry Birds Deployment Pipeline
Feedback to Developers
Freitag, 3. Mai 13

120. Selenium
Acceptance Tests
Commit
and Build
Build Downstream and Test Strategy
SVN
Backup
Storage
Unit Tests
Install on
“latest”
Performance
Tests
Angry Birds Deployment Pipeline
Feedback to Developers
Freitag, 3. Mai 13

121. Angry Birds Deployment Pipeline
Install on
Deployment
System
Copy to S3
Cloud
Deployment
Staging
Integration
Tests
AOE + Rovio
Cloud
deployment
production
Freitag, 3. Mai 13

122. Varnish Array
Autoscaling
Frontend Array
Backend
Array
Worker
Array
Cloud Deployment
ELB
Deployment „Version 88“
Start new release by
✓ cloning old deployment
✓ adjusting release number
RDS
mySQL
S3
V V V
WFE FE FE BE BE BE
R53
DNS
Freitag, 3. Mai 13

123. Varnish Array
Autoscaling
Frontend Array
Backend
Array
Worker
Array
Cloud Deployment
ELB
Deployment „Version 88“
RDS
mySQL
S3
V V V
WFE FE FE BE BE BE
R53
DNS
Freitag, 3. Mai 13

124. Deployment „Version 89“
Varnish Array
Autoscaling
Frontend Array
Backend
Array
Worker
Array
Cloud Deployment
ELB
Deployment „Version 88“
RDS
mySQL
S3
Varnish Array
Autoscaling
Frontend Array
Backend
Array
Worker
Array
V V V
WFE FE FE BE BE BE
R53
DNS
Freitag, 3. Mai 13

125. Deployment „Version 89“
Varnish Array
Autoscaling
Frontend Array
Backend
Array
Worker
Array
Cloud Deployment
ELB
Deployment „Version 88“
RDS
mySQL
S3
Varnish Array
Autoscaling
Frontend Array
Backend
Array
Worker
Array
ELB
W
V V V
WFE FE FE BE BE BE
R53
DNS
Freitag, 3. Mai 13

126. Deployment „Version 89“
Varnish Array
Autoscaling
Frontend Array
Backend
Array
Worker
Array
Cloud Deployment
ELB
Deployment „Version 88“
RDS
mySQL
S3
Varnish Array
Autoscaling
Frontend Array
Backend
Array
Worker
Array
ELB
WFE FE FE BE BE BE
V V VV V V
WFE FE FE BE BE BE
R53
DNS
Freitag, 3. Mai 13

127. Deployment „Version 89“
Varnish Array
Autoscaling
Frontend Array
Backend
Array
Worker
Array
Cloud Deployment
ELB
Deployment „Version 88“
RDS
mySQL
S3
Varnish Array
Autoscaling
Frontend Array
Backend
Array
Worker
Array
ELB
WFE FE FE BE BE BE
V V VV V V
WFE FE FE BE BE BE
R53
DNS
Freitag, 3. Mai 13

128. 0Seconds Downtime
Freitag, 3. Mai 13

129. Freitag, 3. Mai 13

130. Freitag, 3. Mai 13

131. The real life
Freitag, 3. Mai 13

132. Freitag, 3. Mai 13

133. Freitag, 3. Mai 13

134. Day 1 - Release 1
Freitag, 3. Mai 13

135. Day 2 - Release 2 - “Get ready for Space”
Freitag, 3. Mai 13

136. Day 3 - „People want it so badly“
Freitag, 3. Mai 13

137. New release
deployed
Day 3 - „People want it so badly“
Freitag, 3. Mai 13

138. New release
deployed
Space game
launched
Array scales up
Day 3 - „People want it so badly“
Freitag, 3. Mai 13

139. New release
deployed
Space game
launched
Array scales up
USA wakes up
Day 3 - „People want it so badly“
Freitag, 3. Mai 13

140. Day 4 - Release 3 „Performance Improvement“
Freitag, 3. Mai 13

141. 40% average
CPU usage.
1-2 FE Servers are
enough for normal
operations
Day 4 - Release 3 „Performance Improvement“
Freitag, 3. Mai 13

142. Freitag, 3. Mai 13

143. Freitag, 3. Mai 13

144. Thanks!
Follow us on Twitter:
AOE: @aoemedia
Daniel P: @danpoetz
Flickr photo contributions in the slide:
http://www.flickr.com/photos/the-wanderers-eye/4494147652/sizes/l/
http://www.flickr.com/photos/51293088@N05/8195327886/sizes/l/
http://www.flickr.com/photos/k1rsch/3818553037/sizes/l/
Freitag, 3. Mai 13