The document provides slides from a presentation given by Joe Sondow of Netflix about how Netflix uses Grails and AWS to deploy its services. Key points include: - Netflix uses an internal tool called Asgard to manage deployments to AWS. Asgard was open sourced in 2012. - Netflix services are deployed as microservices across many AWS accounts and regions. Asgard helps manage this complex cloud infrastructure. - Custom domain models and conventions are used instead of databases to represent cloud resources and their relationships, avoiding inconsistencies. - Techniques like blue/green deployments, auto-rollback, and caching of cloud metadata help Netflix deploy changes safely at scale across its global infrastructure.

1. // Feed your brain GR8Conf US 2012
gr8.technologies.each{ July 30-31
yourBrain << it Minneapolis, MN, US
}
The Grails App that Deploys Netflix to the Cloud
Extended Edition Slides
Joe Sondow, Netflix
@joesondow #gr8conf

2. Slides online
http://slideshare.net/joesondow
@joesondow #gr8conf

3. Who am I?
@joesondow #gr8conf

4. Who am I?
Still time to sneak out to another session
@joesondow #gr8conf

5. Who am I?
@joesondow #gr8conf

6. Who am I?
Joe Sondow
@joesondow #gr8conf

7. Who am I?
Joe Sondow
Netflix since 2010
@joesondow #gr8conf

8. Who am I?
Joe Sondow
Netflix since 2010
Asgard lead
@joesondow #gr8conf

9. Who am I?
Joe Sondow
Netflix since 2010
Asgard lead
Grails
@joesondow #gr8conf

10. Who am I?
Joe Sondow
Netflix since 2010
Asgard lead
Grails
jQuery
@joesondow #gr8conf

11. Why am I here?
@joesondow #gr8conf

12. Why am I here?

13. Why am I here?
Sell you something

14. Why am I here?
Sell you something
Discuss business plans

15. Why am I here?
Sell you something
Discuss business plans
Answer technical questions

16. Why am I here?
Sell you something
Discuss business plans
Answer technical questions
Be a smaller fish in AWS

17. Why am I here?
Sell you something
Discuss business plans
Answer technical questions
Be a smaller fish in AWS
Give back to community

18. Why am I here?
Sell you something
Discuss business plans
Answer technical questions
Be a smaller fish in AWS
Give back to community
Steal your engineers

19. Asgard

20. Asgard

21. Asgard
Screen shots

22. Asgard
Application list

23. Asgard
Auto Scaling Group list

24. Asgard
Cluster deployment, ready for fast rollback

25. Asgard

26. Asgard
Application deployment

27. Asgard
Application deployment
Cloud management

28. Asgard
Application deployment
Cloud management
Started 2010

29. Asgard
Application deployment
Cloud management
Started 2010
Open source June 2012

30. Asgard
Application deployment
Cloud management
Started 2010
Open source June 2012
http://netflix.github.com

31. Asgard
Application deployment
Cloud management
Started 2010
Open source June 2012
http://netflix.github.com
100’s of Jira tickets

32. Asgard
Application deployment
Cloud management
Started 2010
Open source June 2012
http://netflix.github.com
100’s of Jira tickets
Actively developed

33. User forum
https://groups.google.com/group/AsgardUsers

34. Source code and download
https://github.com/Netflix/asgard

35. Joe Sondow Clay McCoy Jason Gritman
@joesondow @claymccoy @jgritman
Asgard Team
July 2012

37. With more than 25 million streaming members in the
United States, Canada, Latin America, the United
Kingdom and Ireland, Netflix, Inc. (NASDAQ: NFLX) is
the world's leading internet subscription service for
enjoying movies and TV programs.
(from http://ir.netflix.com)

38. Freedom and Responsibility

39. Freedom and Responsibility
Corporate culture and the Cloud

40. Freedom and Responsibility

41. Freedom and Responsibility
Cloud SOA

42. Freedom and Responsibility
Cloud SOA
100’s of services

43. Freedom and Responsibility
Cloud SOA
100’s of services
Small teams

44. Freedom and Responsibility
Cloud SOA
100’s of services
Small teams
Independent releases

45. Freedom and Responsibility
Cloud SOA
100’s of services
Small teams
Independent releases
Controlled chaos

46. Regional URLs

47. Regional URLs
Let’s see some code

48. Regional URLs
Cloud object URLs were unique, conventional, intuitive
Default Grails URL mapping is excellent

49. Regional URLs
Later, Netflix needed to use multiple Amazon regions

50. Regional URLs
Region is another URL dimension
Add URL mapping, but don’t break existing links

52. Regional URLs
RegionService.groovy

53. Regional URLs
RegionService.groovy

54. Regional URLs
RegionService.groovy
Grails may soon provide a better hook

55. Regional URLs
Some controllers should never be regional
RegionService detects RegionAgnostic annotation

56. Cloud deployment model

57. Cloud deployment model
Applications and Clusters

58. Cloud deployment model

59. Cloud deployment model
Auto Scaling
Group

60. Cloud deployment model
Auto Scaling
Group
Launch
Configuration

61. Cloud deployment model
Elastic Load
Balancer
Auto Scaling
Group
Launch
Configuration

62. Cloud deployment model
Elastic Load
Balancer
Auto Scaling
Group
Launch
Configuration
Amazon Machine
Image

63. Cloud deployment model
Elastic Load
Balancer
Auto Scaling
Group
Security
Group
Launch
Configuration
Amazon Machine
Image

64. Cloud deployment model
Elastic Load
Balancer
Auto Scaling
Group
Security Instances
Group
Launch
Configuration
Amazon Machine
Image

65. Cloud deployment model
Elastic Load
Balancer
Auto Scaling
Group
Security Instances
Group
Launch
Configuration
Amazon Machine
Image

66. Cloud deployment model
Elastic Load
Balancer
Auto Scaling
Group
Security Instances
Group
Launch
Configuration
Amazon Machine
Image

67. Cloud deployment model
Elastic Load
Balancer
Auto Scaling
Group
Security Instances
Group
Launch
Configuration
Amazon Machine
Image

68. Cloud deployment model

69. Cloud deployment model
Search

70. Cloud deployment model
API
Search

71. Cloud deployment model
Ratings
API
Search

72. Cloud deployment model
Streaming Starts
Ratings
API
Search

73. Cloud deployment model
Streaming Starts
Ratings
Autocomplete API
Search

74. Cloud deployment model
Sign Up
Streaming Starts
Ratings
Autocomplete API
Search

75. Cloud deployment model
Sign Up
Streaming Starts
Ratings
Application
Application
Application
Autocomplete API
Search
Application Application
Application

76. Inventing the Application

77. Inventing the Application
Problem:
Application is not an Amazon concept

78. Inventing the Application
Problem:
Application is not an Amazon concept
Solution:
Create an Application domain in SimpleDB
Enforce naming conventions on Amazon objects

79. Fast Rollback

80. Fast Rollback
Optimism causes outages

81. Fast Rollback
Optimism causes outages
Production traffic is unique

82. Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running

83. Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Switch traffic to new version

84. Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Switch traffic to new version
Monitor results

85. Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Switch traffic to new version
Monitor results
Revert traffic quickly

86. Fast Rollback

87. Fast Rollback
api-frontend
api-usprod-v007

88. Fast Rollback
api-frontend
api-usprod-v007 api-usprod-v008

89. Fast Rollback
api-frontend
api-usprod-v007 api-usprod-v008

90. Fast Rollback
api-frontend
api-usprod-v007 api-usprod-v008

91. Fast Rollback
api-frontend
api-usprod-v007 api-usprod-v008

92. Fast Rollback
api-frontend
api-usprod-v007

93. Inventing the Cluster

94. Inventing the Cluster
Problem:
Two ASGs with one function but different names

95. Inventing the Cluster
Problem:
Two ASGs with one function but different names
Solution:
Append version number in reserved format
Parse ASG name to determine long-term “cluster”

96. Inventing the Cluster
Instead of keeping a database in sync, use naming
conventions to store the source in truth in Amazon’s API
api Application
api-usprod Cluster
api-usprod-v007 Auto Scaling Group
api-usprod-v008 Auto Scaling Group

97. Database Aversion

98. Database Aversion
Storing metadata on cloud objects

99. Database Aversion

100. Database Aversion
Naming conventions

101. Database Aversion
Naming conventions
Tagging conventions

102. Database Aversion
Naming conventions
Tagging conventions
No GORM domain objects

103. Database Aversion
Naming conventions
Tagging conventions
No GORM domain objects
AWS Java SDK

104. Database Aversion
Naming conventions
Tagging conventions
No GORM domain objects
AWS Java SDK
Less to go out of sync

105. Database Aversion
Naming conventions
Tagging conventions
No GORM domain objects
AWS Java SDK
Less to go out of sync
Shared source of truth

106. Open Source, Closed Config

107. Open Source, Closed Config
Pull company-specific details out of Asgard

108. Open Source, Closed Config

109. Open Source, Closed Config

110. Open Source, Closed Config
Asgard for
Netflix is
configured to
use company-
specific
extension
points such
as standard
utility links
for instances

111. Open Source, Closed Config
Out-of-the-box Asgard installation
has no instance utility links

112. Open Source, Closed Config
Netflix specific $ASGARD_HOME/Config.groovy
link {
// Avoid GStrings here because these Strings are stored dynamic templates for arbitrary server names.
String logUrlStart = 'http://${server}:7777'
String configUrlStart = 'http://${server}:9999/AdminConfig'
instanceLinkGroupingsToLinkTemplateLists = [
'Logs': [
new TextLinkTemplate(logUrlStart + '/Admin/list?view=tomcat/catalina.out', 'catalina.out'),
new TextLinkTemplate(logUrlStart + '/Admin/list', 'Log File Archive'),
new TextLinkTemplate(logUrlStart + '/Admin/threaddumps', 'Thread Dumps'),
new TextLinkTemplate(logUrlStart + '/AdminProxy', 'Admin Proxy Info'),
new TextLinkTemplate(logUrlStart + '/AdminStatus', 'Admin Proxy Status'),
new TextLinkTemplate(logUrlStart + '/GC/index', 'GC Visualization')
],
'Netflix Configuration': [
new TextLinkTemplate(configUrlStart + '/prop.html', 'NetflixConfiguration Properties Console'),
new TextLinkTemplate(configUrlStart + '/libs.html', 'Libraries Console'),
new TextLinkTemplate(configUrlStart + '/machineProps', 'Machine Readable Properties'),
new TextLinkTemplate(configUrlStart + '/webapp/META-INF/MANIFEST.MF', 'Manifest'),
]
]
}

113. Open Source, Closed Config
grails-app/conf/Config.groovy
references external configuration file
~/.asgard/Config.groovy
https://github.com/Netflix/asgard/blob/master/grails-app/conf/Config.groovy
asgardHome = System.getenv('ASGARD_HOME') ?: System.getProperty('ASGARD_HOME') ?:
"${System.getProperty('user.home')}/.asgard"
// Locations to search for config files that get merged into the main config.
// Config files can either be Java properties files or ConfigSlurper scripts.
grails.config.locations = [
"file:${asgardHome}/Config.groovy",
'classpath:sourceVersion.properties'
]

114. Open Source, Closed Config
External Config.groovy also hold the AWS account
credentials, or references for finding them.
grails {
awsAccounts=["178123456789"]
awsAccountNames=["178123456789":"prod"]
}
secret {
accessId="AKIAILAND0E3TMSJA8BA"
secretKey="Od0AMY/dEC3P0/wZQjpmRtm2MVZc9330epDk+pwm"
}
cloud {
accountName="prod"
publicResourceAccounts=["amazon"]
}

115. Caching the Cloud

116. Caching the Cloud
Responsive, massive, multi-region metadata

117. Caching the Cloud

118. Caching the Cloud
Large counts

119. Caching the Cloud
Large counts
Many types

120. Caching the Cloud
Large counts
Many types
Complex relationships

121. Caching the Cloud
Large counts
Many types
Complex relationships
Multiple regions

122. Caching the Cloud
Large counts
Many types
Complex relationships
Multiple regions
Consistent single objects

123. Caching the Cloud
Large counts
Many types
Complex relationships
Multiple regions
Consistent single objects
Eventually consistent lists

124. Caching the Cloud
class Caches {
final CachedMap<AppRegistration> allApplications
final CachedMap<ApplicationMetrics> allApplicationMetrics
final CachedMap<HardwareProfile> allHardwareProfiles
final MultiRegionCachedMap<MetricAlarm> allAlarms
final MultiRegionCachedMap<ApplicationInstance> allApplicationInstances
final MultiRegionCachedMap<AutoScalingGroup> allAutoScalingGroups
final MultiRegionCachedMap<AvailabilityZone> allAvailabilityZones
final MultiRegionCachedMap<Cluster> allClusters
final MultiRegionCachedMap<DBInstance> allDBInstances
final MultiRegionCachedMap<DBSecurityGroup> allDBSecurityGroups
final MultiRegionCachedMap<DBSnapshot> allDBSnapshots
final MultiRegionCachedMap<String> allDomains
final MultiRegionCachedMap<FastProperty> allFastProperties
final MultiRegionCachedMap<Image> allImages
final MultiRegionCachedMap<Instance> allInstances
final MultiRegionCachedMap<InstanceTypeData> allInstanceTypes
final MultiRegionCachedMap<KeyPairInfo> allKeyPairs
final MultiRegionCachedMap<LaunchConfiguration> allLaunchConfigurations
// etc.
// etc.
// etc.
}

125. Caching the Cloud
class Caches {
final CachedMap<AppRegistration> allApplications
final CachedMap<ApplicationMetrics> allApplicationMetrics
final CachedMap<HardwareProfile> allHardwareProfiles
final MultiRegionCachedMap<MetricAlarm> allAlarms
final MultiRegionCachedMap<ApplicationInstance> allApplicationInstances
final MultiRegionCachedMap<AutoScalingGroup> allAutoScalingGroups
final MultiRegionCachedMap<AvailabilityZone> allAvailabilityZones
final MultiRegionCachedMap<Cluster> allClusters
final MultiRegionCachedMap<DBInstance> allDBInstances
final MultiRegionCachedMap<DBSecurityGroup> allDBSecurityGroups
final MultiRegionCachedMap<DBSnapshot> allDBSnapshots
final MultiRegionCachedMap<String> allDomains
final MultiRegionCachedMap<FastProperty> allFastProperties
final MultiRegionCachedMap<Image> allImages
final MultiRegionCachedMap<Instance> allInstances
final MultiRegionCachedMap<InstanceTypeData> allInstanceTypes
final MultiRegionCachedMap<KeyPairInfo> allKeyPairs
final MultiRegionCachedMap<LaunchConfiguration> allLaunchConfigurations
// etc.
// etc.
// etc.
}

126. Caching the Cloud
class AwsRdsService implements CacheInitializer, InitializingBean {
MultiRegionAwsClient<AmazonRDS> awsClient
Caches caches
void initializeCaches() {
caches.allDBInstances.ensureSetUp({ Region region -> retrieveDBInstances(region) })
}
private List<DBInstance> retrieveDBInstances(Region region) {
awsClient.by(region).describeDBInstances(new DescribeDBInstancesRequest()).getDBInstances()
}
Collection<DBInstance> getDBInstances(UserContext userContext) {
caches.allDBInstances.by(userContext.region).list()
}
}

127. Caching the Cloud
class AwsRdsService implements CacheInitializer, InitializingBean {
MultiRegionAwsClient<AmazonRDS> awsClient
Caches caches
void initializeCaches() {
caches.allDBInstances.ensureSetUp({ Region region -> retrieveDBInstances(region) })
}
private List<DBInstance> retrieveDBInstances(Region region) {
awsClient.by(region).describeDBInstances(new DescribeDBInstancesRequest()).getDBInstances()
}
Collection<DBInstance> getDBInstances(UserContext userContext) {
caches.allDBInstances.by(userContext.region).list()
}
}

128. Caching the Cloud
class AwsRdsService implements CacheInitializer, InitializingBean {
MultiRegionAwsClient<AmazonRDS> awsClient
Caches caches
void initializeCaches() {
caches.allDBInstances.ensureSetUp({ Region region -> retrieveDBInstances(region) })
}
private List<DBInstance> retrieveDBInstances(Region region) {
awsClient.by(region).describeDBInstances(new DescribeDBInstancesRequest()).getDBInstances()
}
Collection<DBInstance> getDBInstances(UserContext userContext) {
caches.allDBInstances.by(userContext.region).list()
}
}

129. Caching the Cloud
class AwsRdsService implements CacheInitializer, InitializingBean {
MultiRegionAwsClient<AmazonRDS> awsClient
Caches caches
void initializeCaches() {
caches.allDBInstances.ensureSetUp({ Region region -> retrieveDBInstances(region) })
}
private List<DBInstance> retrieveDBInstances(Region region) {
awsClient.by(region).describeDBInstances(new DescribeDBInstancesRequest()).getDBInstances()
}
Collection<DBInstance> getDBInstances(UserContext userContext) {
caches.allDBInstances.by(userContext.region).list()
}
}

130. Caching the Cloud

131. Visual Language for the Cloud

132. Visual Language for the Cloud
Tango open source icons

133. Visual Language for the Cloud

134. Visual Language for the Cloud
AWS is intimidating

135. Visual Language for the Cloud
AWS is intimidating
Many object types

136. Visual Language for the Cloud
AWS is intimidating
Many object types
Help newbie users

137. Visual Language for the Cloud
AWS is intimidating
Many object types
Help newbie users
Reduce cognitive load

138. Visual Language for the Cloud
AWS is intimidating
Many object types
Help newbie users
Reduce cognitive load
Make it easy

139. Visual Language for the Cloud
AWS is intimidating
Many object types
Help newbie users
Reduce cognitive load
Make it easy
Avoid surprises

140. Visual Language for the Cloud

141. Visual Language for the Cloud
At a glance, these nav bar items look alike.

142. Visual Language for the Cloud
At a glance, these nav bar items look alike.

143. Visual Language for the Cloud

144. Visual Language for the Cloud
Some screens have multiple action
buttons that look too similar.

145. Visual Language for the Cloud
Some screens have multiple action
buttons that look too similar.

146. Visual Language for the Cloud

147. Visual Language for the Cloud
Because of naming conventions,
these links look alike.

148. Visual Language for the Cloud
Because of naming conventions,
these links look alike.

149. Visual Language for the Cloud

150. Visual Language for the Cloud
The indicators for the
current AWS region are
too easy to miss.

151. Visual Language for the Cloud
The indicators for the
current AWS region are
too easy to miss.

152. Visual Language for the Cloud
These availability zones
are important to
recognize at a glance
but their names look
similar, and they appear
on many screens.

153. Visual Language for the Cloud
These availability zones
are important to
recognize at a glance
but their names look
similar, and they appear
on many screens.

154. Visual Language for the Cloud

155. Tango Icons

156. Tango Icons
http://tango.freedesktop.org/

157. Tango Icons
http://tango.freedesktop.org/
http://tango.freedesktop.org/Tango_Icon_Theme_Guidelines

158. Tango Icons
http://tango.freedesktop.org/
http://tango.freedesktop.org/Tango_Icon_Theme_Guidelines
http://commons.wikimedia.org/wiki/Tango_icons

159. Tango Icons
http://tango.freedesktop.org/
http://tango.freedesktop.org/Tango_Icon_Theme_Guidelines
http://commons.wikimedia.org/wiki/Tango_icons
Used by Firefox, Jenkins, GIMP, OpenOffice, VMWare

160. REST API in Grails

161. REST API in Grails
Enable external mashups with cloud data

162. REST API in Grails

163. REST API in Grails

164. REST API in Grails

165. REST API in Grails

166. REST API in Grails

167. REST API in Grails

168. REST API in Grails

169. REST API in Grails

170. REST API in Grails

171. REST API in Grails
ApplicationController.groovy

172. Offline Development

173. Offline Development
Makes on a plane

174. Offline Development
Mock data
Mock behavior
System property switch offline=true

175. Mock Data
http://asgardprod/us-east-1/autoScaling/list.json

176. Mock Data
Parse JSON

177. Mock Behavior
Override Amazon Java client methods

178. System Property
grails run-app -Doffline=true

179. Why not the AWS console?

180. Why not the AWS console?
No, not to keep me busy

181. Why not the AWS console?

182. Why not the AWS console?

183. Why not the AWS console?
Hide keys

184. Why not the AWS console?
Hide keys
Customize model

185. Why not the AWS console?
Hide keys
Customize model
Enforce conventions

186. Why not the AWS console?
Hide keys
Customize model
Enforce conventions
Automate workflow

187. Why not the AWS console?
Hide keys
Customize model
Enforce conventions
Automate workflow
Log changes

188. Why not the AWS console?
Hide keys
Customize model
Enforce conventions
Automate workflow
Log changes
Integrate systems

189. Why not the AWS console?
Hide keys
Customize model
Enforce conventions
Automate workflow
Log changes
Integrate systems
Create REST API

190. @NetflixOSS

191. @NetflixOSS
http://techblog.netflix.com

192. @NetflixOSS
http://techblog.netflix.com
http://netflix.github.com

193. http://github.com/Netflix/asgard
Thank you

194. http://github.com/Netflix/asgard
Thank you
Questions?
github.com/joesondow
@joesondow
slideshare.net/joesondow
linkedin.com/in/joesondow