Amazon.com enables all of its developers to be productive on AWS by operating across tens-of-thousands of team-owned AWS accounts, all while raising the bar on security, visibility and operational control. Amazon has been able to achieve these seemingly conflicting ideals by automating setup and management of these accounts at scale using AWS Management Tools such as CloudFormation, Config, CloudTrail, CloudWatch and EC2 Systems Manager. In this session, discover more about how Amazon.com built ASAP using AWS Management tools, and understand some of the decisions they made as their usage of AWS evolved over time. You will learn about the design, architecture and implementation that Amazon.com went through as part of this effort.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How Amazon.com Uses AWS
Management Tools
M i k e B u r k e , P r i n c i p a l T e c h . P r o g r a m M a n a g e r , A m a z o n . c o m
P r a s h a n t P r a h l a d , S r . M g r P r o d u c t M a n a g e m e n t , A W S
D E V 3 4 0
N o v e m b e r 2 9 , 2 0 1 7
Guardrails, not Gates

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What can you expect from this session
• Management Tools?
• Your governance philosophy first, tools will follow
• Amazon.com’s journey
• Role of management tools in journey
• You too can implement similar mechanisms

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Getting situated in the Cloud
Common questions:
• What AWS Account structure should I use?
• How much governance should I put in place?
• How do I keep up with my company’s AWS usage? (and have a life)
• Do I have adequate metadata for verification/analysis?

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The challenge
- Define
- Discover
- Monitor
- Manage
- Report
- Respond
- Agility
- Innovation
Governance
Developmentspeed

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS enables you to do both
- Define
- Discover
- Monitor
- Manage
- Report
- Respond
- Agility
- Innovation
Governance
Developmentspeed
With AWS you can programmatically:
• Define provisioning and configuration of
resources
• Continuously discover new resources and
changes to existing resources
• Monitor resources and operations for
compliance
• Manage, report on, and respond to
changes to your resources

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
 CloudFormation
 Service Catalog
 CloudTrail
 Config
 CloudWatch
 Trusted Advisor
 EC2 Systems Manager
 Parameter Store // State Manager // Inventory // Maintenance
Windows // Patch Manager // Run Command
Introducing AWS Management Tools

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Discover
Manage
Monitor
Report
Respond
Define
AWS CloudFormation
AWS Service Catalog
Amazon EC2 Systems Manager
AWS Config
AWS CloudTrail
Amazon CloudWatch
Trusted Advisor
AWS Management Tools: You choose

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Management Tools: CliffsNotes
1.Tools that automate lifecycle management of AWS resources,
especially as you scale usage
2.No need to trade off visibility/control against agility
3.Provide better visibility and control than customers experienced
previously
4.Primitives that can match changes to customers governing
philosophy

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customers who use AWS Management Tools

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS at Amazon

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Behind The Scenes Exclusive

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Behind The Scenes Exclusive

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How We Roll
Two-Pizza Teams
+

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How We Roll
Two-Pizza Teams
+
Single Threaded Ownership
+

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How We Roll
Two-Pizza Teams
+
Single Threaded Ownership
+
Bias For Action
+

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How We Roll
Two-Pizza Teams
+
Single Threaded Ownership
+
Bias For Action
+
Continuous Deployment
=

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How We Roll
Two-Pizza Teams
+
Single Threaded Ownership
+
Bias For Action
+
Continuous Deployment
=
A LOT OF CHANGE

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A Brief History

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Initial Approach
On-Premise Network
us-east-1 eu-west-1 us-west-2 …
Shared Account

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
On-Premise Network
Team Accounts
Today
us-east-1 eu-west-1 us-west-2 …
Shared Account

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Today
• Prime Day 2016
“Amazon retail team increased the size of their EC2 fleet, adding capacity that
was equal to all of AWS and Amazon.com back in 2009”
• Prime Day 2017
• DynamoDB: 3.34 trillion requests, peaking at 12.9 million per second
• AWS Config: over 14 million configuration items tracked
• AWS CloudTrail: 50 billion events, 419 billion API calls
• CloudFormation: Nearly 31,000 stacks created for Prime Day
https://aws.amazon.com/blogs/aws/prime-day-2017-powered-by-aws

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Two Models, Different Challenges
Team AccountsShared Account

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Access Control

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Access Control
Amazon RDS Instance

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Access Control
Amazon RDS Instance

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Access Control
Upgrade DB
Engine Version
Amazon RDS Instance

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Access Control
Amazon RDS Instance
Delete Instance

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Traditional Approach – Central Team

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Traditional Approach – Central Team

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Alternative 1 – Build A Wrapper

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Alternative 2 – Resource Permissions
"Action":[ ”rds:*” ],
"Effect":"Allow",
"Resource":"*",
"Condition":{
"StringEquals":{
”rds:db-tag/team-name":
[”finance"]
}
}

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Many Accounts

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Initial Approach - Restricting IAM
• Take away root credentials
• All IAM Roles are created through a management layer
• AuthZ through an identity broker

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Restricting IAM - Identity Broker
1. Authenticate + Authorize 2. Assume Role
3. STS Token
4. URL
5. Launch Console
Identity Broker
I am: Bob Roberts
I want to: Manage-RDS
On: AWS Account 1234367

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Restricting IAM – Lessons
• Yet Another Shim!
• Prevents some automation
• Not granular enough

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Getting Out Of The Way

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ASAP
SNS Topic
AWS ConfigAWS Cloudtrail
AWS Account
Amazon
Cloudwatch
Events

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ASAP
SNS Topic
SQS
Notification
Queue
Event Dispatcher Rule Evaluators
AWS ConfigAWS Cloudtrail
ASAP
AWS Account
Describe state
Amazon
Cloudwatch
Events

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ASAP
SNS Topic
SQS
Notification
Queue
Event Dispatcher Rule Evaluators
Reactor
SNS Topics
AWS ConfigAWS Cloudtrail
ASAP
AWS Account
Describe state
Reactors
Amazon
Cloudwatch
Events

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Things that changed
• Baselines and drift detection
• Change notification with rich data
AWS Config
AWS Account
SNS Topic
AWS Config

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudWatch Events
• Things that happened
• Not just user-initiated events
• Scheduled and custom events
SNS Topic
AWS Account
Amazon
Cloudwatch
Events

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• User and application activity
• Every API call captured and logged
• Broad coverage of AWS features
• With Cloudwatch Events – no log crawls
AWS Cloudtrail
SNS Topic
AWS Account
AWS Cloudtrail
Amazon
Cloudwatch
Events

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ASAP IRL
AWS Account
Amazon RDS
Instance

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ASAP IRL
AWS Account
AWS Cloudtrail
[{ ...
"arn:aws:iam::123456789012:user/Mike",
"eventTime": "2017-11-10T21:22:54Z",
"eventSource": ”rds.amazonaws.com",
"eventName": ”CreateDbInstance",
"awsRegion": "us-east-2",
"requestParameters": {
”dbInstanceId": ”mine-all-mine”,
“MultiAZ” : “false” }
...
Amazon RDS
Instance

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ASAP IRL
AWS Account
AWS Cloudtrail
Amazon
Cloudwatch
Events
Amazon RDS
Instance

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ASAP IRL
AWS Account
AWS Cloudtrail
SNS Topic
Amazon
Cloudwatch
Events
Amazon RDS
Instance

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ASAP IRL
AWS Account
AWS Cloudtrail
SNS Topic
ASAP
SQS
Notification
Queue
Event Dispatcher Rule Evaluators
Amazon
Cloudwatch
Events
Amazon RDS
Instance

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ASAP IRL
AWS Account
Reactor
AWS Cloudtrail
SNS Topic
ASAP
SQS
Notification
Queue
Event Dispatcher Rule Evaluators Reactor
SNS TopicsAmazon
Cloudwatch
Events
Amazon RDS
Instance

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ASAP IRL
AWS Account
Reactor
AWS Cloudtrail
SNS Topic
ASAP
SQS
Notification
Queue
Event Dispatcher Rule Evaluators Reactor
SNS Topics
Ticket / Notification
Amazon
Cloudwatch
Events
Amazon RDS
Instance

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ASAP in Action
SNS Topic
SQS
Notification
Queue
Event Dispatcher Rule Evaluators Reactor
SNS TopicsAmazon
Cloudwatch
Events
AWS Cloudtrail
ASAP
AWS Account
Reactor
Amazon RDS
Instance
Ticket / Notification

51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What do we get?
• Scale
• Ability to assess the broader context
• Catch problems early
• Rich feedback and teaching tools
• Ability to automatically escalate, mitigate or reverse (careful!)

52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Other considerations
• Anti-entropy checks
• Tamper detection
• Exceptions
• Regular sweeps

53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Looking ahead
• Not just security
• Simplify fixes
• AWS CloudFormation StackSets and AWS Config Rules

54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Avoiding Problems

55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS CloudFormation
• Be secure by default
• ‘Known Good’ configurations:
• VPC
• Static Website via S3 + CloudFront
• Amazon Aurora Clusters
• Standard resources:
• Cloudwatch Log Groups
• Default IAM roles

56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon EC2 Systems Manager (SSM)
• Automated patching
• Patch Group – what instances get patched?
• Patch Baseline – what patches are applied?
{PatchFilterGroup=
{PatchFilters=[
{Key=PRODUCT,Values=[AmazonLinux2017.09]},
{Key=CLASSIFICATION,Values=[Security]}]},
ApproveAfterDays=0}

57. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Takeaways
• Shared accounts have scaling limits
• AWS enables people to move faster .. try to stay out of their way.
• Everything changes when everything is an API.
• Set people off on the right path, help them stay on it.

58. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Takeaways
• Shared accounts have scaling limits
• AWS enables people to move faster .. try to stay out of their way.
• Everything changes when everything is an API.
• Set people off on the right path, help them stay on it.
Governance vs. Agility doesn’t have to be a binary decision!

59. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Actions
• Pick your governance philosophy early and know it will evolve
• Select management tools that help you implement your governance
philosophy
• Adopt services when you’re ready and be comfortable to deprecate what
you have built
• You can build ASAP too!

60. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Management Tools: CliffsNotes
1.Tools that automate lifecycle management of AWS resources,
especially as you scale usage
2.No need to trade off visibility/control against agility
3.Provide better visibility and control than customers experienced
previously
4.Primitives that can match changes to customers governing
philosophy

61. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Easy to try Management Tools
1. On the internet: https://aws.amazon.com/products/management/
2. AWS Management Tools blog post : https://aws.amazon.com/blogs/mt/
3. You can learn a lot just by turning on AWS Config Rules: https://github.com/awslabs/aws-config-rules
4. Use the console!

62. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
P l e a s e s u b m i t y o u r s u r v e y