More Related Content Similar to DEV337_Deploy a Data Lake with AWS CloudFormation (20) More from Amazon Web Services (20) DEV337_Deploy a Data Lake with AWS CloudFormation1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Luis Colon, Sr. Developer Advocate, AWS Cloud Formation
Chuck Meyer, Sr. Developer Advocate, AWS Cloud Formation
D E V 3 3 7
N o v e m b e r 2 8 , 2 0 1 7
Deploy a Data Lake
with AWS CloudFormation
2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CLOUDFORMATION AT A GLANCE
Code in YAML or
JSON directly or use
sample templates
Upload local
files or from
an S3 bucket
Create stack
using console,
API or CLI
Stacks and
resources are
provisioned
3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS CLOUDFORMATION
Enables the provisioning and management of your
infrastructure as code (IaC)
Supports YAML and JSON formats
Allows for reusable component design strategies
Understands dependencies, supports rollbacks and
versioning
4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CLOUDFORMATION STACKS
A template is a file listing the resources of a stack in a
declarative way: tell CloudFormation what you need created,
rather than how to create it
• When you change a template, you are declaring the need
to update your stack
• You can do so directly, or with an update plan called a
changeset
Virtually every kind of resource can be managed with stacks
5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
APPLICATION ARCHITECTURES
CDN
AppServer
Database
Static Objects
Multi-tier
Transactional
(Java, LAMP)
Cache
Containers
NoSQL Cluster
Orchestration
Microservices
Transactional
(Docker, K8s)
Endpoints
State Machine
IoT
Functions
Serverless
Transactional
(Lambda, APIs, IoT)
Event Streams
Spark Clusters
Warehouse
ETL
Big Data
Analytical
(Data Lakes, Hubs)
…
6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Build the data lake Add safety guardrails Add test pipeline (TBD?)
OUR WORKSHOP PLAN
8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
LAB 1
Allotted time: 30–35 min
Objectives:
Implement first version of data lake
Verify all components are working
9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ADDING SAFETY GUARDRAILS
STACK TERMINATION PROTECTION
RESOURCE DELETION POLICIES
STACK POLICIES
IAM POLICIES
ROLLBACK TRIGGERS
P A R T I I
10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
STACK TERMINATION PROTECTION
Prevent a stack from being accidentally deleted
11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
RESOURCE DELETION POLICY
Control deletion on a resource-by-resource basis
12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
STACK POLICIES
Prevent operations on groups of resources
13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
IAM POLICIES
Prevent users, groups, and roles from executing risky operations
14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ROLLBACK TRIGGERS
Set up alarms
while updating
stacks
If alarms fire,
automatically
roll back
Revert changes impacting performance
15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
LAB 2
Allotted time: 30–35 min
Objectives:
Implement safety guardrail controls
Verify controls are working
16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ADD A VALIDATION PIPELINE
P A R T I I I
17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
VALIDATION PIPELINE
Run a set of customizable tests for logical and
functional integrity against CloudFormation
templates
• Designed to integrate with an existing AWS
CodeCommit repo
• Automatically provisions and configures necessary
services
• AWS CodePipeline
• AWS CodeBuild
• AWS Lambda
18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
VALIDATION PIPELINE
Triggered by a commit on your repo
1. Run logical pre-create tests,
including syntax checks
2. Launch test stacks in multiple
regions
3. Runs functional post-create sets on
the test stacks
4. If tests are successful, an email is
sent to indicate template is ready for
approval
5. Final template is deployed to an S3
bucket, also storing CloudWatch
data for each Lambda function
6. Deploy stack directly from the S3
bucket
19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
LAB 3
Allotted time: 30–35 min
Objectives:
Implement validation pipeline
Verify all components are working
20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudFormation works
for virtually all
application and resource
types, including data
lakes
It also allows
implementation of
several layers of safety
guardrails for critical
resources
Treat your templates as
you would any other code,
including implementing
versioning and automated
testing
WRAP UP
21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!
Luis Colon, Sr. Developer Advocate, AWS Cloud Formation (licolon@amazon.com)
Chuck Meyer, Sr. Developer Advocate, AWS Cloud Formation (cmmeyer@amazon.com)
D E V 3 3 7 D E P L O Y A D A T A L A K E W I T H C L O U D F O R M A T I O N