You already know that AWS CloudFormation is a powerful tool for provisioning and managing your AWS infrastructure, but did you know that it can also provision and manage resources outside of AWS? Did you know that CloudFormation can fully bootstrap your EC2 instances, securely download data from S3, and even supports Mustache templates? In this session you will go on a deep dive, touring of some of CloudFormation's most advanced features with a member of the team that built the service. Explore custom resources, cfn-init, S3 authentication, and Mustache templates in a series of technical demos with code samples available for download afterwards.

1. AWS CloudFormation Under the Hood
Adam Thomas, Amazon Web Services
DJ Edwards, Amazon Web Services
November 14, 2013
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. So, what is CloudFormation?

3. This talk will not answer that question
• DMG201 - Zero to Sixty: AWS
CloudFormation
– Has already happened, but will be available online
• Hands-on Labs
– Working with CloudFormation
– Launching and Managing a Web Application with
CloudFormation
– Creating an Amazon Virtual Private Cloud (VPC) with
CloudFormation

4. This talk will answer these questions:
•
•
•
•
What is a custom resource?
What can they do for me?
How do I write one for myself?
What’s new in cfn-init?

5. Custom Resources

6. What can custom resources do?
•
•
•
•
Add New Resources
Interact with the CloudFormation Workflow
Inject dynamic data into a stack
Extend the capabilities of existing resources

7. What is a custom resource?
• An SNS topic…
• …hooked up to a service that can:
– Respond to JSON messages from CloudFormation
– Manage the lifecycle of resources

8. How are custom resources defined?
"myCustomResource" : {
"Type": "Custom::MyCustomResource",
"Version" : "1.0",
"Properties" : {
"ServiceToken": "arn:aws:sns:us-east-1:84969EXAMPLE:CRTest",
"CustomProperty" : "foo"
}
}

9. How are custom resources defined?
"myCustomResource" : {
"Type": “Custom::MyCustomResource",
"Version" : "1.0",
"Properties" : {
"ServiceToken": "arn:aws:sns:us-east-1:84969EXAMPLE:CRTest",
"CustomProperty" : "foo"
}
}

10. How are custom resources defined?
"myCustomResource" : {
"Type": “Custom::MyCustomResource",
"Version" : "1.0",
"Properties" : {
"ServiceToken": "arn:aws:sns:us-east-1:84969EXAMPLE:CRTest",
"CustomProperty" : "foo"
}
}

11. What can custom resources do?
•
•
•
•
Add New Resources
Interact with the CloudFormation Workflow
Inject dynamic data into a stack
Extend the capabilities of existing resources

12. Adding New Resources
• Something that can be Created, Updated,
and/or Deleted
• Can be a software resource
– Database schema, Docker container

13. Meet Steve
• Steve loves RDBMS
• The schema is very
important to Steve – it
defines his application
• Running SQL scripts by
hand is Steve’s worst
nightmare

14. Steve’s requirements
• The Template should
define the schema
explicitly
• The schema should be
updated by updating the
stack
• If the update fails, the
schema should roll back

15. Steve’s solution
• Steve is very familiar with
Liquibase
• Liquibase supports JSON
formatting!
• Steve writes a custom
resource with inline JSON
schema

16. DB Schema Template Snippet
"appSchema" : {
"Type" : "Custom::DatabaseSchema",
"Properties" : {
"databaseChangeLog" :
[{ "changeSet" : {
"id" : "1",
"author" : “adam",
"changes" :
[{ "createTable" : {
"tableName" : "person",
"columns" : …

17. DB Schema Template Snippet
"appSchema" : {
"Type" : "Custom::DatabaseSchema",
"Properties" : {
"databaseChangeLog" :
[{ "changeSet" : {
"id" : "1",
"author" : “adam",
"changes" :
[{ "createTable" : {
"tableName" : "person",
"columns" : …

18. DB Schema Demo

19. What can custom resources do?
•
•
•
•
Add New Resources
Interact with the CloudFormation Workflow
Inject dynamic data into a stack
Extend the capabilities of existing resources

20. Interacting with the CloudFormation
Workflow
• Use custom resources as a hook into
create/update/delete workflows
• Built-in example: WaitCondition
• Can react to workflow, halt it, or fail it under
certain conditions

21. Meet Frank
• Frank analyzes data
stored on EBS
• Frank uses
CloudFormation’s
Snapshot on Delete
feature to save his
analysis results

22. Frank’s requirements
• Frank wants a consistent
EBS snapshot when the
stack is deleted
• Before CloudFormation
attempts to detach his EBS
volume, it should:
– Cleanly shut down his
analysis service
– Unmount the volume

23. Why is this a challenge?
• CloudFormation can detach volumes without
any issues – if you never mount them
• What CloudFormation does not do, it cannot
undo
• Custom resources let you model your steps
within the workflow

24. Frank’s solution
• 3 simple bash scripts
• A “local” Custom
Resource – runs directly
on the instance
• Create and Update mount
the drive; Delete
unmounts it.

25. Volume Mount Template Snippet
“VolumeAttach" : {
"Type" : "AWS::EC2::VolumeAttachment",
"Properties" : …
},
"VolumeMount" : {
"Type" : "Custom::VolumeMount",
"Properties" : {
"Device" : “/dev/xvdh”,
“MountPoint” : “/mnt/analysis”
}
}

26. Volume Mount Template Snippet
“VolumeAttach" : {
"Type" : "AWS::EC2::VolumeAttachment",
"Properties" : …
},
"VolumeMount" : {
"Type" : "Custom::VolumeMount",
"Properties" : {
"Device" : “/dev/xvdh”,
“MountPoint” : “/mnt/analysis”
}
}

27. Volume Mount Template Snippet
“VolumeAttach" : {
"Type" : "AWS::EC2::VolumeAttachment",
"Properties" : …
},
"VolumeMount" : {
"Type" : "Custom::VolumeMount",
"Properties" : {
"Device" : “/dev/xvdh”,
“MountPoint” : “/mnt/analysis”
}
}

28. Volume Mount Template Snippet
“VolumeAttach" : {
"Type" : "AWS::EC2::VolumeAttachment",
"Properties" : …
},
"VolumeMount" : {
"Type" : "Custom::VolumeMount",
"Properties" : {
"Device" : “/dev/xvdh”,
“MountPoint” : “/mnt/analysis”
}
}

29. Volume Mount Demo

30. What can custom resources do?
•
•
•
•
Add New Resources
Interact with the CloudFormation Workflow
Inject dynamic data into a stack
Extend the capabilities of existing resources

31. Injecting Dynamic Data into a Stack
• Parameters are standard route into a stack
– Allow free-form user input
– Constrainable, but on a per-stack level
• Mappings are traditionally used to map humanreadable input to static values
– AMI IDs, instance type architectures, regional URLs

32. Injecting Data into a Stack
• Custom resources allow for centralized selection
logic
• Lookups in:
–
–
–
–
S3
DynamoDB/RDS
APIs (EC2.DescribeImages, etc)
Third Party datastore

33. Meet Bill
• Bill is the head of operations
at a large tech firm
• Each of Bill’s 44 services
must run on a fully validated
and tested AMI
• Bill keeps track of these
AMIs in a sweet multitabbed Excel spreadsheet

34. Bill’s requirements
• New AMIs should be
rolled out centrally
• Bill does not want to edit
the Mappings section of
44 templates for every
release
• Bill wants to audit where
AMIs are being used

35. Bill’s solution
• A manifest of named,
approved AMIs stored in a
versioned S3 file
• A simple python script that
looks up the AMI ID by
region and os,
architecture, and version

36. AMI Lookup Template Snippet
"AMILookup": {
"Type": "Custom::AmiLookup",
"Properties": {
"os": "ubuntu",
"version": “13.04",
"arch": "64"
}
},
"WebServer": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId" : { “Ref" : “AMILookup” }
}
}

37. AMI Lookup Template Snippet
"AMILookup": {
"Type": "Custom::AmiLookup",
"Properties": {
"os": "ubuntu",
"version": "13.04",
"arch": "64"
}
},
"WebServer": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId" : { “Ref" : “AMILookup” }
}
}

38. AMI Lookup Template Snippet
"AMILookup": {
"Type": "Custom::AmiLookup",
"Properties": {
"os": "ubuntu",
"version": "13.04",
"arch": "64"
}
},
"WebServer": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId" : { “Ref" : “AMILookup” }
}
}

39. AMI Lookup Demo

40. What can custom resources do?
•
•
•
•
Add New Resources
Interact with the CloudFormation Workflow
Inject dynamic data into a stack
Extend the capabilities of existing resources

41. Extending Resource Capabilities
• CloudFormation is concerned only with Create,
Update, and Delete
• Some services, like AutoScaling, have lifecycles
outside of these phases
• No place in template to encapsulate longrunning, resource-based business logic

42. Meet Tom
• Tom manages a fleet of
virtual desktops in AWS
• Tom uses AutoScaling for
consistent fleet size
• Tom’s users use VNC to
connect to their virtual
desktops

43. Tom’s requirements
• Servers should be named
using his clever, easy-toremember Simpsons
scheme
• Names should be
recycled as machines are
replaced

44. Tom’s solution
• Python scripts respond to
Auto Scaling notifications
to manage Route53
records
• Names are managed in a
simple DynamoDB table

45. Auto Scaled DNS Snippet (1 of 2)
"DNSProcessor" : {
"Type": "Custom::DNSProcessor",
"Properties": {
"HostedZoneId" : { "Ref" : "HostedZone" },
"DNSPattern" : {"Fn::Join" : [".",[“{{simpsons_name}}", { "Ref" :
"AWS::Region" }, “{{hosted_zone_name}}"]] }
}
},

46. Auto Scaled DNS Snippet (1 of 2)
"DNSProcessor" : {
"Type": "Custom::DNSProcessor",
"Properties": {
"HostedZoneId" : { "Ref" : "HostedZone" },
"DNSPattern" : {"Fn::Join" : [".",[“{{simpsons_name}}", { "Ref" :
"AWS::Region" }, “{{hosted_zone_name}}"]] }
}
},

47. Auto Scaled DNS Snippet (2 of 2)
"WebServerGroup" : {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties" : {
"NotificationConfiguration" : {
"TopicARN" : { "Fn::GetAtt" : ["DNSProcessor", “Topic"] },
"NotificationTypes" : [
"autoscaling:EC2_INSTANCE_LAUNCH","autoscaling:EC2_INSTANCE_TERMINATE"]
},
"Tags" : [{ "Key" : "ProcessorId",
"Value" : { "Ref" : "DNSProcessor" },
"PropagateAtLaunch" : false }]
}
}

48. Auto Scaled DNS Snippet (2 of 2)
"WebServerGroup" : {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties" : {
"NotificationConfiguration" : {
"TopicARN" : { "Fn::GetAtt" : ["DNSProcessor", “Topic"] },
"NotificationTypes" : [
"autoscaling:EC2_INSTANCE_LAUNCH","autoscaling:EC2_INSTANCE_TERMINATE"]
},
"Tags" : [{ "Key" : "ProcessorId",
"Value" : { "Ref" : "DNSProcessor" },
"PropagateAtLaunch" : false }]
}
}

49. Auto Scaled DNS Demo

50. Building Your Own Custom Resource
• Write code to respond to Create, Update, and
Delete events
• Route Custom Resource SNS Topic to an SQS
Queue for maximum fault tolerance

51. Can you give me a diagram?
CloudFormation Stack Workflow starts building Custom Resource
CloudFormation sends CREATE notification to Custom Resource
Custom Resource creates resource and returns JSON message
CloudFormation processes JSON message and stores result
Stack workflow continues
Other resources access Custom Resource attributes via GetAtt and Ref

52. How about an architectural overview?
SQS Queue
Custom Resource Topic
AWS CloudFormation
Custom Resource
Implementation
Auto scaling Group
Region

53. Can you add VPC?
Custom Resource Topic
SQS Queue
AWS CloudFormation
Custom Resource
Implementation
Region
VPN
Existing Service
Corporate Data center

54. What makes for a good resource?
• Good resources are: idempotent
– One unique request, n times == one unique response
• Immediately usable when complete
• Can be deleted cleanly from any state
• Represent one standalone piece of functionality
– Embedded resources look convenient, but are hard to update
– Elastic Load Balancers embed Policies, which can depend on
each other, yet this is not modeled in the template

55. You keep telling me it’s simple…
• It’s really simple if you use aws-cfn-resourcebridge
• Cross-platform hook-based daemon
• Simply supply scripts for Create, Update, and
Delete
• Open source (Apache 2.0)
• Install or fork from https://github.com/aws/awscfn-resource-bridge

56. Example Code
• All examples from this talk are available at
https://github.com/awslabs/aws-cfn-customresource-examples
• Stealing from others is the easiest way to get
started
– And the best way to use CloudFormation!

57. cfn-init

58. cfn-init
• Simple library for “getting bits on the box”
• Install packages, download files, start services
• Works on Windows, Linux, and any platform with
Python 2.6, 2.7

59. {{cfn-init}}
•
•
•
•
Fn::Join can be hard to follow
Many configuration files are largely boilerplate
Files can process Mustache templates
Simply add context

60. Wordpress config
"/var/www/html/wordpress/wp-config.php" : {
"content" : { "Fn::Join" : ["", [
"<?phpn",
"define('DB_NAME',
'", {"Ref" : "DBName"}, "');n",
"define('DB_USER',
'", {"Ref" : "DBUser"}, "');n",
"define('DB_PASSWORD',
'", {"Ref" : "DBPassword" }, "');n",
"define('DB_HOST',
'", {"Fn::GetAtt" : [“MyDB",
"Endpoint.Address"]},"');n",
"define('DB_CHARSET',
'utf8');n",
"define('DB_COLLATE',
'');n",
"define('AUTH_KEY',
'f@A17vs{
mO0}:&I,6SB.QzV`E?!`/tN5:~GZX%=@ZA%!_T0-]9>g]4ll6~,6G|R');n",

61. Wordpress config
"/var/www/html/wordpress/wp-config.php" : {
"content" : { "Fn::Join" : ["", [
"<?phpn",
"define('DB_NAME',
'", {"Ref" : "DBName"}, "');n",
"define('DB_USER',
'", {"Ref" : "DBUser"}, "');n",
"define('DB_PASSWORD',
'", {"Ref" : "DBPassword" }, "');n",
"define('DB_HOST',
'", {"Fn::GetAtt" : [“MyDB",
"Endpoint.Address"]},"');n",
"define('DB_CHARSET',
'utf8');n",
"define('DB_COLLATE',
'');n",
"define('AUTH_KEY',
'f@A17vs{
mO0}:&I,6SB.QzV`E?!`/tN5:~GZX%=@ZA%!_T0-]9>g]4ll6~,6G|R');n",

62. Wordpress config
"define('SECURE_AUTH_KEY',
'gTFTI|~rYHY)|mlu:Cv7RN]GQ^3ngyUbw;L0o!12]0c-ispR<-yt3qj]xjquz^&9');n",
"define('LOGGED_IN_KEY',
'Jd:HG9M)1p5t2<v~+R-vd{pQ*|*RB^&PUI{vIrydAEEiV!{HS{jN:nErCmLv`p}');n",
"define('NONCE_KEY',
'4aMj4KZV;,Gu7(B|qOCve[c5?*J5x1+x93i:Ey6hh/6jXh+V_{V4+hw!qE^d*U,-');n",
"define('AUTH_SALT',
'_Y_&8m)FH)Cns)8}Yb8b88KDSn:p1#p(qBa<~VW&Y1v}P.*9/8S8@P`{mkNxV lC');n",
"define('SECURE_AUTH_SALT',
'%nG3Ag41^Lew5c86,#zbN:yPFs.GA5a)z5*:Oce1>v6uF~D`,.o1pzS)F8[bM9i[');n",
"define('LOGGED_IN_SALT',
'~K<y+Ly+_Ww1~dtq>;rSQ^+{P5/k|=!]k%RXAFY@XMY6GSp+wJ5{(|rCzaWjZ%/');n",
"define('NONCE_SALT',
',Bs_*Y9:b/1Z:apVLHtz35uim|okkA,b|Jt[&Nla=T{<l_#D?~6Tj-.2.]FonI~');n",

63. Wordpress config
"define('WPLANG'
, '');n",
"define('WP_DEBUG'
, false);n",
"$table_prefix = 'wp_';n",
"if ( !defined('ABSPATH') )n",
"
define('ABSPATH', dirname(__FILE__) . '/');n",
"require_once(ABSPATH . 'wp-settings.php');n"
]] }
}

64. {{cfn-init}} Wordpress Config
“files” : {
"/var/www/html/wordpress/wp-config.php" : {
“source” : “https://github.com/FAKEPATH/wp-config.mustache”,
“context” : {
“DbEndpoint” : {“Fn::GetAtt” : [“MyDB”, “Endpoint.Address”]},
“DbName” : { “Ref” : “DbName” },
“DbUser” : { “Ref” : “DbUser” },
“DbPassword” : { “Ref” : “DbPassword” }
}
}
}

65. {{cfn-init}} Wordpress Config
“files” : {
"/var/www/html/wordpress/wp-config.php" : {
“source” : “https://github.com/FAKEPATH/wp-config.mustache”,
“context” : {
“DbEndpoint” : {“Fn::GetAtt” : [“MyDB”, “Endpoint.Address”]},
“DbName” : { “Ref” : “DbName” },
“DbUser” : { “Ref” : “DbUser” },
“DbPassword” : { “Ref” : “DbPassword” }
}
}
}

66. {{cfn-init}} Wordpress Config
“files” : {
"/var/www/html/wordpress/wp-config.php" : {
“source” : “https://github.com/FAKEPATH/wp-config.mustache”,
“context” : {
“DbEndpoint” : {“Fn::GetAtt” : [“MyDB”, “Endpoint.Address”]},
“DbName” : { “Ref” : “DbName” },
“DbUser” : { “Ref” : “DbUser” },
“DbPassword” : { “Ref” : “DbPassword” }
}
}
}

67. Roleplaying
• cfn-init can use roles to download from S3
• Secured files are not just for proprietary code
– Non-AWS credentials
– Private service endpoints
– Dynamic code (enabling or disabling features)

68. Roleplaying Template Snippet
“AWS::CloudFormation::Authentication” : {
“roleCreds” : {
“type” : “S3”,
“roleName” : “MyS3Role”
}
}
…
“files” : {
“/etc/secrets.txt” : {
“source” : “https://s3.amazonaws.com/mybucket/secrets.txt”,
“authentication” : “roleCreds”
}
}

69. Roleplaying Template Snippet
“AWS::CloudFormation::Authentication” : {
“roleCreds” : {
“type” : “S3”,
“roleName” : “MyS3Role”
}
}
…
“files” : {
“/etc/secrets.txt” : {
“source” : “https://s3.amazonaws.com/mybucket/secrets.txt”,
“authentication” : “roleCreds”
}
}

70. Roleplaying Template Snippet
“AWS::CloudFormation::Authentication” : {
“roleCreds” : {
“type” : “S3”,
“roleName” : “MyS3Role”
}
}
…
“files” : {
“/etc/secrets.txt” : {
“source” : “https://s3.amazonaws.com/mybucket/secrets.txt”,
“authentication” : “roleCreds”
}
}

71. cfn-hup
•
•
•
•
Not new, but not often used in samples
Installed in same package as cfn-init
Available as Linux and Windows service
Listens for changes to the stack and runs scripts
when they occur
– Usually just runs or re-runs cfn-init

72. Custom Resources vs. cfn-hup
• Custom Resources require an SNS topic, and
usually an SQS queue
• cfn-hup cannot interact with CloudFormation
workflow
– Workflow will not wait for cfn-hup
– cfn-hup cannot fail workflow
– cfn-hup cannot inject data into stack

73. Summary
• Custom Resources let you extend
CloudFormation beyond the existing Resource
Library
• For more than just “things that can be created”
• cfn-init lets you use Mustache and Roles to
create simple, secure configuration

74. Corner us in the Developer Lounge
Adam Thomas
DJ Edwards

75. Please give us your feedback on this
presentation
DMG303 - AWS CloudFormation Under the Hood
As a thank you, we will select prize
winners daily for completed surveys!