Sydney based cloud consultancy Cloudten's Richard Tomkinson shows how masterless Puppet can be used in concert with AWS's services including Lambda to automate server builds and manage code deployments

1. ©opyright
2015
Cloudten
Industries

2. ©opyright
2015
Cloudten
Industries
Copyright statement:
This document contains a presentation given to the
Puppet User Group by Cloudten Industries in January
2016. It has been made available freely for
educational purposes. No part of this document may
be reproduced or modified without the express written
consent of the author.

3. Copyright 2015
Cloudten
Industries
• Project Overview
• Technology Stack
• Puppet builds and deployments
• Security
• Issues
• Q&A

4. Copyright 2015
Cloudten
Industries
Project
Overview
• Client was a startup with VC backing.
• They wanted to build a mobile social media app:
• Dynamically scalable up to 100,000 hits per second
• Cross site active/active with real time DR
• Multiple test environments to be stood up/down at will
• End to end security with encryption at rest
• Engaged separate mobile and web app developers

5. Copyright 2015
Cloudten
Industries
Multiple
Development
Teams
Company
A: Mobile
development
team
in
Melbourne
Company
B: Web
App
development
team
in
Perth/India

6. Copyright 2015
Cloudten
Industries
Multiple
Development
Teams
• Using
Xamarin
Studio
to
create
Apple
and
Android
front
ends
• Require
access
to
code
base
to
publish
WSDLs
under
web
app
• Require
access
to
app
logs
• Using
a
JDK
to
create
a
J2EE
app
running
in
Tomcat
• Require
access
to
entire
web
app
code
base
• Require
access
to
more
logs

7. Copyright 2015
Cloudten
Industries
What
Did
We
Need
to
Solve
?
• Client didn’t have (or want) any full time IT staff.
• Client did want:
– Hands off builds and app deployments
– Self managed consistent server fleet
– No outage deployments
– No direct access to infrastructure from developers

8. Copyright 2015
Cloudten
Industries
Hosting
in
AWS
• AWS provide on-demand scalable resources
• Facility to implement “Infrastructure as Code”
• Secure and durable object storage for code drops
• Fine grained security controls to create server roles
and limit developer access.
• Additional services to co-ordinate deployments
(Lambda, SNS, SQS)

9. Copyright 2015
Cloudten
Industries
AWS
Autoscaling
AWS can automatically add (and remove) servers to a load
balancer pool based on a given metric ( eg. CPU or number of
connections)
Scaling
trigger
hit Scale
out
to
share
the
load

10. Copyright 2015
Cloudten
Industries
How
does
it
work
?
• Launches and builds identical virtual machines
• Destroys them with reckless abandon.
• Essential to get all logs into a central store
• Any generated content must be shared (i.e. can’t
be stored on a local machine)
• The instance launch can trigger a build process
(e.g. Puppet )

11. Copyright 2015
Cloudten
Industries
Automated
Server
Builds
Start
with
base
Amazon
Linux
Image
Invoke
bootstrap
script
Install
Puppet
RPM
S3
sync
config and
Puppet
manifests
Puppet
Apply

12. Copyright 2015
Cloudten
Industries
Puppet
Build
Tasks
• Linux security patching and kernel hardening
• Define custom package repositories
• AWS tagging using facter
– Querying metadata to set instance specific tags
– Set tags for environment variables to be used later
• Package installation, config and version enforcement
– NGINX, WAF, Tomcat, monit, sumo agents etc
• Configuration management
– functional users, public keys, cron jobs, log rotations, system health checks

13. Copyright 2015
Cloudten
Industries
EC2
Tagging
with
Facter
# Sets hostname
class common::hostname {
require aws
file { '/etc/hostname':
ensure => 'present',
owner => 'root',
group => 'root',
mode => '0644',
content => template('common/hostname.erb'),
notify => [ Exec['Set Hostname'],
Exec['Set EC2 Name-tag']
],
}
...
exec { 'Set Hostname':
command => "/bin/hostname -F /etc/hostname",
unless => "/usr/bin/test `hostname` = `/bin/cat /etc/hostname`",
refreshonly => 'true',
}
exec { 'Set EC2 Name-tag':
command => '/usr/local/bin/setec2nametag',
unless => '/usr/bin/test `/usr/local/bin/facter ec2_tag_Name` = `/bin/cat /etc/hostname`',
}
}

14. Copyright 2015
Cloudten
Industries
Kernel
Tuning
# sysctl class
class common::sysctl {
file { '/etc/sysctl.conf':
source => 'puppet:///modules/common/sysctl.conf',
owner => 'root',
group => 'root',
mode => '0644',
notify => Exec['Refresh sysctl'],
}
exec { 'Refresh sysctl':
command => '/sbin/sysctl -q -p',
refreshonly => 'true',
}
}

15. Copyright 2015
Cloudten
Industries
Developer
Code
Drops
MySQL
App

16. Copyright 2015
Cloudten
Industries
Event
Based
Triggers
cron
1
2
334 4
5 5
cron
control
script
6
7
Complete
deployment
and
set
a
marker
file
as
a
trigger
Check
for
marker.
If
present
put
message
on
SQS
queue
CS
checks
for
message
on
queue
Get
message
from
queue
S3
sync
of
changes
to
staging
area
Puppet
applies
changes
and
restarts
services

17. Copyright 2015
Cloudten
Industries
How
Does
that
Queue
Work
?
• Lambda creates the queue once it detects trigger file (if it doesn’t exist already)
• Lambda queries the auto-scaling group and creates a message on the SQS
queue for each member. It then deletes trigger file
• Each message has a 30 minute expiry ( deployments usually take <5 minutes)
• SQS queue has multiple consumers ( app servers)
• App server checks for its own message, retrieves then deletes it once deployment
is successful.
• Lambda periodically checks for messages about to expire and the dead letter
queue. Triggers an email alert if there is an issue.

18. Copyright 2015
Cloudten
Industries
Puppet
Deployment
Tasks
Check for changes in
staging content area
Sync content
Check for configuration
file changes in staging
areas
Apply changes
Restart/reload
relevant services
Random sleep

19. Copyright 2015
Cloudten
Industries
Puppet
Deployment
Tasks
# Conf dir file { '/etc/nginx/conf':
ensure => ‘directory’,
source => 'puppet:///modules/nginx/conf',
recurse => true
notify => Service[’nginx'],
require => Package[’nginx'],
}
# WAF rules
file { '/etc/nginx/waf/modsec_waf.rules':
source => 'puppet:///modules/nginx/waf/modsec_waf.rules',
...
notify => Service[’waf'],
require => Service[‘nginx’].
require => Package[’nginx'],
}

20. Copyright 2015
Cloudten
Industries
AWS
IAM
Users/Groups/Roles
• IAM (Identity & Access Management) allows fine grained
user, group and role definitions
• S3 Bucket policies add a further level of security to restrict
access to resources stored in S3
• Web developer bucket policy allows full access for web dev
group
• Mobile developer bucket policy allows full access for mobile
dev group and read only access for web dev group

21. Copyright 2015
Cloudten
Industries
IAM
Users

22. Copyright 2015
Cloudten
Industries
IAM
Users
{
…
{
"Sid":
"AllowS3ListAccessToBucket",
"Effect":
"Allow",
"Principal":
{
"AWS":
"arn:aws:iam::123456789101:group/webdevs"
},
"Action":
"s3:ListBucket",
"Resource":
"arn:aws:s3:::fdsr-­‐webcontent-­‐puppet-­‐prod"
},
{
"Sid":
"AllowS3AccessToModules",
"Effect":
"Allow",
"Principal":
{
"AWS":
[
"arn:aws:iam::123456789101:group/webdevs",
"arn:aws:iam::123456789101:role/iam-­‐ec2-­‐webrole",
]
},
"Action":
[
"s3:AbortMultipartUpload",
"s3:GetObjectAcl",
"s3:DeleteObject",
"s3:GetObject",
"s3:PutObjectAcl",
"s3:PutObject"
],
"Resource":
"arn:aws:s3:::fdsr-­‐webcontent-­‐puppet-­‐prod/modules/webapp/files*"
}
]
}

23. Copyright 2015
Cloudten
Industries
IAM
Users/Groups/Roles
App
Server

24. Copyright 2015
Cloudten
Industries
IAM
Users/Groups/Roles
IAM roles allow dynamically launched virtual servers to securely
access credentials by querying locally accessible metadata at
the special use IP address 169.254.169.254
$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/s3read-only
{
"Code" : "Success",
"LastUpdated" : "2015-04-26T16:39:16Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "AKIAIOSFODNN7EXAMPLE",
"SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
"Token" : "token",
"Expiration" : "2015-04-27T22:39:16Z"
}

25. Copyright 2015
Cloudten
Industries
Why
Masterless Puppet
• Less instances to manage ( no IT staff to manage it )
• No access for developers to internal infrastructure
• Not Enterprise Puppet
• No need to manage Puppet certificates ( AWS API calls are all
encrypted and IAM enforces authentication)

26. Copyright 2015
Cloudten
Industries
What
Issues
Did
We
Have
?
• Enforced versions being removed from repos
• Auto-scaling and healthcheck tuning
• S3 sync not handling zero byte files properly
• Event trigger mechanism needed tweeking
• Developers storing files on local instances
• S3FS. Don’t use it ! EVER !

27. Copyright 2015
Cloudten
Industries
Who
Are
Cloudten ?
• Advanced AWS Consulting Partner
• Specialise in the design, delivery and support of
cloud based infrastructure projects
• Focus on cloud security and hybrid integration
• We are a Puppet shop !

28. ©opyright
2015
Cloudten
Industries