Learn how the Obama campaign leveraged Amazon Web Services (AWS) and Puppet to rapidly scale their infrastructure up for the needs of the election in a sustainable manner. Using the automation that AWS and Puppet enabled -- the Obama campaign build a significant AWS infrastructure (http://awsofa.info) while having a lean DevOps team, tight deadlines and applications that needed to be highly available. Learn about using bootstrapping puppet on Amazon EC2 instances with CloudInit, using it with autoscaling groups and secure handling of credentials in manifests. Find out how to scale puppet masters and take advantage of Amazon S3 backed RPM/Debian repos with them. Leo Zhadanovsky Senior Solutions Architect, Amazon Web Services Leo Zhadanovsky is a Senior Solutions Architect at Amazon Web Services. He helps customers best leverage AWS services, in order to help them succeed in building highly-available, scalable and elastic architectures for their business needs. He was previously the Director of Systems Engineering at the Democratic National Committee. From 2009 to early 2013, he ran the DNC's physical server and cloud footprint and supported infrastructure which was in use by the Obama campaign, state and local Democratic parties. In 2010, the DNC successfully ran and deployed many applications, such as a Call Tool and Voter Registration website, that were written in Ruby and ran on AWS. In 2012, the DNC supported the Obama campaign with various backend APIs, web sites, voter file databases and a large data warehouse.

1. The Road to the White House
with Puppet & AWS
Leo Zhadanovsky – Solutions Architect – leo@amazon.com
@leozh

2. What am I talking about today?
What was OFA Tech?
• Who did it?
• What did they build?
How did they do that?
• Technologies and Tradeoffs
• Services vs. Software
How did they leverage puppet?
What did they learn from building something so big?

3. Who Am I?
I work for AWS
I worked for the DNC 2009-2012
I was embedded at OFA
AWS does not endorse
political candidates
I love Star Trek (TNG is the best)

4. So here’s the Idea
~30th biggest E-commerce operation, globally
~200 distinct new applications, many mobile
Hundreds of new, untested analytical approaches
Processing hundreds of TB of data on thousands of servers
Spikes of hundreds of thousands of concurrent users
FUN FUN FUN

5. a few constraints…
~30th biggest E-commerce operation, globally
~200 distinct applications, many mobile
Hundreds of new, untested analytical approaches
Processing hundreds of TB of data on thousands of servers
Spikes of hundreds of thousands of concurrent users
Critically compressed budget
Less than a year to execute
Volunteer and near-volunteer development team
Core systems will be used for a single critical day
Constitutionally-mandated completion date
NOT
NOT

6. Built by guys and gals like these: Obama For
America

7. Business as usual..
…for a technology startup

8. Election Day – OFA Headquarters

9. So they built it all, and it worked

11. Typical Charts

13. How?

14. The old approach, even from Amazon 

15. The old approach.. Might have some problems..

16. No Up-Front
Capital Expense
Pay Only for
What You Use
Self-Service
Infrastructure
Easily Scale
Up and Down
Improve Agility &
Time-to-Market
Low Cost
Cloud Computing Benefits
Deploy

17. OFA’s Infrastructure
awsofa.info

18. Web-Scale Applications

19. 500k+ IOPS DB Systems

20. Services API

21. Ingredients
Ubuntu nginx boundary Unity jQuery SQLServer hbase
NewRelic EC2 node.js Cybersource hive ElasticSearch
Ruby Twilio EE S3 ELB boto Magento PHP EMR SES
Route53 SimpleDB Campfire nagios Paypal CentOS
CloudSearch levelDB mongoDB python securitygroups
Usahidhi PostgresSQL Github apache bootstrap SNS
cloudformation Jekyll RoR EBS FPS VPC Mashery
Vertica RDS Optimizely MySQL puppet tsunamiUDP R
asgard cloudwatch ElastiCache cloudopt SQS cloudinit
DirectConnect BSD rsync STS Objective-C DynamoDB

22. Data Stores
Ubuntu nginx boundary Unity jQuery SQLServer hbase
NewRelic EC2 node.js Cybersource hive ElasticSearch
Ruby Twilio EE S3 ELB boto Magento PHP EMR SES
Route53 SimpleDB Campfire nagios Paypal CentOS
CloudSearch levelDB mongoDB python securitygroups
Usahidhi PostgresSQL Github apache bootstrap SNS
cloudformation Jekyll RoR EBS FPS VPC Mashery
Vertica RDS Optimizely MySQL puppet tsunamiUDP R
asgard cloudwatch ElastiCache cloudopt SQS cloudinit
DirectConnect BSD rsync STS Objective-C DynamoDB

23. Development Frameworks
Ubuntu nginx boundary Unity jQuery SQLServer hbase
NewRelic EC2 node.js Cybersource hive ElasticSearch
Ruby Twilio EE S3 ELB boto Magento PHP EMR SES
Route53 SimpleDB Campfire nagios Paypal CentOS
CloudSearch levelDB mongoDB python securitygroups
Usahidhi PostgresSQL Github apache bootstrap SNS
cloudformation Jekyll RoR EBS FPS VPC Mashery
Vertica RDS Optimizely MySQL puppet tsunamiUDP R
asgard cloudwatch ElastiCache cloudopt SQS cloudinit
DirectConnect BSD rsync STS Objective-C DynamoDB

24. Infrastructure, Configuration
Management & Monitoring
Ubuntu nginx boundary Unity jQuery SQLServer hbase
NewRelic EC2 node.js Cybersource hive ElasticSearch
Ruby Twilio EE S3 ELB boto Magento PHP EMR SES
Route53 SimpleDB Campfire nagios Paypal CentOS
CloudSearch levelDB mongoDB python securitygroups
Usahidhi PostgresSQL Github apache bootstrap SNS
cloudformation Jekyll RoR EBS FPS VPC Mashery
Vertica RDS Optimizely MySQL puppet tsunamiUDP R
asgard cloudwatch ElastiCache cloudopt SQS cloudinit
DirectConnect BSD rsync STS Objective-C DynamoDB

25. Configuration Management: Puppet
In mid-2011, we look at options for configuration
management and chose Puppet
We needed to make it scale, and to get it to work with state-
less, horizontally scalable infrastructure
How did we do this?

26. Bootstrapping Puppet with CloudInit
CloudInit is built
into Ubuntu and
Amazon Linux
• Allows you to
pass bootstrap
parameters in
Amazon EC2
user-data field, in
YAML format

27. Bootstrapping Puppet with CloudInit
Don’t store creds in puppet manifests, store them in private
Amazon S3 buckets
Either pass Amazon S3 creds through CloudInit:
Even better – avoid this by using AWS Identity and Access
Management (IAM) roles and the version of s3cmd in github

28. Bootstrapping Puppet with CloudInit
Built-in puppet support
Use certname with %i for instance id to name the node
Puppetmaster must have auto sign turned on
• Use security groups and/or NACLs for network-level security
In nodes.pp, use regex to match node names

29. Puppet Tips
Use a base class to define your standard install

30. Use runstages
Don’t store credentials in puppet, store them in private
Amazon S3 buckets
• Use AWS IAM to secure the credentials bucket/folders within that
bucket
Puppet Tips

31. Puppet Tips
Use puppet only for configuration files and what makes your
apps unique
For undifferentiated parts of apps, use Amazon S3 backed
RPM/Debian repositories
• Can be either public or private repos, depending on your needs
• Amazon S3 Private RPM Repos: http://git.io/YAcsbg
• Amazon S3 Private Debian Repos: http://git.io/ecCjWQ

32. Puppet Tips
By using packages for applications deploys, you can set ensure
=> latest, and just bump the package in the repo to update
Log everything with rsyslog/graylog/loggly/NewRelic/splunk

33. Scaling the Puppet Masters
Use an Auto Scaling group for puppet masters
• Min size => 2, use multiple Availability Zones
Either have them build themselves off of existing puppet
masters in the group or off packages storied in Amazon S3 and
bootstrapped through user-data
Auto-sign must be on

34. Sites
Communications
Ad Targeting
Ops Tools
Analytics
Apps
Micro-targeting
Micro-listening
Reporting
Registrations
Volunteer
Coordination
Etc, etc, etc.

35. Technology Choice
Polyglot Development
Cloud Hosting
Diverse, App-centered
Databases
SOA, queue-based system
integrations
Expected Tradeoff
More Complex Ops
Less Infra Control,
performance
More Complex Ops,
Fragility, Data Corruption
Dev Complexity, slower
system performance

36. Technology Choice
Polyglot
Development
Cloud Hosting
Diverse, App-
centered Databases
SOA, queue-based
system integrations
Expected Tradeoff
More Complex
Ops
Less Infra Control,
performance
More Complex
Ops, Fragility,
Data Corruption
Dev Complexity,
slower system
performance
Upside
Build as little as
possible, rev-1 faster,
reuse dev skills
Scale, Speed, Cost
Heterogeneous
Resilience, right
tools for the job
Scalability,
serviceability,
operational
flexibility, and
substantially faster
in aggregate

37. $5.2B retail business
7,800 employees
A whole lot of servers
2003

38. 2012
Every day,
AWS adds enough
server capacity to
power this $5B
enterprise
$5.2B retail business
7,800 employees
A whole lot of servers
2003

39. 2012
Amazon Simple
Queuing Service
(SQS)
Thousands of customers
A whole lot of servers
Over 5 Billion Queued
Events
2006-8

40. 2012
OFA
Produced 8.4 Billion
Amazon SQS Queued
Events
Amazon Simple
Queuing Service
(SQS)
Thousands of customers
A whole lot of servers
Over 5 Billion Queued
Events
2006-8

41. 2012
OFA
Produced 8.4 Billion
Amazon SQS Queued
Events
Just the last month of
the campaign
2006-8
Amazon Simple
Queuing Service
(SQS)
Thousands of customers
A whole lot of servers
Over 5 Billion Queued
Events

44. No time to waste

45. This applies to lots of services!
Elastic Load Balancing
Amazon ElastiCache
Amazon RDS
Amazon CloudSearch
Amazon Route53
Amazon S3
Amazon CloudFront
Amazon DynamoDB
You can mostly
do these on your
own…
But do you have extra:
focus, expertise, time, research,
money, risk-tolerance, staff, dedication to
innovate, operations coverage, scalability in design...

46. Looks pretty simple.
Inserts 7.5m records in Amazon DynamoDB, in 8 minutes

47. One thing that is difficult to prepare for…

48. No pressure…

49. They had this built for the previous 3
months, all on the East Coast.

50. They had this built for the previous 3
months, all on the East Coast.
We built this
part in 9 hours
to be safe.
AWS +
Puppet +
Netflix Asgard +
CloudOpt +
DevOps =
Cross-Continent Fault-
Tolerance On-Demand

53. Replication across the continent..
http://tsunami-udp.sourceforge.net/
478.18 Mbps
cross-continental data transit
rate for a single cc2.8xlarge
instance
1.72 Tb an hour
27 Tb of data to move
3.92 Hours
required to move the data
across the continent with
four cc2.8xlarge instances

54. So what did they learn?
HA in Depth: Amazon S3 static pages, de-coupled UI,
jekyll/hyde
Game Day: Practice failures so you know what to do.
( http://www.awsgameday.com )
Loose-Coupling: Ops easy, scale easy, test easy, fix easy…
Fail-Forward: features, quality, and focus are all critical.
Cloud works.

55. We showed it to the world at re: Invent 2012

56. together with the OFA DevOps crew

57. We presented in Tokyo…

58. Born from the Campaign

59. What will you do next?

60. Maybe look at some of their Ruby code?

61. Register Now!
reinvent.awsevents.com
$200 Off Discount Code:
Zoltan2013
Gain New Skills & Knowledge
Choose from 175+ technical sessions,
training bootcamps, hands-on labs,
and hackathons.
Dive Deeper into AWS
Dive deep into foundational AWS
services and learn about the latest
services and features.
Get Your Questions Answered
Get your technical questions answered
by AWS architects, engineers, and
product leads.
Learn Best Practices
Discover best practices, tips and
tricks, and lessons learned from
expert customers.

62. Thank you!
Questions?
• Come talk to an AWS Solutions Architect at Table 22
Contact me!
• @leozh
• leo@amazon.com