How we (at Yelp) use Puppet to manage AWS VPCs and what we learned along the way

1. Keeping AWS in check with Puppet
or
“Managing cloud networking with Puppet with a healthy dose of
trolling yourself on the side”
Matt Carroll

2. HELLO.

4. What’s all this about then?
● We use AWS extensively, especially EC2
● We needed a way to organise our AWS networking (VPCs)
● It needs to be:
○ Centrally managed
○ Reproducible
○ Declarative (idempotent)
○ Ideally not another solution on top of a stack of solutions

5. Spoiler
We (Tom) decided to use Puppet
We wrote a set of types and providers to use
with the AWS API

6. What I’m going to cover
YOU ARE HERE
1. Intro
2. AWS networking basics
3. Why we chose Puppet
4. How it actually works
5. Some of the “interesting” things we learned
6. Summary
7. Questions

7. What I hope you’ll get from this
● A sense of Puppet as an extensible framework that can
manage dependencies in external APIs
● Nodes are just units of computing capacity - an operation
does not have to be a subset of a node
● Some insight into the types and providers system
● Some of the strange things we learned along the way
… and as always...

8. To serve as a lesson to others.

9. SOME AWS BASICS.

10. Some
of this
isn’t
exactly
retina
resolution

11. The (simplified) Hierarchy
Account
Region
VPC
Routetable
Subnet

12. The (needlessly complicated simplified)
Hierarchy
Account:
• Region
• dopt: DHCP Options
• vpc: Virtual Private Cloud
• igw: Internet Gateway
• cgw: Customer Gateway
• vgw: Virtual Gateway
• vpn: VPN
• routetable
• subnet

13. How do we make all these objects?
Just make them in the console!

14. How do we make all these objects?
IF YOU HATE YOURSELF
Just make them in the console!

15. How do we actually make all of these objects?
• AWS::SDK!
• We’ll have to ensure that the resources get created (and
purged)...
• Idempotently…
• With all their dependencies…
• Remind you of something?

16. USE PUPPET!

18. Create Puppet types for VPC objects!
● Resources for each of the objects
○ All API calls made on this level
○ Will contain all code for reading and creating individual
resources
○ No dependencies other than autorequires in the types -
those in the hierarchy earlier
● Business logic in manifests
○ For your site-specific dependencies and network structure
○ e.g. we have a separate VPC for each environment
● Data in hiera

19. (Actually though) Why Puppet?
• With the ability to query and create and modify objects
through the API we can state them declaratively
• We can thus create resources which can be included or
purged idempotently
• Rather than specify order, we can state dependencies
and allow Puppet to figure order out

20. Why Puppet?
Using this model you can even collect dependencies and
resources:
• Make all resources of a type with a parameter evaluate after
another
• Control purge and no-op of all aws resources

21. Why Puppet?
Admittedly it seems a little asymmetrical:
• Puppet runs on nodes which creates
resources in AWS
• AWS networking is not a subset of a node
• Nodes just act as executors for creating AWS
resource
But actually this plays to our advantage (more on
this later)

22. HOW IT WORKS.

23. Puppet Types
• The bit that you specify in
the manifest
• Really just a DDL for the
resource’s metadata
• The “front end” for the
pluggable “back end” (the
provider)
• Interface to specify all
properties and parameters

24. Puppet Providers
• The business end
• This is where the API
code lives
• Quite a lot longer
• Handles all application
and querying of the
resource including
prefetch

25. What’s a “prefetch”?
• Some resources are
expensive to read, so
you read them all once
when you first come
across one
• This is done before the
catalog is completely
compiled

26. Manifest examples

27. Associated Infrastructure
• How do you handle multiple accounts?
• Dedicated AWS admin box within each
account to apply the resources on
• IAM roles to handle credentials
• Logging resource changes separately
• Why not do it in the code...?

28. LESSONS LEARNED.
(the hard way)

29. Don’t
do
what
Donny
Don’t
does

30. Why not put credentials in the code?

31. Why not put credentials in the code?
• We (I) tried to make an
aws_credentials type
• This requires access to the
catalog in the prefetch phase so
other resources can query it
• You ALSO need to guarantee
type evaluation order and
access to credentials in the
catalog so that you can use
prefetch

32. Why not put credentials in the code?
• The instances method suddenly
requires an argument
• In fact, so does anything that
isn’t an instance method
• Actually ended up copying and
pasting the resources resource
to aws_resources and adding in
a “credentials” parameter

33. On second thoughts...

34. Prefetch isn’t… Exceptional
• https://tickets.puppetlabs.com/browse/PUP-3656
• This means that if anything goes awry in prefetch, puppet will
swallow it
• In our case, we hit the API limit occasionally, meaning we got
duplicate resources

35. So we did a bad thing...

36. Asynchronous APIs tell lies
• When you create an object you get
a 200 OK
• This doesn’t mean “I’ve done it”,
this means “I’ll do it”
• In a dependency chain, this can
mean that a resource is about to be
created, but when it’s checked by
another resource it’s not there yet
• In the Puppet paradigm it’s best
just to run until convergence.

37. SO IN SUMMARY...

38. What I learned (the hard way)
• The “resources” resource
• Is hard to understand unless
you read the code (type only)
because Googling it is
impossible.
• Could do with being able to
apply other arbitrary
properties and parameters?*
*this may be an awful idea

39. What I learned (the hard way)
• Instances vs. class methods paradigm gets very confusing
because it is tied to applying catalogue vs. prefetch
• Prefetch is generally pretty confusing when you add in
dependencies TO the prefetch

40. What really cool things I learned
• You can reproduce AWS networking (and other objects
thanks to other contributors!)
• A really cool insight into the types and providers system and
how it could grow in the future
• Learning to treat servers not like pets OR cattle but as a
medium by which you do useful stuff
… and you could do this with a whole bunch of
APIs

41. THANKS FOR LISTENING!
github.com/bobtfish/puppet-aws_api
Matt Carroll
SRE at Yelp
@oholiab
mattc@yelp.com
yelp.com/careers (we’re hiring)