Puppet can be used as a control panel to perform a wide variety of tasks within your infrastructure. It can be used to trigger failover between hot standby servers or entire data centres. It can be used to deploy package updates across large clusters. It can be used to deploy code to staging and then onto production, ensuring the right versions are present. It can be used to replicate production environments locally for all our engineers. And it can even be used for config management! This talk will take a hands on technical look at how we use Puppet to achieve all these things. It will include code samples, hints and tips and explain how the flexibility of Puppet can be used to really control your entire infrastructure.

1. Puppet at the centre of everything

2. David Mytton
Woop Japan!

4. Server Density Architecture
•100 servers - Ubuntu 12.04

5. Server Density Architecture
•100 servers - Ubuntu 12.04
•50% virtual

6. Server Density Architecture
•100 servers - Ubuntu 12.04
•50% virtual
•Nginx, Python, MongoDB

7. Server Density Architecture
•100 servers - Ubuntu 12.04
•50% virtual
•Nginx, Python, MongoDB
•25TB data per month

8. • Config
Use cases

9. • Config
• Failover
Use cases

10. • Config
• Failover
• Code deploys
Use cases

11. • Config
• Failover
• Code deploys
• System updates
Use cases

12. • Base manifest
Config

15. • Base manifest
• Sub manifests
Config

17. • base
• softlayer
• softlayer-dedicated
• rackspace
• ...

18. Failover
• /etc/hosts
<%= metrics2IP %> alerts.honshuu.prod
audit.honshuu.prod auth.honshuu.prod
cloud.honshuu.prod datastore.honshuu.prod
inventory.honshuu.prod ...

23. Failover
• /etc/hosts
108.168.254.33
puppet

25. Dealing with outages
Expected

26. Dealing with outages
Really test your vendors

27. Dealing with outages
Simulations

28. • Commit
Code deploys

30. • Commit
• Build
Code deploys

32. • Commit
• Build
• Deploy
Code deploys

37. /opt/puppet/bin/mco puppetd runonce -I <host>

38. • Commit
• Build
• Deploy
• Puppet run
Code deploys

40. Code deploys
Advantages
APIs

41. Code deploys
Advantages
APIs
Already there

42. Code deploys
Advantages
APIs
Already there
System facts

43. Code deploys
Disadvantages
Slow

44. Code deploys
Disadvantages
Slow
Puppet run already

45. Code deploys
Disadvantages
Slow
Puppet run already
Eventually consistent

46. • Bash script
System updates

47. System updates
• Bash script
• Remote commands

49. System updates
• Bash script
• Remote commands
• Notifications

50. System updates
• Bash script
• Remote commands
• Notifications
• Canaries

52. •Build master (buildbot): VM x2 CPU 2.0Ghz, 2GB
RAM – $89/m
Colo experiment
•Build slave (buildbot): VM x1 CPU 2.0Ghz, 1GB
RAM
– $40/m
•Staging load balancer: VM x1 CPU 2.0Ghz, 1GB
RAM – $40/m
•Staging server 1: VM x2 CPU 2.0Ghz, 8GB RAM
– $165/m
•Staging server 2: VM x1 CPU 2.0Ghz, 2GB RAM
– $50/m
•Puppet master: VM x2 CPU 2.0Ghz, 2GB RAM
– $89/m

54. Colo experiment
•Dell 1U R415
•x2 8C AMD 2.8Ghz
•32GB RAM

55. Colo experiment
•Dell 1U R415
•x2 8C AMD 2.8Ghz
•32GB RAM
•Dual PSU, NIC

56. Colo experiment
•Dell 1U R415
•x2 8C AMD 2.8Ghz
•32GB RAM
•Dual PSU, NIC
•x4 1TB SATA hot swappable

57. Dedicated?

58. Dedicated?
•Hardware replacement

59. Dedicated?
•Hardware replacement
•Managed/support

60. Dedicated?
•Hardware replacement
•Managed/support
•Networking

61. Colo?

62. •Hardware spec/value
Colo?

63. •Hardware spec/value
•Total cost
Colo?

64. •Hardware spec/value
•Total cost
•Internal skills?
Colo?

65. •Hardware spec/value
•Total cost
•Internal skills?
•More fun?!
Colo?

67. •£300-350/kWh/m
•4.5A = £520/m
•9A = £900/m
Colo: Power

68. David Mytton
@davidmytton
david@serverdensity.com
blog.serverdensity.com
Woop Japan!