Introduction to Chef by Ned Harris - Presented at SoCalDevOpsUserGroup

1. Introduction to Infrastructure as
Code & Automation / Introduction to
Chef
Ned Harris, Solution Architect
Chef

2. http://www.flickr.com/photos/michaelheiss/3090102907/
Complexity

3. Managing Complexity
• SSH, make with the typey typey

4. Managing Complexity
• SSH, make with the typey typey
• Keep notes in ~/server.txt

5. Managing Complexity
• SSH, make with the typey typey
• Keep notes in ~/server.txt
• Move notes to the wiki

6. Managing Complexity
• SSH, make with the typey typey
• Keep notes in ~/server.txt
• Move notes to the wiki
• Custom scripts (setup.sh)

7. Managing Complexity
• SSH, make with the typey typey
• Keep notes in ~/server.txt
• Move notes to the wiki
• Custom scripts (setup.sh)
• Golden Images

8. Golden Images are not the answer
• Gold is heavy
• Hard to transport
• Hard to mold
• Easy to lose
configuration detail
http://www.flickr.com/photos/garysoup/2977173063/

9. Typical Infrastructure
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
Jboss App

10. New Compliance Mandate!
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
• Move SSH off port 22
• Lets put it on 2022

11. 6 Golden Image Updates
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
• edit /etc/ssh/sshd_config
1 2
3
4
5
6

12. 12 Instance Replacements
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
• Delete, launch
1 2
3 4 5 6 7
8 9
10 11
12
• Repeat
• Typically manually

13. Done in Maintenance Windows
• Don’t break anything!
• Bob just got fired =(
5
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite 1 2
4 5 6 7
8 9
10 11
12
3
12

14. Different IP Addresses?
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
• Invalid configs!

15. Managing Complexity
• SSH, make with the typey typey
• Keep notes in ~/server.txt
• Move notes to the wiki
• Custom scripts (setup.sh)
• Golden Images
• Policy-driven configuration management

16. Policies
• Declarations about the state of thing in a system
• applied repeatedly and repair the system when needed
• often change

17. Repeatable Operations
• Idempotent
• can be applied an infinite number of times and yield the same result every time
• Convergent
• test state and repair if needed

18. Policy Evolves

19. Following Policy
• A control loop keeps the system stable and allows for change when policy is
updated

20. Policy Evolves
That's great and all, !
but tell me about !
Chef!!

21. What is Chef?
• Framework for managing complexity
• Infrastructure as code
• a domain-specific language (DSL) for describing convergent operations
• A community of professionals
• A company

22. The Chef Software Platform
Chef
Development Kit
Cookbook and
Policy Authoring
Test-Driven
Infrastructure
Chef Server
Management
Console
Analytics
Platform
High Availability
and Replication
Chef
Client
Nodes
Data
Center
The
Cloud

23. How does Chef work?
• Ensure desired state by continually testing and repairing individual resources in
the system
• Compose policies using a series of abstractions

24. Desired Configuration
Node
Chef Server
What policy should I follow?

25. Desired Configuration
Node
Chef Server
What policy should I follow?
"recipe[ntp::client]"
"recipe[users]"
"role[webserver]"

26. Desired Configuration
Chef Server
What policy should I follow?
"recipe[ntp::client]"
"recipe[users]"
"role[webserver]"

27. Recipes
package "apache2"resource one

28. Recipes
package "apache2"
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "root"
group "root"
mode "0644"
variables(:allow_override => "All")
notifies :reload, "service[apache2]"
end
resource one
resource two

29. Recipes
package "apache2"
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "root"
group "root"
mode "0644"
variables(:allow_override => "All")
notifies :reload, "service[apache2]"
end
service "apache2" do
action [:enable,:start]
supports :reload => true
end
resource one
resource two
resource three

30. Recipes
resource one

31. Recipes
resource one
resource two

32. Recipes
resource one
resource two
resource three

33. Built-in Resources
• package
• template
• service
• cron
• directory
• mount
• user
• group
• registry_key
• remote_directory
• route
• ...and many more!

35. Managing Complexity
• Organizations
• Environments
• Roles
• Nodes
• Recipes
• Cookbooks
• Search
• Data

36. Their Infrastructure
Organizations
My Infrastructure Your Infrastructure

37. Environments
Development Staging Production

38. Roles
Load Balancers
Application Servers
DB Cache
Database

39. Nodes

40. Search
• Search for nodes with Roles
• Find Topology Data
• IP addresses
• Hostnames
• FQDNs
http://www.flickr.com/photos/kathycsus/2686740

41. Search for Nodes
pool_members
=
search("node","role:webserver")
template
"/etc/haproxy/haproxy.cfg"
do
source
"haproxy-­‐app_lb.cfg.erb"
owner
"root"
group
"root"
mode
0644
variables
:pool_members
=>
pool_members.uniq
notifies
:restart,
"service[haproxy]"
end

42. Webservers
HAProxy Configuration
HA Proxy

43. Webservers
HAProxy Load Balancer
HA ProxyEnterprise
Chef
pool_members =
search("node","role:webserver")

44. Webservers
HAProxy Load Balancer
HA ProxyEnterprise
Chef
Webservers?
pool_members =
search("node","role:webserver")

45. Webservers
HAProxy Load Balancer
HA ProxyEnterprise
Chef
Webservers?
pool_members =
search("node","role:webserver")

46. Webservers
HAProxy Load Balancer
HA ProxyEnterprise
Chef
Webservers?
pool_members =
search("node","role:webserver")
{
"web01" : {
"hostname" : "web01",
"ipaddress" : "10.1.1.1"
},
"web02" : {
"hostname" : "web02",
"ipaddress" : "10.1.1.2"
},
"web03" : {
"hostname" : "web03",
"ipaddress" : "10.1.1.3"
},
"web04" : {
"hostname" : "web04",
"ipaddress" : "10.1.1.4"
},
"web05" : {
"hostname" : "web05",
"ipaddress" : "10.1.1.5"
},
"web06" : {
"hostname" : "web06",
"ipaddress" : "10.1.1.6"
}
}

47. Webservers
HAProxy Load Balancer
HA ProxyEnterprise
Chef
Webservers?
pool_members =
search("node","role:webserver")
{
"web01" : {
"hostname" : "web01",
"ipaddress" : "10.1.1.1"
},
"web02" : {
"hostname" : "web02",
"ipaddress" : "10.1.1.2"
},
"web03" : {
"hostname" : "web03",
"ipaddress" : "10.1.1.3"
},
"web04" : {
"hostname" : "web04",
"ipaddress" : "10.1.1.4"
},
"web05" : {
"hostname" : "web05",
"ipaddress" : "10.1.1.5"
},
"web06" : {
"hostname" : "web06",
"ipaddress" : "10.1.1.6"
}
}

48. Webservers
HAProxy Load Balancer
HA ProxyEnterprise
Chef
Webservers?
pool_members =
search("node","role:webserver")
{
"web01" : {
"hostname" : "web01",
"ipaddress" : "10.1.1.1"
},
"web02" : {
"hostname" : "web02",
"ipaddress" : "10.1.1.2"
},
"web03" : {
"hostname" : "web03",
"ipaddress" : "10.1.1.3"
},
"web04" : {
"hostname" : "web04",
"ipaddress" : "10.1.1.4"
},
"web05" : {
"hostname" : "web05",
"ipaddress" : "10.1.1.5"
},
"web06" : {
"hostname" : "web06",
"ipaddress" : "10.1.1.6"
}
}
pool_members
{
"web01" : {
"hostname" : "web01",
"ipaddress" : "10.1.1.1"
},
"web02" : {
"hostname" : "web02",
"ipaddress" : "10.1.1.2"
},
"web03" : {
"hostname" : "web03",
"ipaddress" : "10.1.1.3"
},
"web04" : {
"hostname" : "web04",
"ipaddress" : "10.1.1.4"
},
"web05" : {
"hostname" : "web05",
"ipaddress" : "10.1.1.5"
},
"web06" : {
"hostname" : "web06",
"ipaddress" : "10.1.1.6"
}
}

49. Search for Nodes
pool_members
=
search("node","role:webserver")
template
"/etc/haproxy/haproxy.cfg"
do
source
"haproxy-­‐app_lb.cfg.erb"
owner
"root"
group
"root"
mode
0644
variables
:pool_members
=>
pool_members.uniq
notifies
:restart,
"service[haproxy]"
end

50. Pass results into Templates
#
Set
up
application
listeners
here.
listen
application
0.0.0.0:80
balance
roundrobin
<%
@pool_members.each
do
|member|
-­‐%>
server
<%=
member[:hostname]
%>
<%=
member[:ipaddress]
%>:>
weight
1
maxconn
1
check
<%
end
-­‐%>
<%
if
node["haproxy"]["enable_admin"]
-­‐%>
listen
admin
0.0.0.0:22002
mode
http
stats
uri
/
<%
end
-­‐%>

51. Webservers
HAProxy Configuration
HA Proxypool_members
{
"web01" : {
"hostname" : "web01",
"ipaddress" : "10.1.1.1"
},
"web02" : {
"hostname" : "web02",
"ipaddress" : "10.1.1.2"
},
"web03" : {
"hostname" : "web03",
"ipaddress" : "10.1.1.3"
},
"web04" : {
"hostname" : "web04",
"ipaddress" : "10.1.1.4"
},
"web05" : {
"hostname" : "web05",
"ipaddress" : "10.1.1.5"
},
"web06" : {
"hostname" : "web06",
"ipaddress" : "10.1.1.6"
}
}
<% @pool_members.each do |member| -%>
server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check
<% end -%>

52. Webservers
HAProxy Configuration
HA Proxy
<% @pool_members.each do |member| -%>
server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check
<% end -%>
pool_members
{
"web01" : {
"hostname" : "web01",
"ipaddress" : "10.1.1.1"
},
"web02" : {
"hostname" : "web02",
"ipaddress" : "10.1.1.2"
},
"web03" : {
"hostname" : "web03",
"ipaddress" : "10.1.1.3"
},
"web04" : {
"hostname" : "web04",
"ipaddress" : "10.1.1.4"
},
"web05" : {
"hostname" : "web05",
"ipaddress" : "10.1.1.5"
},
"web06" : {
"hostname" : "web06",
"ipaddress" : "10.1.1.6"
}
}
haproxy.cfg
server web01 10.1.1.1 weight 1 maxconn 1 check

53. Webservers
HAProxy Configuration
HA Proxy
<% @pool_members.each do |member| -%>
server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check
<% end -%>
pool_members
{
"web01" : {
"hostname" : "web01",
"ipaddress" : "10.1.1.1"
},
"web02" : {
"hostname" : "web02",
"ipaddress" : "10.1.1.2"
},
"web03" : {
"hostname" : "web03",
"ipaddress" : "10.1.1.3"
},
"web04" : {
"hostname" : "web04",
"ipaddress" : "10.1.1.4"
},
"web05" : {
"hostname" : "web05",
"ipaddress" : "10.1.1.5"
},
"web06" : {
"hostname" : "web06",
"ipaddress" : "10.1.1.6"
}
}
haproxy.cfg
server web01 10.1.1.1 weight 1 maxconn 1 check
server web02 10.1.1.2 weight 1 maxconn 1 check

54. Webservers
HAProxy Configuration
HA Proxy
<% @pool_members.each do |member| -%>
server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check
<% end -%>
pool_members
{
"web01" : {
"hostname" : "web01",
"ipaddress" : "10.1.1.1"
},
"web02" : {
"hostname" : "web02",
"ipaddress" : "10.1.1.2"
},
"web03" : {
"hostname" : "web03",
"ipaddress" : "10.1.1.3"
},
"web04" : {
"hostname" : "web04",
"ipaddress" : "10.1.1.4"
},
"web05" : {
"hostname" : "web05",
"ipaddress" : "10.1.1.5"
},
"web06" : {
"hostname" : "web06",
"ipaddress" : "10.1.1.6"
}
}
haproxy.cfg
server web01 10.1.1.1 weight 1 maxconn 1 check
server web02 10.1.1.2 weight 1 maxconn 1 check
server web03 10.1.1.3 weight 1 maxconn 1 check

55. Webservers
HAProxy Configuration
HA Proxy
<% @pool_members.each do |member| -%>
server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check
<% end -%>
pool_members
{
"web01" : {
"hostname" : "web01",
"ipaddress" : "10.1.1.1"
},
"web02" : {
"hostname" : "web02",
"ipaddress" : "10.1.1.2"
},
"web03" : {
"hostname" : "web03",
"ipaddress" : "10.1.1.3"
},
"web04" : {
"hostname" : "web04",
"ipaddress" : "10.1.1.4"
},
"web05" : {
"hostname" : "web05",
"ipaddress" : "10.1.1.5"
},
"web06" : {
"hostname" : "web06",
"ipaddress" : "10.1.1.6"
}
}
haproxy.cfg
server web01 10.1.1.1 weight 1 maxconn 1 check
server web02 10.1.1.2 weight 1 maxconn 1 check
server web03 10.1.1.3 weight 1 maxconn 1 check
server web04 10.1.1.4 weight 1 maxconn 1 check

56. Webservers
HAProxy Configuration
HA Proxy
<% @pool_members.each do |member| -%>
server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check
<% end -%>
pool_members
{
"web01" : {
"hostname" : "web01",
"ipaddress" : "10.1.1.1"
},
"web02" : {
"hostname" : "web02",
"ipaddress" : "10.1.1.2"
},
"web03" : {
"hostname" : "web03",
"ipaddress" : "10.1.1.3"
},
"web04" : {
"hostname" : "web04",
"ipaddress" : "10.1.1.4"
},
"web05" : {
"hostname" : "web05",
"ipaddress" : "10.1.1.5"
},
"web06" : {
"hostname" : "web06",
"ipaddress" : "10.1.1.6"
}
}
haproxy.cfg
server web01 10.1.1.1 weight 1 maxconn 1 check
server web02 10.1.1.2 weight 1 maxconn 1 check
server web03 10.1.1.3 weight 1 maxconn 1 check
server web04 10.1.1.4 weight 1 maxconn 1 check
server web05 10.1.1.5 weight 1 maxconn 1 check

57. Webservers
HAProxy Configuration
HA Proxy
<% @pool_members.each do |member| -%>
server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check
<% end -%>
haproxy.cfg
server web01 10.1.1.1 weight 1 maxconn 1 check
server web02 10.1.1.2 weight 1 maxconn 1 check
server web03 10.1.1.3 weight 1 maxconn 1 check
server web04 10.1.1.4 weight 1 maxconn 1 check
server web05 10.1.1.5 weight 1 maxconn 1 check
server web06 10.1.1.1 weight 1 maxconn 1 check
pool_members
{
"web01" : {
"hostname" : "web01",
"ipaddress" : "10.1.1.1"
},
"web02" : {
"hostname" : "web02",
"ipaddress" : "10.1.1.2"
},
"web03" : {
"hostname" : "web03",
"ipaddress" : "10.1.1.3"
},
"web04" : {
"hostname" : "web04",
"ipaddress" : "10.1.1.4"
},
"web05" : {
"hostname" : "web05",
"ipaddress" : "10.1.1.5"
},
"web06" : {
"hostname" : "web06",
"ipaddress" : "10.1.1.6"
}
}

58. So when this…
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite

59. …becomes this…
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite

60. …this can happen automaticaly!
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite

61. Count the resources
NagiosGraphite NagiosGraphite
Memcache
Postgres Slaves
• Load balancer config
• Nagios host ping
• Nagios host ssh
• Nagios host HTTP
• Nagios host app health
• Graphite CPU
• Graphite Memory
• Graphite Disk
• Graphite SNMP
• Memcache firewall
• Postgres firewall
• Postgres authZ config
• 12+ resource changes for 1 node addition
Jboss App

62. Build Anything
• Simple internal applications
• Complex external applications
• Workstations
• Hadoop clusters
• IaaS infrastructure
• PaaS infrastructure
• SaaS applications
• Storage systems
• You name it
http://www.flickr.com/photos/hyku/245010680/

63. And Manage it Simply
• Automatically reconfigure
everything
• Linux, Windows, Unixes,
BSDs
• Load balancers
• Metrics collection systems
• Monitoring systems
• Cloud migrations become
trivial
http://www.flickr.com/photos/helico/404640681/

64. ChefDK

65. Increase CHEF adoption through ChefDK
● ChefDK: CHEF Software Development Kit, fully
supported with the Chef Premium Subscription
● Workflow Definition: Our recommendation on the
process to test and verify your infrastructure code
before committing it to source control and shipping
it to production.
● Workflow Enhancement: Based on customer
feedback and use cases

66. ChefDK: In the Box
First Class Support on Windows, Linux, and
OSX for the entire suite of Chef development
tools
● Test Kitchen: Virtualized testing harness
● Berkshelf: Dependency solver
● Chef-Vault: Secrets management
● Rubocop / Foodcritic: Code linting
● Chefspec: In-memory Unit Testing
● Chef.bin: New wrapper binary to tie it all
together, with new extensible cookbook
generators.

67. The Chef workflow
Create new
skeleton
cookbook.
Create a VM
environment for
cookbook
development.
Write/debug
cookbook
recipes
(iterative step).
Perform
acceptance
tests.
Deploy to
production.

68. Questions?