5. Infrastructure
"It is common to think in terms of individual machines
rather than view an entire infrastructure as a
combined whole"
“A good infrastructure, whether departmental,
divisional, or enterprise-wide, is a single loosely-
coupled virtual machine, with hundreds or
thousands of hard drives and CPU's.”
-- Bootstrapping an Infrastructure USENIX LISA ’98
http://www.infrastructures.org/papers/bootstrap/bootstrap.html
6. .... as code!
• Programmatically provision and
configure
• Treat like any other code base
• Reconstruct operations from code
repository, data backup, and bare
metal resources.
http://www.flickr.com/photos/louisb/4555295187/
7. Considerations
• Infrastructure changes over time
• Entropy
• Changing business requirements
http://www.flickr.com/photos/seatbelt67/502255276/
10. Manual
Configuration
• Labor intensive
• Error prone
• Hard to reproduce
• Unsustainable
http://www.flickr.com/photos/pureimaginations/4805330106/
11. Scripting
• Typically very brittle
• Throw away, one off scripts
• grep sed awk perl
• curl | bash
http://www.flickr.com/photos/40389360@N00/2428706650/
12. File
Distribution
• NFS mounts
• rdist
• scp-on-a-for-loop
• rsync on cron
http://www.flickr.com/photos/walkadog/4317655660
13. This used to be
awesome
for i in `cat servers.txt` ; do scp ntp.conf root@$i:/etc/
ntpd.conf ; done
for i in `cat servers.txt` ; do ssh root@$i /etc/init.d/ntpd
restart ; done
for i in `cat servers.txt` ; do ssh root@$i chkconfig ntpd
on ; done
• ^ does not scale
http://www.flickr.com/photos/alexerde/3479006495
14. Declarative
Syntax
• Define policy
• Say what, not how
• Abstract interface to
resources
• Enables some interesting
behavior
http://www.flickr.com/photos/bixentro/2591838509/
16. Declarative Syntax
package "ntp" do
action :install
Idempotence
•
end
cookbook_file "/etc/ntp.conf" do
You’ll hear this a lot
source "ntp.conf"
owner "root" • Property of declarative
group "root" interface
mode 0644
action :create
notifies :restart, “service[ntpd]”
• Eliminates brittleness of
end
scripting
service "ntpd" do
action [:enable,:start]
• Identity function: f(x)=x
end
17. Declarative Syntax
while true do
package "ntp" do
action :install
Idempotence
•
end
cookbook_file "/etc/ntp.conf" do
You’ll hear this a lot
source "ntp.conf"
owner "root" • Property of declarative
group "root" interface
mode 0644
action :create
notifies :restart, “service[ntpd]”
• Eliminates brittleness of
end
scripting
service "ntpd" do
action [:enable,:start]
• Identity function: f(x)=x
end • Safe to repeat
end
18. Declarative Syntax
Convergence
• Agents “converge” a system to
desired state
• Repetition inches closer to
desired state
• It eventually gets there
• SCIENCE!
http://www.flickr.com/photos/tolomea/4852616645/
19. Declarative Syntax
Convergence
service "ntpd" do
action [:enable,:start]
ignore_failure true
end
cookbook_file "/etc/ntp.conf" do • Agents “converge” a system to
source "ntp.conf" desired state
owner "root"
group "root"
mode 0644 • Repetition inches closer to
action :create desired state
•
notifies :restart, “service[ntpd]”
ignore_failure true It eventually gets there
•
end
SCIENCE!
package "ntp" do
action :install
ignore_failure true
end
20. Declarative Syntax
# echo “boom” > /etc/ntp.conf ;
chef-client
Convergence
$ grep server /etc/ntp.conf | head -n 1
us.pool.ntp.org
$ ps -e | grep ntp
• Fights entropy, unauthorized
1799 ? 00:00:00 ntpd
changes, and gingivitis
# /etc/init.d/ntpd stop ; chef-client • Update function inputs to deal
with changing requirements
ps -e | grep ntp
1822 ? 00:00:00 ntpd
21. Config Generation
• Often made by hand (still!?)
• Stop that.
• Generate them based on
database content
• Infrastructures evolve
http://www.flickr.com/photos/jabella/4753170413/
33. Generate configs
• Centralized generation
• Version control!
• Distribute with packages, Chef,
git, whatever.
http://www.flickr.com/photos/ssoosay/5126146763/
34. Generate configs
• Local generation directly on nodes
• Reduces management complexity
• No need to distribute
• Version control the programs instead
http://www.flickr.com/photos/ssoosay/5126146763/
36. All That Stuff
• Declarative interface to resources
• Database of nodes and their roles
• Grab remote configs
• Generate configs locally
37. and more!
• Data Driven Infrastructure
• Use APIs to obtain data
• chef-server, SQL, anything.
• Feed resources parameters
• IPs, FQDNs, memory sizes,
• Templates, package, firewall
rules
38. Architecture
• Code Repository
• Chef Server
• Chef Clients
• Data Bags
• Recipes and Cookbooks
• Roles and Run Lists
http://www.flickr.com/photos/boedker/3871267007
39. Code Repository
• Version control
• Development
workflows
• Sharing is Caring
40. Chef Server
Server
Server
chef-server
Server
Server
• Upload from laptop
with knife
RESTful API
Cookbook
Cookbook
Cookbook
Data Bag
Knife Knife
Role
Knife
41. Chef Clients
Server
Server
chef-server
Server
Server
Knife
• Clients are API users
• Read RESTful API Knife
• Write
• Search
chef-client
chef-client chef-client chef-client chef-client
42. Chef Clients
Server
Server
someara.pub chef-server
Server
Server
jtimberman.pub
node5.fqdn.pub
Knife someara.pem
• Clients are API users
• Public keys on server RESTful API Knife jtimberman.pem
• Private keys local to
machines
chef-client
chef-client chef-client chef-client chef-client
node5.fqdn.pem
43. Run Lists
Server
Server
chef-server
Server
Server
Ohai!
API chef-client
Give me
recipe[ntp::client]
ntp
node
client.rb
44. Run Lists
Server
Server
chef-server
Server
Server
Ohai!
chef-client
API Give me
“ntp::client”,
ntp “openssh::server”
openssh
node
client.rb
server.rb
45. Run Lists
Server
Server
chef-server
Server
Server
Ohai!
chef-client Give me
API
“recipe[ntp::client]”,
ntp “recipe[openssh::server]”,
“recipe[apache]”,
openssh
node “recipe[php]”
client.rb apache
server.rb php
default.rb
default.rb
46. Roles
Server
Server
chef-server
Server
Server
Role Recipe
API
Role
Role Recipe
Role Recipe
Recipe
Knife
Recipe
Recipe
Recipe
47. Roles
Server
Server
chef-server
Server
Server
chef-client Ohai!
API
Give me
ntp “role[base]”,
“role[webserver]”
openssh
node
client.rb apache
server.rb php
default.rb
default.rb
48. Roles
Server
Server
chef-server
Server
Server
ntp
openssh
chef-client
API client.rb apache
php
server.rb “role[webserver]”
default.rb
ntp default.rb node
openssh
chef-client
client.rb mysql
server.rb
server.rb
“role[database]”
node
49. Bootstrapping
nodes
• Get chef-client installed
• Write run list to a file
• “Press go”
http://www.flickr.com/photos/liftarn/1447521121/
51. Bootstrapping nodes
{
"kernel": {
"machine": "x86_64",
"name": "Darwin",
"os": "Darwin",
"version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010;
•
root:xnu-1504.7.4~1/RELEASE_I386",
Ohai generates a JSON },
"release": "10.4.0"
attributes list "platform_version": "10.6.4",
•
"platform": "mac_os_x",
Run list and attributes are "platform_build": "10F569",
"domain": "local",
combined into a Node object "os": "darwin",
"current_user": "mray",
•
"ohai_time": 1278602661.60043,
Can be viewed and "os_version": "10.4.0",
"uptime": "18 days 17 hours 49 minutes 18 seconds",
searched through API "ipaddress": "10.13.37.116",
"hostname": "morbo",
"fqdn": "morbomorbo.local",
"uptime_seconds": 1619358
}
52. Bootstrapping nodes
• Run list is requested
• Cookbooks downloaded
• Recipes executed
• Node saved to chef-server
http://www.flickr.com/photos/architopher/457885721
53. Cookbooks
and Recipes
• Cookbooks contain recipes
• And everything they need to
work
• Templates, files, custom
resources, etc
http://www.flickr.com/photos/shutterhacks/4474421855/
54. Cookbooks
$ tree -a cookbooks/haproxy/
README.md
attributes
default.rb
• Cookbooks contain recipes metadata.rb
• And everything they need to recipes
work app_lb.rb
default.rb
• Templates, files, custom templates
resources, etc default
haproxy-app_lb.cfg.erb
haproxy-default.erb
haproxy.cfg.erb
55. Recipes
package "haproxy" do
action :install
end
template "/etc/default/haproxy" do
source "haproxy-default.erb"
• Recipes contain lists of owner "root"
group "root"
resources mode 0644
notifies :restart, "service[haproxy]"
end
service "haproxy" do
action [:enable, :start]
end
57. Resources
package "apache2" do
version "2.2.11-2ubuntu2.6"
action :install
end
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
end
58. Resources
package "apache2" do
•
version "2.2.11-2ubuntu2.6"
Have a type action :install
end
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
end
59. Resources
package "apache2" do
•
version "2.2.11-2ubuntu2.6"
Have a type action :install
•
end
Have a name
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
end
60. Resources
package "apache2" do
•
version "2.2.11-2ubuntu2.6"
Have a type action :install
•
end
Have a name
•
template "/etc/apache2/apache2.conf" do
Have parameters source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
end
61. Resources
package "apache2" do
•
version "2.2.11-2ubuntu2.6"
Have a type action :install
•
end
Have a name
•
template "/etc/apache2/apache2.conf" do
Have parameters source "apache2.conf.erb"
owner "root"
• Take action to put the resource group "root"
mode 0644
in the declared state action :create
end
63. Searching
• All object in Chef server are
indexed by Solr
http://www.flickr.com/photos/fotos_medem/3399096196/
64. Searching
• All object in Chef server are
indexed by Solr
• Can search through the API
http://www.flickr.com/photos/fotos_medem/3399096196/
65. Searching
• All object in Chef server are
indexed by Solr
• Can search through the API
• From knife and in recipes
http://www.flickr.com/photos/fotos_medem/3399096196/
66. Searching
• All object in Chef server are
indexed by Solr
• Can search through the API
• From knife and in recipes
• Returns an array of JSON Node
objects
http://www.flickr.com/photos/fotos_medem/3399096196/