Delivered at Chef-Boston meetup on June 13, 2013.

1. Chef Cookbook Testing and
Continuous Integration
Julian C. Dunn
Senior Consultant, Opscode, Inc.
<jdunn@opscode.com>
Friday, June 14, 13

2. Why Write Cookbook Tests?
• Catch development errors
• Catch regressions
• Code to requirements (TDD, BDD)
• Smaller batch sizes == lower blast radius
• Continuous delivery of cookbooks!
• Build confidence that your infrastructure is
working correctly
Friday, June 14, 13

3. Software Development
Lifecycle for
Sysadmins
Friday, June 14, 13

4. Devops Is A Two-Way Street
• It’s great when
developers care
about
• uptime!
• scaling!
• deployment!
• Put them on call!
etc. etc. etc.
Friday, June 14, 13

5. Devops Is A Two-Way Street
• Systems Administrators
also have as much or more to
learn from developers as well!
Friday, June 14, 13

6. Source
Code
Compiler
Artifact
Test
Typical Development Workflow
• Typical Software Development Workflow:
• Write Source
• Compile Source
• Deploy Artifacts
• Write Tests
• Run Tests
Friday, June 14, 13

7. Source
Code
Compiler
Artifact
Test
Typical Development Workflow
• Source Code
• The recipe for a computer program
• Edited directly
• Managed via Source Control Software
Friday, June 14, 13

8. Source
Code
Compiler
Artifact
Test
Typical Development Workflow
• Compiler
• Takes source code and converts it into
executable programs
Friday, June 14, 13

9. Source
Code
Compiler
Artifact
Test
Typical Development Workflow
• Artifact
• Artifacts are executable programs
created by compilers.
• Compiled artifacts cannot be edited
directly. Source code must be changed
and re-compiled to produce a new build
artifact.
Friday, June 14, 13

10. Source
Code
Compiler
Artifact
Test
Typical Development Workflow
• Testing
• Write tests to verify that code works as
intended
• Run tests at different stages of the code
lifecycle to ensure correctness
Friday, June 14, 13

11. Source
Code
Compiler
Artifact
Test
Typical Development Workflow
• When developing software, most time
isn’t actually spent coding
• 10-second changes to source code can
take minutes to vet
• Compiling code
• Deploying code
• Writing & Running tests
Friday, June 14, 13

12. Friday, June 14, 13

13. Source
Code
Compiler
Artifact
Test
Typical Development Workflow
Fun!
Boring!
• Things that are fun:
• Designing programs!
• Writing source code!
• Things that are boring:
• Compiling code
• Deploying artifacts
• Running tests
Friday, June 14, 13

14. Look familiar to any Chefs?Friday, June 14, 13

15. Worksta(on:+
Knife+cookbook+
create+
Worksta(on:+
Edit+cookbook+
Worksta(on:+
Knife+cookbook+
upload+
Provision+target+
Bootstrap+
target+
Worksta(on:+
Edit+target+run+
list+
ssh+target+
Target:+Run+
chef>client+
Traditional Cookbook Development: First Chef Run
Friday, June 14, 13

16. Worksta(on:+
edit+cookbook+
Worksta(on:+
knife+cookbook+
upload+
Target:+Run+
chef9client+
Traditional Cookbook Development: Subsequent Runs
Friday, June 14, 13

17. Source'Code'
Compiler'
Ar/fact'
Test'
Boooooooring
• Too much time doing
“paperwork”
• vi recipes/something.rb
• knife cookbook upload
• sudo pkill -USR1 chef-
client
• #%$#%$ something
broke, let me do that all
again
• Not enough time doing fun
stuff!
• Writing recipes
Less Fun!
More Boring!
X
Friday, June 14, 13

18. Source'Code'
Compiler'
Ar/fact'
Test'
Boooooooring
• Too much time doing
“paperwork”
• vi recipes/something.rb
• knife cookbook upload
• sudo pkill -USR1 chef-
client
• #%$#%$ something
broke, let me do that all
again
• Not enough time doing fun
stuff!
• Writing recipes
Less Fun!
More Boring!
XTH
IS
SUCKS!
Friday, June 14, 13

19. Chefs on a Plane
What if...
Friday, June 14, 13

20. Chefs on a Plane
What if...Worksta(on:+
Knife+cookbook+
create+
Worksta(on:+
Edit+cookbook+
Worksta(on:+
Knife+cookbook+
upload+
Provision+target+
Bootstrap+
target+
Worksta(on:+
Edit+target+run+
list+
ssh+target+
Target:+Run+
chef>client+
Target:(Run(
chef/client(
Worksta6on:(
edit(cookbook(
Worksta6on:(
knife(cookbook(
upload(
...we could
automate all
of this...
Friday, June 14, 13

21. Chefs on a Plane
What if...Worksta(on:+
Knife+cookbook+
create+
Worksta(on:+
Edit+cookbook+
Worksta(on:+
Knife+cookbook+
upload+
Provision+target+
Bootstrap+
target+
Worksta(on:+
Edit+target+run+
list+
ssh+target+
Target:+Run+
chef>client+
Target:(Run(
chef/client(
Worksta6on:(
edit(cookbook(
Worksta6on:(
knife(cookbook(
upload(
...we could
automate all
of this...
...to run entirely on this...
Friday, June 14, 13

22. Chefs on a Plane
What if...
...even while
aboard this?
Worksta(on:+
Knife+cookbook+
create+
Worksta(on:+
Edit+cookbook+
Worksta(on:+
Knife+cookbook+
upload+
Provision+target+
Bootstrap+
target+
Worksta(on:+
Edit+target+run+
list+
ssh+target+
Target:+Run+
chef>client+
Target:(Run(
chef/client(
Worksta6on:(
edit(cookbook(
Worksta6on:(
knife(cookbook(
upload(
...we could
automate all
of this...
...to run entirely on this...
Friday, June 14, 13

23. Source'Code'
Compiler'
Ar/fact'
Test'
Developing for Chef: Rapid Iteration
• Less time waiting around
for cookbook deploys and
Chef runs
• More frequent testing
• Better code
• Business needs met
more quickly
More Fun!
Less Boring!
X
Friday, June 14, 13

24. Code Can (Help)
Friday, June 14, 13

25. The Toolchain
Friday, June 14, 13

26. The Toolchain
• You’re becoming a developer!
• Reasonably powerful computer
• Good editor
Friday, June 14, 13

27. Have a Good Computer
• Virtualization is used to run
acceptance tests
• Running on a real virtualized
“node”
• Lots of memory (4GB+, 8GB
recommended)
• Fast disk (SSD)
• Good processor (Intel i5+)
• Modern OS
X
Friday, June 14, 13

28. Tools
• Chef (obviously)
• Virtualization provider
• Virtualization automation
(Vagrant)
• Cookbook dependency
manager (Berkshelf,
Librarian)
• Unit and acceptance testing
frameworks
Friday, June 14, 13

29. Software Testing
Terminology and Chef
Friday, June 14, 13

30. Different Types of Testing
• There are formal
Software Engineering
definitions for testing
Friday, June 14, 13

31. Different Types of Testing
• There are formal
Software Engineering
definitions for testing
• Unit test
Friday, June 14, 13

32. Different Types of Testing
• There are formal
Software Engineering
definitions for testing
• Unit test
• Integration test
Friday, June 14, 13

33. Different Types of Testing
• There are formal
Software Engineering
definitions for testing
• Unit test
• Integration test
• Acceptance test
Friday, June 14, 13

34. Different Types of Testing
• There are formal
Software Engineering
definitions for testing
• Unit test
• Integration test
• Acceptance test
• An easy explanation for
SAs is ...
Friday, June 14, 13

35. • Unit Test: Signal Input
What and When To Test
Flickr user: Rain Rabbit
Friday, June 14, 13

36. • Unit Test: Signal Input
• Did I send Chef the correct
command?
What and When To Test
Flickr user: Rain Rabbit
Friday, June 14, 13

37. • Unit Test: Signal Input
• Did I send Chef the correct
command?
• Signal Processing
What and When To Test
Flickr user: Rain Rabbit
Friday, June 14, 13

38. • Unit Test: Signal Input
• Did I send Chef the correct
command?
• Signal Processing
• Did Chef interpret my command
correctly?
What and When To Test
Flickr user: Rain Rabbit
Friday, June 14, 13

39. • Unit Test: Signal Input
• Did I send Chef the correct
command?
• Signal Processing
• Did Chef interpret my command
correctly?
• Acceptance Test: Signal Output
What and When To Test
Flickr user: Rain Rabbit
Friday, June 14, 13

40. • Unit Test: Signal Input
• Did I send Chef the correct
command?
• Signal Processing
• Did Chef interpret my command
correctly?
• Acceptance Test: Signal Output
• Did my expressed intent, executed
by Chef, achieve the desired
result?
What and When To Test
Flickr user: Rain Rabbit
Friday, June 14, 13

41. Chef Testing Tools
Friday, June 14, 13

42. Chef Testing Tools
• Signal Input
• ChefSpec
Friday, June 14, 13

43. Chef Testing Tools
• Signal Input
• ChefSpec
• Signal out
• Chef Minitests & Minitest Handler
Friday, June 14, 13

44. Chef Testing Tools
• Signal Input
• ChefSpec
• Signal out
• Chef Minitests & Minitest Handler
• Signal processing
• Who cares; that’s what Opscode’s own tests for
Chef are for!
Friday, June 14, 13

45. General Philosophy for Chef Testing
Friday, June 14, 13

46. General Philosophy for Chef Testing
• Unit tests
• ChefSpec assertion for every resource of interest
• Each recipe should get its own test file
Friday, June 14, 13

47. General Philosophy for Chef Testing
• Unit tests
• ChefSpec assertion for every resource of interest
• Each recipe should get its own test file
• Acceptance tests
• Anything that you couldn’t test in a unit test
Friday, June 14, 13

48. General Philosophy for Chef Testing
• Unit tests
• ChefSpec assertion for every resource of interest
• Each recipe should get its own test file
• Acceptance tests
• Anything that you couldn’t test in a unit test
• Don’t repeat yourself!
Friday, June 14, 13

49. Unit Testing: ChefSpec
Friday, June 14, 13

50. ChefSpec Overview
• Built on top of RSpec
• Converge a Chef run in memory
• Overrides all providers to take no action
• Make assertions about the in-memory Chef run
• Mock Ohai data (automatic attributes) with Fauxhai
Friday, June 14, 13

51. Example Recipe Part One
include_recipe “java”
user node['sauceproxy']['server']['user'] do
comment "SauceLabs Proxy User"
system true
action :create
end
directory node['sauceproxy']['server']['install_dir'] do
owner node['sauceproxy']['server']['user']
mode 00755
action :create
end
# Can't assume we have unzip
package "unzip" do
action :install
end
execute "unzip-saucelabs-proxy" do
cwd node['sauceproxy']['server']['install_dir']
command "unzip -o #{Chef::Config[:file_cache_path]}/#{node['sauceproxy']['server']['zipfile']}"
action :nothing
notifies :restart, "service[sauceproxy]"
end
https://github.com/juliandunn/sauceproxy/blob/master/recipes/server.rb
Friday, June 14, 13

52. Example Recipe Part Two
remote_file "#{Chef::Config[:file_cache_path]}/#{node['sauceproxy']['server']['zipfile']}" do
source "#{node['sauceproxy']['server']['download_url']}/#{node['sauceproxy']['server']['zipfile']}"
action :create_if_missing
notifies :run, "execute[unzip-saucelabs-proxy]", :immediately
end
template "/etc/init.d/sauceproxy" do
source "sauceproxy.init.erb"
mode 00755
owner "root"
group "root"
action :create
end
template "/etc/sysconfig/sauceproxy" do
source "sauceproxy.sysconfig.erb"
mode 00644
owner "root"
group "root"
action :create
end
service "sauceproxy" do
supports :restart => true
action [:enable, :start]
end
Friday, June 14, 13

53. Example ChefSpec Test: Setup
describe 'sauceproxy::server' do
let (:chef_run) do
runner = ChefSpec::ChefRunner.new(
platform: 'centos',
version: '6.3'
)
runner.node.set['sauceproxy']['server']['user'] = 'fake'
runner.node.set['sauceproxy']['server']['install_dir'] =
'/tmp/fake'
runner.node.set['sauceproxy']['server']['version'] = '3.14159'
runner.converge('sauceproxy::server')
end
Friday, June 14, 13

54. Example ChefSpec Test: Expectations
it 'should create a sauceproxy directory with the right
ownership' do
expect(chef_run).to create_directory('/tmp/fake')
expect(chef_run.directory('/tmp/fake')).to
be_owned_by('fake')
end
it 'should start a service called sauceproxy' do
expect(chef_run).to start_service('sauceproxy')
expect(chef_run).to
set_service_to_start_on_boot('sauceproxy')
end
end
Friday, June 14, 13

55. ChefSpec In Action
Friday, June 14, 13

56. ChefSpec: Expectations
• Many expectations (assertions) possible
• https://github.com/acrmp/chefspec#making-
assertions
• Don’t (just) restate static resources
• Mock out data
• Static resource assertions: good for regressions
• Whoops, I deleted the service resource
Friday, June 14, 13

57. Acceptance Testing:
MiniTest
Friday, June 14, 13

58. Chef Client Run
Friday, June 14, 13

59. Chef Client Run
Friday, June 14, 13

60. Acceptance Tests as Report Handler
• Acceptance testing as a Chef Report Handler
• Many tools (serverspec, bats, minitest)
• I’ll demo Chef-MiniTest-Handler; most widely used
Friday, June 14, 13

61. MiniTest Handler Cookbook
• http://community.opscode.com/cookbooks/minitest-
handler
• Installs Minitest Gems
• Installs Chef Minitest Gems
• Installs the Chef-Minitest-Handler Notification
Handler for Chef-Client
• Places test files from cookbooks on the target node
as part of a Chef-client run
Friday, June 14, 13

62. MiniTest Files
• Each recipe gets an individual test file.
• recipes/server.rb
• files/default/test/server_test.rb
• Tests for each recipe are automatically loaded by the
handler
Friday, June 14, 13

63. Example MiniTest
require 'minitest/spec'
describe_recipe 'sauceproxy::server' do
it 'runs as a daemon' do
service('sauceproxy').must_be_running
end
end
http://tinyurl.com/minitest-examples
for many more example tests
Friday, June 14, 13

64. Make this go on a
with virtualization!
Friday, June 14, 13

65. Using Vagrant & Berkshelf to Iterate
• Vagrant is virtualization middleware
• Driven from Vagrantfile
• Defines VM images, customization parameters,
provisioners (Chef in our case)
• Berkshelf is a cookbook dependency manager
• Get dependent cookbooks from community API
• Feed them to Vagrant Chef provisioner
Friday, June 14, 13

66. Example Vagrantfile
Vagrant.configure("2") do |config|
config.vm.hostname = "sauceproxy-berkshelf"
config.vm.box = "opscode-centos-6.4"
config.vm.box_url = "https://opscode-vm.s3.amazonaws.com/vagrant/
opscode_centos-6.4_chef-11.4.4.box"
config.ssh.max_tries = 40
config.ssh.timeout = 120
config.vm.provision :chef_solo do |chef|
chef.json = {}
chef.run_list = [
"recipe[minitest-handler]",
"recipe[sauceproxy::server]"
]
end
end
Friday, June 14, 13

67. % vagrant up
Bringing Up Vagrant
borkbork:~/Dropbox/devel/github/juliandunn/sauceproxy (master)$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
[default] Importing base box 'centos-6.4'...
[default] Matching MAC address for NAT networking...
[default] Setting the name of the VM...
[default] Clearing any previously set forwarded ports...
[Berkshelf] Updating Vagrant's berkshelf: '/Users/juliandunn/.berkshelf/vagrant/
berkshelf-20130611-20810-rt7k01'
[Berkshelf] Using sauceproxy (0.1.8) at path: '/Users/juliandunn/Dropbox/devel/github/juliandunn/
sauceproxy'
[Berkshelf] Using minitest-handler (0.2.1)
[Berkshelf] Using java (1.11.4)
[Berkshelf] Using windows (1.8.10)
[Berkshelf] Using chef_handler (1.1.4)
[default] Creating shared folders metadata...
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Forwarding ports...
[default] -- 22 => 2222 (adapter 1)
[default] Booting VM...
[default] Waiting for VM to boot. This can take a few minutes.
Friday, June 14, 13

68. % vagrant up
Bringing Up Vagrant
borkbork:~/Dropbox/devel/github/juliandunn/sauceproxy (master)$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
[default] Importing base box 'centos-6.4'...
[default] Matching MAC address for NAT networking...
[default] Setting the name of the VM...
[default] Clearing any previously set forwarded ports...
[Berkshelf] Updating Vagrant's berkshelf: '/Users/juliandunn/.berkshelf/vagrant/
berkshelf-20130611-20810-rt7k01'
[Berkshelf] Using sauceproxy (0.1.8) at path: '/Users/juliandunn/Dropbox/devel/github/juliandunn/
sauceproxy'
[Berkshelf] Using minitest-handler (0.2.1)
[Berkshelf] Using java (1.11.4)
[Berkshelf] Using windows (1.8.10)
[Berkshelf] Using chef_handler (1.1.4)
[default] Creating shared folders metadata...
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Forwarding ports...
[default] -- 22 => 2222 (adapter 1)
[default] Booting VM...
[default] Waiting for VM to boot. This can take a few minutes.
Friday, June 14, 13

69. Vagrant Run Continued
[default] Waiting for VM to boot. This can take a few minutes.
[default] VM booted and ready for use!
GuestAdditions 4.2.12 running --- OK.
[default] Setting hostname...
[default] Configuring and enabling network interfaces...
[default] Mounting shared folders...
[default] -- /vagrant
[default] -- /tmp/vagrant-chef-1/chef-solo-1/cookbooks
[default] Running provisioner: chef_solo...
Generating chef JSON and uploading...
Running chef-solo...
[2013-06-12T02:52:26+00:00] INFO: *** Chef 11.4.4 ***
[2013-06-12T02:52:26+00:00] INFO: Setting the run_list to ["recipe[minitest-
handler]", "recipe[sauceproxy::server]"] from JSON
[2013-06-12T02:52:26+00:00] INFO: Run List is [recipe[minitest-handler],
recipe[sauceproxy::server]]
[2013-06-12T02:52:26+00:00] INFO: Run List expands to [minitest-handler,
sauceproxy::server]
[2013-06-12T02:52:26+00:00] INFO: Starting Chef Run for sauceproxy-berkshelf
Friday, June 14, 13

70. Vagrant Run Continued
[2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds
[2013-06-12T02:55:22+00:00] INFO: Running report handlers
Run options: -v --seed 31883
# Running tests:
recipe::java::openjdk#test_0001_installs the correct version of the jdk =
0.11 s = .
recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =
0.04 s = .
recipe::sauceproxy::server#test_0001_runs as a daemon =
0.09 s = .
Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.
3 tests, 3 assertions, 0 failures, 0 errors, 0 skips
[2013-06-12T02:55:22+00:00] INFO: Report handlers complete
Friday, June 14, 13

71. Vagrant Run Continued
[2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds
[2013-06-12T02:55:22+00:00] INFO: Running report handlers
Run options: -v --seed 31883
# Running tests:
recipe::java::openjdk#test_0001_installs the correct version of the jdk =
0.11 s = .
recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =
0.04 s = .
recipe::sauceproxy::server#test_0001_runs as a daemon =
0.09 s = .
Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.
3 tests, 3 assertions, 0 failures, 0 errors, 0 skips
[2013-06-12T02:55:22+00:00] INFO: Report handlers complete
Friday, June 14, 13

72. Minitests from Java cookbook ran too!
[2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds
[2013-06-12T02:55:22+00:00] INFO: Running report handlers
Run options: -v --seed 31883
# Running tests:
recipe::java::openjdk#test_0001_installs the correct version of the jdk =
0.11 s = .
recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =
0.04 s = .
recipe::sauceproxy::server#test_0001_runs as a daemon =
0.09 s = .
Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.
3 tests, 3 assertions, 0 failures, 0 errors, 0 skips
[2013-06-12T02:55:22+00:00] INFO: Report handlers complete
Friday, June 14, 13

73. Minitests from Java cookbook ran too!
[2013-06-12T02:55:22+00:00] INFO: Chef Run complete in 175.788656866 seconds
[2013-06-12T02:55:22+00:00] INFO: Running report handlers
Run options: -v --seed 31883
# Running tests:
recipe::java::openjdk#test_0001_installs the correct version of the jdk =
0.11 s = .
recipe::java::openjdk#test_0002_properly sets JAVA_HOME environment variable =
0.04 s = .
recipe::sauceproxy::server#test_0001_runs as a daemon =
0.09 s = .
Finished tests in 0.244690s, 12.2604 tests/s, 12.2604 assertions/s.
3 tests, 3 assertions, 0 failures, 0 errors, 0 skips
[2013-06-12T02:55:22+00:00] INFO: Report handlers complete
Friday, June 14, 13

74. New Test/Dev Cycle
Write Unit Tests
Worked?
Run ChefSpec
No
Write Recipe
Code
Write Acceptance
Tests
Yes
Worked?
Run Vagrant + Minitest Handler
No
vagrant destroy
Yes Commit/Tag Code,
etc.
Friday, June 14, 13

75. CI Pipeline Integration
Friday, June 14, 13

76. Continuous Integration Pipeline
• Run unit tests in your pipeline
• Run acceptance tests in your
pipeline
• Drive cookbook uploads as the
output!
Friday, June 14, 13

77. Travis-CI Example for SauceProxy
language: ruby
gemfile:
- gemfiles/travis.gemfile
rvm:
- "1.9.2"
- "1.9.3"
script: bundle exec rake test:syntax test:lint test:spec
notifications:
email:
- jdunn@opscode.com
Friday, June 14, 13

78. Rakefile Example
• Too long to post here
• Rake tasks for Foodcritic
and ChefSpec
• https://github.com/
juliandunn/sauceproxy/
blob/master/Rakefile
Friday, June 14, 13

79. Travis-CI Example Dashboard
Friday, June 14, 13

80. ^#$%(#$ I broke the build
Friday, June 14, 13

81. Test Kitchen (alpha)
• Run acceptance test suite on
multiple OSes
• Different fixtures for each, if desired
• Different drivers (vagrant, ec2, lxc,
etc.)
• Hook up to Jenkins or other CI
system if desired
Friday, June 14, 13

82. Test Kitchen Config File
---
driver_plugin: vagrant
platforms:
- name: centos-6.4
driver_config:
box: opscode-centos-6.4
box_url: https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-6.4_provisionerless.box
require_chef_omnibus: true
- name: centos-5.9
driver_config:
box: opscode-centos-5.9
box_url: https://opscode-vm.s3.amazonaws.com/vagrant/opscode_centos-5.9_provisionerless.box
require_chef_omnibus: true
suites:
- name: default
run_list: ["recipe[minitest-handler]", "recipe[sauceproxy::server]"]
attributes: {}
Friday, June 14, 13

83. Test Kitchen Demo
Friday, June 14, 13

84. Resources
Friday, June 14, 13

85. Resources
• ChefSpec: https://github.com/acrmp/chefspec
• Chef Minitest Handler: https://github.com/calavera/
minitest-chef-handler
• Berkshelf: http://berkshelf.com/
• Vagrant: http://vagrantup.com/
• Test Kitchen: https://github.com/opscode/test-kitchen
• Bento: https://github.com/opscode/bento
• SauceProxy Sample Cookbook: https://github.com/
juliandunn/sauceproxy
Friday, June 14, 13

86. More on Testing
• Chef-NYC: The Hows and Whys
of Cookbook Testing
• Seth Vargo (Author of ChefSpec,
Fauxhai, many others)
• http://www.meetup.com/Chef-
NYC/events/122219772/
• June 25th in Manhattan
Friday, June 14, 13

87. Chef Fundamentals Training
• Boston, July 11-12
(CompuWorks, 263
Summer St.)
• eventbrite.com/event/
6652057483
• Use code BOSTON-
MEETUP to save 25%!
Friday, June 14, 13

88. Questions and Prizes
Also, we’re hiring! Automate all the things for fun and profit!
http://www.opscode.com/blog/careers/
(and/or see me after)
Friday, June 14, 13