As Infrastructure as Code gets more widely adopted and more heavily automated, the need for tests is on the rise. It has now become a common practice to ship Puppet modules with unit and functional tests. The Puppet-spec and Mspectator projects aim to validate Puppet catalogs and test your fleet on the fly, using the serverspec library.
4. Conformity Tests
■ Check if machines comply to standards
■ Avoid permanent heavy monitoring checks
■ Tests must be inter-dependent
■ Focus on getting sysadmins to fix one thing at a time to converge
toward standards
www.camptocamp.com / 4/32
5. Treetester
■ Back in 2008
■ Written in Perl
■ Orchestrate conformity tests on a 4k+ server fleet
www.camptocamp.com / 5/32
6. Treetester: modules output
■ For all hosts/modules
■ Number of hosts filtered per module
■ Modules dependency tree
■ Colors by priority
www.camptocamp.com / 6/32
7. Treetester: host output
■ For each host
■ Failed steps in the module tree
■ Green: OK, Red: KO, Purple: Ignored
www.camptocamp.com / 7/32
8. Treetester architecture
■ All data in a database (MySQL)
■ Tests scripts output YAML
■ Tests scripts can be local (hosts as STDIN) or remote (ssh or http)
■ Tests are inter-dependent
■ Generate filtered data as a tree
■ Generate graphs (graphviz)
www.camptocamp.com / 8/32
9. Treetester filters
■ For each test/module
■ Based on data in MySQL (joins and additional SQL conditions)
■ Allows to link tests to each other
■ Like multiple sieves
www.camptocamp.com / 9/32
10. Treetester: future?
■ Not open-sourced :'-(
■ Too monolithic/not flexible enough
■ Heavily linked to specific architecture
■ Needed a rewrite
www.camptocamp.com / 10/32
11. Adding specs to Puppet runs
■ Testing the catalog before it gets applied
■ Testing the node after the catalog is applied
Enter the Puppet-spec module
www.camptocamp.com / 11/32
12. Rspec-puppet
■ http://rspec-puppet.com
■ Now the standard to unit test Puppet manifests
■ Generates catalogs in clean environments
■ Asserts catalogs for resources/classes
require 'spec_helper'
describe 'logrotate::rule' do
let(:title) { 'nginx' }
it { should compile.with_all_deps }
it { should contain_class('logrotate::setup') }
end
www.camptocamp.com / 12/32
13. Puppet-spec
■ Runs tests from within Puppet runs
■ Test catalogs using rspec-puppet
■ Test hosts using serverspec
www.camptocamp.com / 13/32
14. Puppet-spec: Unit testing
■ Catalog exposed by PuppetSpec::Catalog.instance.catalog
■ Uses rspec-puppet matchers
■ Asserts real catalogs
■ Runs on the master or agent side (as catalog indirection terminii)
describe 'puppet' do
subject { PuppetSpec::Catalog.instance.catalog }
it { should contain_package('puppet') }
it { should contain_package('ppet') }
it { should include_class('puppet') }
it { should include_class('puppet::client::base') }
end
www.camptocamp.com / 14/32
15. Puppet-spec: Unit tests output
# puppet agent -t
info: Retrieving plugin
err: Could not retrieve catalog from remote server: Unit tests failed:
F..
Failures:
1) package
Failure/Error: it { should contain_package('augeas') }
expected that the catalogue would contain Package[augeas]
# /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3
# /var/lib/puppet/lib/puppet/indirector/catalog/rest_spec.rb:31:in `find'
Finished in 0.00092 seconds
3 examples, 1 failure
Failed examples:
rspec /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3 # package
info: Not using expired catalog for foo.example.com from cache; expired at Tue Apr 02 17:40:21 +0200 2013
notice: Using cached catalog
www.camptocamp.com / 15/32
16. Puppet-spec: Deploying unit tests
■ On the master side:
○ Tests are located in the spec/catalog/class directory of the
environment
○ Only the directories named after classes declared in the catalog
are tested
■ On the agent side:
○ Deploy tests using pluginsync
○ Tests are located in the lib/spec/catalog/class directory of each
module
○ Only the directories named after classes declared in the catalog
are tested
www.camptocamp.com / 16/32
17. Puppet-spec: Unit tests limits
■ When to apply the tests (currently based on class names)
■ Tests on master, or need to deploy all tests with pluginsync
■ Redundant with existing unit tests, or additional security?
www.camptocamp.com / 17/32
18. Puppet-spec: Setting up Unit testing
■ Tests achieved from catalog indirection terminii
■ Plugins (terminii) deployed with pluginsync
■ Setup done in routes.yaml:
agent:
catalog:
# Either on the agent side
terminus: rest_spec
cache: yaml
master:
catalog:
# Or on the master side
terminus: compiler_spec
www.camptocamp.com / 18/32
19. Serverspec
■ http://serverspec.org
■ Provides RSpec matchers for local functional tests (packages,
users, services, ports, etc.)
■ Independant from configuration management tools
require 'spec_helper'
describe service('httpd') do
it { should be_enabled }
it { should be_running }
end
describe port(80) do
it { should be_listening }
end
describe file('/etc/httpd/conf/httpd.conf') do
it { should be_file }
its(:content) { should match /ServerName www.example.jp/ }
end
www.camptocamp.com / 19/32
20. Serverspec backends
Allows to use various means of launching tests:
■ SSH (default)
■ Exec
■ Puppet (RAL, removed from core)
$ serverspec-init
Select OS type:
1) UN*X
2) Windows
Select number: 1
Select a backend type:
1) SSH
2) Exec (local)
Select number: 1
www.camptocamp.com / 20/32
21. Puppet-spec: Functional testing
■ Uses serverspec/specinfra matchers
■ Tests the machine state (not the catalog)
require 'spec_helper'
describe service('httpd') do
it { should be_enabled }
it { should be_running }
end
describe port(80) do
it { should be_listening }
end
describe file('/etc/httpd/conf/httpd.conf') do
it { should be_file }
its(:content) { should match /ServerName www.example.jp/ }
end
www.camptocamp.com / 21/32
22. Puppet-spec: Function tests output
# puppet agent -t
info: Retrieving plugin
info: Caching catalog for foo.example.com
info: Applying configuration version 'raphink/a2c8e0f [+]'
... Applying changes ...
notice: Finished catalog run in 59.19 seconds
err: Could not send report: Unit tests failed:
FF
Failures:
1) augeas
Failure/Error: it { should be_installed }
expected "augeas" to be installed
# /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2
# /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save'
2) /usr/share/augeas/lenses/dist
Failure/Error: it { should be_file }
expected "/usr/share/augeas/lenses/dist" to be file
# /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6
# /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save'
Finished in 0.06033 seconds
2 examples, 2 failures
Failed examples:
rspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2 # augeas
rspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6 # /usr/share/augeas/lenses/www.camptocamp.com / 22/32
23. Puppet-spec: Deploying functional
tests
■ Tests are run after catalog application
■ Tests can be distributed via pluginsync (in the spec/server/class)
directory of each module
■ Tests can be distributed with file Puppet resources, optionally
using the spec::serverspec defined resource type
www.camptocamp.com / 23/32
27. Mspectator syntax
Own matchers, mapping to specinfra backend methods:
require 'mspectator'
describe 'apache' do
it { should find_nodes(100).or_less } # Counts discovered nodes
it { should pass_puppet_spec } # Runs the `spec` agent
it { should have_certificate.signed } # Uses the `puppetca` agent
context 'when on Debian',
:facts => { :operatingsystem => 'Debian' } do # Filter by facts
it { should find_nodes(5).with_agent('spec') }
it { should have_package('apache2.2-common') }
it { should_not have_package('httpd') }
it { should have_service('apache2').with(
:ensure => 'running'
) }
it { should have_file('/etc/apache2/apache2.conf') }
it { should have_directory('/etc/apache2/conf.d') }
it { should have_user('www-data') }
end
context 'when using SSL', :classes => ['apache::ssl'] do # Filter by classes
it { should find_nodes(50).or_more }
it { should have_package('ca-certificates') }
end
end
www.camptocamp.com / 27/32
28. Mspectator output
$ rake spec SPEC=apache_spec.rb
/home/rpinson/.rvm/rubies/ruby-1.8.7-p371/bin/ruby -S rspec apache_spec.rb
apache
should find nodes 100
should pass puppet spec (FAILED - 1)
should have certificate
when on Debian
should find nodes 5 (FAILED - 2)
...
when using SSL
should find nodes 50 (FAILED - 3)
No request sent, we did not discover any nodes. should have package "ca-certificates"
Failures:
1) apache
Failure/Error: it { should pass_puppet_spec }
expected that all hosts would pass tests, the following didn't:
soekris01.wrk.cby.camptocamp.com:
soekris02.wrk.cby.camptocamp.com:
# ./apache_spec.rb:5
...
www.camptocamp.com / 28/32