Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
ORCHESTRATED FUNCTIONAL 
TESTING WITH PUPPET-SPEC 
AND MSPECTATOR 
RAPHAËL PINSON
Who am I? 
Raphaël Pinson (@raphink) 
■ Infrastructure Developer & Trainer @ Camptocamp 
■ Augeas & Augeasproviders develo...
Monitoring vs. Functional Tests 
Complementary or redundant? 
www.camptocamp.com / 3/32
Conformity Tests 
■ Check if machines comply to standards 
■ Avoid permanent heavy monitoring checks 
■ Tests must be inte...
Treetester 
■ Back in 2008 
■ Written in Perl 
■ Orchestrate conformity tests on a 4k+ server fleet 
www.camptocamp.com / ...
Treetester: modules output 
■ For all hosts/modules 
■ Number of hosts filtered per module 
■ Modules dependency tree 
■ C...
Treetester: host output 
■ For each host 
■ Failed steps in the module tree 
■ Green: OK, Red: KO, Purple: Ignored 
www.ca...
Treetester architecture 
■ All data in a database (MySQL) 
■ Tests scripts output YAML 
■ Tests scripts can be local (host...
Treetester filters 
■ For each test/module 
■ Based on data in MySQL (joins and additional SQL conditions) 
■ Allows to li...
Treetester: future? 
■ Not open-sourced :'-( 
■ Too monolithic/not flexible enough 
■ Heavily linked to specific architect...
Adding specs to Puppet runs 
■ Testing the catalog before it gets applied 
■ Testing the node after the catalog is applied...
Rspec-puppet 
■ http://rspec-puppet.com 
■ Now the standard to unit test Puppet manifests 
■ Generates catalogs in clean e...
Puppet-spec 
■ Runs tests from within Puppet runs 
■ Test catalogs using rspec-puppet 
■ Test hosts using serverspec 
www....
Puppet-spec: Unit testing 
■ Catalog exposed by PuppetSpec::Catalog.instance.catalog 
■ Uses rspec-puppet matchers 
■ Asse...
Puppet-spec: Unit tests output 
# puppet agent -t 
info: Retrieving plugin 
err: Could not retrieve catalog from remote se...
Puppet-spec: Deploying unit tests 
■ On the master side: 
○ Tests are located in the spec/catalog/class directory of the 
...
Puppet-spec: Unit tests limits 
■ When to apply the tests (currently based on class names) 
■ Tests on master, or need to ...
Puppet-spec: Setting up Unit testing 
■ Tests achieved from catalog indirection terminii 
■ Plugins (terminii) deployed wi...
Serverspec 
■ http://serverspec.org 
■ Provides RSpec matchers for local functional tests (packages, 
users, services, por...
Serverspec backends 
Allows to use various means of launching tests: 
■ SSH (default) 
■ Exec 
■ Puppet (RAL, removed from...
Puppet-spec: Functional testing 
■ Uses serverspec/specinfra matchers 
■ Tests the machine state (not the catalog) 
requir...
Puppet-spec: Function tests output 
# puppet agent -t 
info: Retrieving plugin 
info: Caching catalog for foo.example.com ...
Puppet-spec: Deploying functional 
tests 
■ Tests are run after catalog application 
■ Tests can be distributed via plugin...
Puppet-spec: MCollective agent 
■ Communicates with distant nodes 
■ Sends action and values to specinfra check commands 
...
Mspectator 
■ https://github.com/raphink/mspectator 
■ RSpec matchers 
■ Calls MCollective to achieve tests 
■ Uses MColle...
Mspectator architecture 
■ Client runs RSpec 
■ RSpec calls MCollective 
■ MCollective calls distant spec agent 
■ spec ag...
Mspectator syntax 
Own matchers, mapping to specinfra backend methods: 
require 'mspectator' 
describe 'apache' do 
it { s...
Mspectator output 
$ rake spec SPEC=apache_spec.rb 
/home/rpinson/.rvm/rubies/ruby-1.8.7-p371/bin/ruby -S rspec apache_spe...
Mspectator demo 
www.camptocamp.com / 29/32
Contribute 
On GitHub: 
■ puppet-spec: https://github.com/raphink/puppet-spec 
■ mspectator: https://github.com/raphink/ms...
Thank you! 
■ raphael.pinson@camptocamp.com 
■ @raphink on Twitter/Github 
■ raphink on Freenode 
■ Slides: slideshare.net...
Orchestrated Functional Testing with Puppet-spec and Mspectator - PuppetConf 2014
Prochain SlideShare
Chargement dans…5
×

Orchestrated Functional Testing with Puppet-spec and Mspectator - PuppetConf 2014

2 520 vues

Publié le

Orchestrated Functional Testing with Puppet-spec and Mspectator - Raphaël Pinson, Camptocamp

Publié dans : Technologie
  • Soyez le premier à commenter

Orchestrated Functional Testing with Puppet-spec and Mspectator - PuppetConf 2014

  1. 1. ORCHESTRATED FUNCTIONAL TESTING WITH PUPPET-SPEC AND MSPECTATOR RAPHAËL PINSON
  2. 2. Who am I? Raphaël Pinson (@raphink) ■ Infrastructure Developer & Trainer @ Camptocamp ■ Augeas & Augeasproviders developer ■ Various contributions to Puppet & ecosystem www.camptocamp.com / 2/32
  3. 3. Monitoring vs. Functional Tests Complementary or redundant? www.camptocamp.com / 3/32
  4. 4. Conformity Tests ■ Check if machines comply to standards ■ Avoid permanent heavy monitoring checks ■ Tests must be inter-dependent ■ Focus on getting sysadmins to fix one thing at a time to converge toward standards www.camptocamp.com / 4/32
  5. 5. Treetester ■ Back in 2008 ■ Written in Perl ■ Orchestrate conformity tests on a 4k+ server fleet www.camptocamp.com / 5/32
  6. 6. Treetester: modules output ■ For all hosts/modules ■ Number of hosts filtered per module ■ Modules dependency tree ■ Colors by priority www.camptocamp.com / 6/32
  7. 7. Treetester: host output ■ For each host ■ Failed steps in the module tree ■ Green: OK, Red: KO, Purple: Ignored www.camptocamp.com / 7/32
  8. 8. Treetester architecture ■ All data in a database (MySQL) ■ Tests scripts output YAML ■ Tests scripts can be local (hosts as STDIN) or remote (ssh or http) ■ Tests are inter-dependent ■ Generate filtered data as a tree ■ Generate graphs (graphviz) www.camptocamp.com / 8/32
  9. 9. Treetester filters ■ For each test/module ■ Based on data in MySQL (joins and additional SQL conditions) ■ Allows to link tests to each other ■ Like multiple sieves www.camptocamp.com / 9/32
  10. 10. Treetester: future? ■ Not open-sourced :'-( ■ Too monolithic/not flexible enough ■ Heavily linked to specific architecture ■ Needed a rewrite www.camptocamp.com / 10/32
  11. 11. Adding specs to Puppet runs ■ Testing the catalog before it gets applied ■ Testing the node after the catalog is applied Enter the Puppet-spec module www.camptocamp.com / 11/32
  12. 12. Rspec-puppet ■ http://rspec-puppet.com ■ Now the standard to unit test Puppet manifests ■ Generates catalogs in clean environments ■ Asserts catalogs for resources/classes require 'spec_helper' describe 'logrotate::rule' do let(:title) { 'nginx' } it { should compile.with_all_deps } it { should contain_class('logrotate::setup') } end www.camptocamp.com / 12/32
  13. 13. Puppet-spec ■ Runs tests from within Puppet runs ■ Test catalogs using rspec-puppet ■ Test hosts using serverspec www.camptocamp.com / 13/32
  14. 14. Puppet-spec: Unit testing ■ Catalog exposed by PuppetSpec::Catalog.instance.catalog ■ Uses rspec-puppet matchers ■ Asserts real catalogs ■ Runs on the master or agent side (as catalog indirection terminii) describe 'puppet' do subject { PuppetSpec::Catalog.instance.catalog } it { should contain_package('puppet') } it { should contain_package('ppet') } it { should include_class('puppet') } it { should include_class('puppet::client::base') } end www.camptocamp.com / 14/32
  15. 15. Puppet-spec: Unit tests output # puppet agent -t info: Retrieving plugin err: Could not retrieve catalog from remote server: Unit tests failed: F.. Failures: 1) package Failure/Error: it { should contain_package('augeas') } expected that the catalogue would contain Package[augeas] # /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3 # /var/lib/puppet/lib/puppet/indirector/catalog/rest_spec.rb:31:in `find' Finished in 0.00092 seconds 3 examples, 1 failure Failed examples: rspec /var/lib/puppet/lib/spec/class/augeas/package_spec.rb:3 # package info: Not using expired catalog for foo.example.com from cache; expired at Tue Apr 02 17:40:21 +0200 2013 notice: Using cached catalog www.camptocamp.com / 15/32
  16. 16. Puppet-spec: Deploying unit tests ■ On the master side: ○ Tests are located in the spec/catalog/class directory of the environment ○ Only the directories named after classes declared in the catalog are tested ■ On the agent side: ○ Deploy tests using pluginsync ○ Tests are located in the lib/spec/catalog/class directory of each module ○ Only the directories named after classes declared in the catalog are tested www.camptocamp.com / 16/32
  17. 17. Puppet-spec: Unit tests limits ■ When to apply the tests (currently based on class names) ■ Tests on master, or need to deploy all tests with pluginsync ■ Redundant with existing unit tests, or additional security? www.camptocamp.com / 17/32
  18. 18. Puppet-spec: Setting up Unit testing ■ Tests achieved from catalog indirection terminii ■ Plugins (terminii) deployed with pluginsync ■ Setup done in routes.yaml: agent: catalog: # Either on the agent side terminus: rest_spec cache: yaml master: catalog: # Or on the master side terminus: compiler_spec www.camptocamp.com / 18/32
  19. 19. Serverspec ■ http://serverspec.org ■ Provides RSpec matchers for local functional tests (packages, users, services, ports, etc.) ■ Independant from configuration management tools require 'spec_helper' describe service('httpd') do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end describe file('/etc/httpd/conf/httpd.conf') do it { should be_file } its(:content) { should match /ServerName www.example.jp/ } end www.camptocamp.com / 19/32
  20. 20. Serverspec backends Allows to use various means of launching tests: ■ SSH (default) ■ Exec ■ Puppet (RAL, removed from core) $ serverspec-init Select OS type: 1) UN*X 2) Windows Select number: 1 Select a backend type: 1) SSH 2) Exec (local) Select number: 1 www.camptocamp.com / 20/32
  21. 21. Puppet-spec: Functional testing ■ Uses serverspec/specinfra matchers ■ Tests the machine state (not the catalog) require 'spec_helper' describe service('httpd') do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end describe file('/etc/httpd/conf/httpd.conf') do it { should be_file } its(:content) { should match /ServerName www.example.jp/ } end www.camptocamp.com / 21/32
  22. 22. Puppet-spec: Function tests output # puppet agent -t info: Retrieving plugin info: Caching catalog for foo.example.com info: Applying configuration version 'raphink/a2c8e0f [+]' ... Applying changes ... notice: Finished catalog run in 59.19 seconds err: Could not send report: Unit tests failed: FF Failures: 1) augeas Failure/Error: it { should be_installed } expected "augeas" to be installed # /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2 # /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save' 2) /usr/share/augeas/lenses/dist Failure/Error: it { should be_file } expected "/usr/share/augeas/lenses/dist" to be file # /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6 # /var/lib/puppet/lib/puppet/indirector/report/rest_spec.rb:45:in `save' Finished in 0.06033 seconds 2 examples, 2 failures Failed examples: rspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:2 # augeas rspec /var/lib/puppet/lib/spec/server/class/foo.example.com/package_spec.rb:6 # /usr/share/augeas/lenses/www.camptocamp.com / 22/32
  23. 23. Puppet-spec: Deploying functional tests ■ Tests are run after catalog application ■ Tests can be distributed via pluginsync (in the spec/server/class) directory of each module ■ Tests can be distributed with file Puppet resources, optionally using the spec::serverspec defined resource type www.camptocamp.com / 23/32
  24. 24. Puppet-spec: MCollective agent ■ Communicates with distant nodes ■ Sends action and values to specinfra check commands ■ Does not implement serverspec syntax ■ Returns true/false ■ Uses MCollective as transport (instead of SSH) Examples: $ mco rpc spec check action=user values=rpinson $ mco rpc spec check action=file values=/etc/passwd $ mco rpc spec check action=resolvable values=google.fr,A $ mco rpc spec check action=listening values=80 $ mco rpc spec check action=process values=mcollectived $ mco rpc spec check action=file_contain values=/etc/passwd,rpinson www.camptocamp.com / 24/32
  25. 25. Mspectator ■ https://github.com/raphink/mspectator ■ RSpec matchers ■ Calls MCollective to achieve tests ■ Uses MCollective spec agent (among others) www.camptocamp.com / 25/32
  26. 26. Mspectator architecture ■ Client runs RSpec ■ RSpec calls MCollective ■ MCollective calls distant spec agent ■ spec agent calls specinfra backend www.camptocamp.com / 26/32
  27. 27. Mspectator syntax Own matchers, mapping to specinfra backend methods: require 'mspectator' describe 'apache' do it { should find_nodes(100).or_less } # Counts discovered nodes it { should pass_puppet_spec } # Runs the `spec` agent it { should have_certificate.signed } # Uses the `puppetca` agent context 'when on Debian', :facts => { :operatingsystem => 'Debian' } do # Filter by facts it { should find_nodes(5).with_agent('spec') } it { should have_package('apache2.2-common') } it { should_not have_package('httpd') } it { should have_service('apache2').with( :ensure => 'running' ) } it { should have_file('/etc/apache2/apache2.conf') } it { should have_directory('/etc/apache2/conf.d') } it { should have_user('www-data') } end context 'when using SSL', :classes => ['apache::ssl'] do # Filter by classes it { should find_nodes(50).or_more } it { should have_package('ca-certificates') } end end www.camptocamp.com / 27/32
  28. 28. Mspectator output $ rake spec SPEC=apache_spec.rb /home/rpinson/.rvm/rubies/ruby-1.8.7-p371/bin/ruby -S rspec apache_spec.rb apache should find nodes 100 should pass puppet spec (FAILED - 1) should have certificate when on Debian should find nodes 5 (FAILED - 2) ... when using SSL should find nodes 50 (FAILED - 3) No request sent, we did not discover any nodes. should have package "ca-certificates" Failures: 1) apache Failure/Error: it { should pass_puppet_spec } expected that all hosts would pass tests, the following didn't: soekris01.wrk.cby.camptocamp.com: soekris02.wrk.cby.camptocamp.com: # ./apache_spec.rb:5 ... www.camptocamp.com / 28/32
  29. 29. Mspectator demo www.camptocamp.com / 29/32
  30. 30. Contribute On GitHub: ■ puppet-spec: https://github.com/raphink/puppet-spec ■ mspectator: https://github.com/raphink/mspectator www.camptocamp.com / 30/32
  31. 31. Thank you! ■ raphael.pinson@camptocamp.com ■ @raphink on Twitter/Github ■ raphink on Freenode ■ Slides: slideshare.net/raphink www.camptocamp.com / 31/32

×