The document discusses KPN's use of Beaker and VMpooler for multi-platform testing of Puppet modules. It outlines the requirements for an automated testing pipeline that runs syntax, unit, and acceptance tests on Linux and Windows platforms in parallel. The solution involved setting up a Jenkins master and build slaves along with VMpooler to provision VMs for testing. Beaker is used to run acceptance tests across multiple platforms. Details are provided on configuring the testing components and writing acceptance tests. Metrics on modules, test types, runs, and failed tests are presented.

1. Delivering Premium Quality Puppet Modules
René Last & Dimitri Tischenko
Puppetconf 2016
Using Beaker and VMpooler for Mul4-Pla7orm Tes4ng

2. Who are we?

3. Agenda
About KPN
Our problem
Our soluBon

4. About KPN

5. Our problem
KPN decided to manage 10K
servers for various clients
with Puppet…

6. Our problem
… most of them Windows.

7. Our problem – consequences (1)
Need to
support Windows

8. Our problem – consequences (2)
Need to guarantee
stable and predictable
infrastructure state

9. Our problem – consequences (3)
Need to guarantee
code quality

10. Our soluEon (1)
A dedicated team
to develop modules

11. Our soluEon (2)
An automated
tesBng factory

12. TesEng Factory Requirements
Automated pipeline configuraBon
Puppet 3.x tests
Puppet 4.x tests
Unit tests on Linux and Windows
Acceptance tests on all plaQorms (currently 8)
Run tests in parallel

13. Module CI/CD Pipeline Overview
syntax
check
Auto
puppet-
lint
rspec
test
acceptance
test
Release
to TEST
Auto Auto Manual
Release
to PROD
Manual
Change in
git
Periodic
trigger
NoEfy by mail & Hipchat
Fail Fail Fail Fail

14. Puppet CI Components
Jenkins Master
Jenkins
Linux Slave
Jenkins
Windows Slave
vmpooler
Github Enterprise
vcenter
beaker
Create VMs
Linux pipeline
Windows pipeline
MulE-PlaXorm
Acceptance Tests
Syntax, lint & rspec
Syntax, lint & rspec
Systems Under Test

16. Jenkins master and build slaves
rtyler/jenkins puppet module

17. Ruby
maestrodev/rvm
PIK or uru != rvm L

18. Jenkins jobs
Jenkins job builder
stankevich/python

19. PuppeEzing Jenkins Jobs
puppet Yaml file
Jenkins job
builder
xml Jenkins job

20. How to add a module
1. kpn_jenkins::pipeline::module { 'kpn-puppet-forge/puppet-kpn-hosts':
2. ensure => present,
3. test_platform => ['linux', 'windows'],
4. pe3_syntax => false,
5. pe4_syntax => true,
6. pe4_unit => true,
7. pe3_unit => false,
8. acceptance_test => true,
9. acceptance_platform => [ 'centos-7-x64-pe4',
10. 'windows-2008r2-x64',
11. 'windows-2012r2-x64',
12. 'windows-2008r2-x64-pe4',
13. 'windows-2012r2-x64-pe4',
14. 'windows-2016-x64-pe4',
15. ],
16. hipchat_notification => true,
17. hipchat_room => 'Module team',
18. email => ‘puppetteam@kpn.com',
19. }

21. How to add a module

22. Jenkins Master
Jenkins
Linux Slave
Jenkins
Windows Slave
vmpooler
Github Enterprise
vcenter
Create VMs
Windows pipeline
MulE-PlaXorm
Acceptance Tests
Syntax, lint & rspec
Syntax, lint & rspec
Systems Under Test
Beaker
beaker
Linux pipeline

24. Beaker
A test harness focused on acceptance tesBng
using (virtual) machines

25. Beaker
Runs tests on “real” servers

26. Beaker
Supports vm providers like Vagrant, vSphere,
docker, AWS or vmpooler

27. Beaker InstallaEon
# gem install beaker beaker-rspec beaker-puppet_install_helper

28. Beaker ConfiguraEon
spec/acceptance/nodesets/*.yml
1. HOSTS:
2. centos-7-x64-pe4:
3. roles:
4. - agent
5. platform: el-7-x86_64
6. hypervisor: vmpooler
7. template: centos-7-x64
8. CONFIG:
9. masterless: true
10. pe_ver: '2016.2.0'
11. pe_promoted_builds_url: 'http://repo.kpn.com/pe'
12. pooling_api: 'http://vmpooler.kpn.com:4567'
13. ssh:
14. auth_methods:
15. - password
16. user: root
17. password: XXXXXXXX

29. Prepare a System For TesEng
spec/spec_helper_acceptance.rb
1. require 'beaker-rspec/spec_helper'
2. require 'beaker-rspec/helpers/serverspec'
3. require 'beaker/puppet_install_helper'
4.
5. # Install puppet agent on a node
6. run_puppet_install_helper
7.
8. # Copy our module dependencies to the SUT
9. system('git clone git@github.kpn.com/stdlib spec/fixtures/modules/stdlib')
10.
11. copy_module_to(hosts, :source => 'spec/fixtures/modules/stdlib',
12. :module => 'stdlib')
13.
14. # Or if you have internet access
15. install_puppet_module_via_pmt_on(host, :module_name => 'stdlib')

30. Example Test
spec/acceptance/*_spec.rb
1. require 'spec_helper_acceptance'
2.
3. describe 'kpn-centerity_agent module' do
4. context 'install centerity_agent software' do
5. pp = <<-EOS
6. class { 'centerity_agent': }
7. EOS
8. it '1st apply should run without errors' do
9. apply_manifest(pp, :catch_failures => true)
10. end
11. it '2nd apply should run without changes' do
12. apply_manifest(pp, :catch_changes => true)
13. end
14. end
15. end

31. TesEng for Installed Packages
1. context 'Test installed software' do
2. case fact('osfamily')
3. when /RedHat/
4. describe package('sshd') do
5. it { should be_installed }
6. end
7. when /windows/
8. describe command('C:ProgramsChocolateybinclist -l') do
9. its(:stdout) { should match /sshd/ }
10. end
11. end
12. end
13. end

32. Run, beaker, Run!
# export BEAKER_setfile=~/nodesets/${platform}.yml
# rake beaker

33. Rules for WriEng Acceptance Tests (1)
Only test your puppet code,
not the supplier’s socware

34. Rules for WriEng Acceptance Tests (2)
A beaker test should be self-contained

35. Rules for WriEng Acceptance Tests (3)
If you can rspec it, don’t beaker it

36. Improving Beaker DocumentaEon
heps://github.com/RARYates/beaker

37. Jenkins Master
Jenkins
Linux Slave
Jenkins
Windows Slave
Github Enterprise
vcenter
Create VMs
Windows pipeline
MulE-PlaXorm
Acceptance Tests
Syntax, lint & rspec
Syntax, lint & rspec
Systems Under Test
beaker
vmpooler
vmpooler
Linux pipeline

38. vmpooler
Vmpooler pools VMs
Provides configurable pools of instantly-available
(running) virtual machines

39. vmpooler
Needs its own DHCP zone
and properly configured DDNS

40. vmpooler
System running beaker must use vmpooler’s
nameserver

41. vmpooler
vmpooler has a strange logo

42. vmpooler installaEon
include ‘vmpooler’

43. vmpooler
❝
Template set-up is lec as an exercise to the reader. Somehow, either via PXE, embedded
bootstrap scripts, or some other method -- clones of VM templates need to be able to set
their hostname, register themselves in your DNS, and be resolvable by the vmpooler
applicaBon acer compleBng the clone task and booBng up.
❞

44. vmpooler host naming puzzle
vm name != host name
need to set host name automaBcally to vm name
don’t want to give each vm vsphere credenBals
so - how to do it?

45. Just 5 lines of code (2 months later)
# Annotate with creation time, origin template, etc.
# Add extraconfig options that can be queried by vmtools
configSpec = RbVmomi::VIM.VirtualMachineConfigSpec(
annotation: JSON.pretty_generate(
name: vm['hostname'],
created_by: $config[:vsphere]['username'],
base_template: vm['template'],
creation_timestamp: Time.now.utc
),
extraConfig: [
{ key: 'guestinfo.hostname',
value: vm['hostname']
}
])

46. Seang the hostname
Linux:
# hostname=$(vmtoolsd --cmd "info-get guestinfo.hostname")
Windows:
> $hostname = cmd.exe /c 'C:Progra~1VMwarevmtoolsd.exe --cmd
"info-get guestinfo.hostname"'

47. vmpooler lessons learned
• Can use more logging (PRs forthcoming)
• Pool name needs to equal template name
• Docs refer to vm templates, need to be vms
• Will not install if ESX host is not “green”
• Can use beeer excepBon handling

48. Conclusions
Automated pipeline configuraBon
Puppet 3.x tests
Puppet 4.x tests
Unit tests on Linux and Windows
Acceptance tests on all plaQorms (currently 8)
Run tests in parallel
P
P
P
P
P
P

49. What did we learn?
+ Regression test
+ MulB plaQorm
+ Catch errors early

50. What did we learn?

51. Future Improvements
Speed of tests

53. Future Improvements
Stability of vmpooler
Windows support

55. And now … for some numbers

56. Modules per OperaEng System
Windows
32
Linux
43
Linux and
Windows
16

57. Test Types
102 94
79
0
20
40
60
80
100
120
Syntax Rspec (unit) Acceptance
# modules

58. Test Runs
Flow
8%
Init
14%
Syntax
21%
Unit
18%
Acceptance
39%

59. Top 10 Modules with Failed Tests

60. Vmpooler: Average Time to Ready

61. Q&A

62. Windows Modules
• kpn-ad_ds
• kpn-dotnet
• kpn-rdp
• kpn-kms
• kpn-chocolatey_client
• kpn-chocolatey_server
• kpn-local_security_policy