Presentation on how Puppet has been introduced in Seat Pagine Gialle to automate system administration tasks and easy the cooperation between Ops and Others.

1. Seat Internet Factory Tech Talk 5

3. Tempus fugit
Setup. Configure. Deploy. Troubleshoot. Fix.

4. Panta rei
Emails. Tickets. Calls. Meetings.

5. Mala
tempora
currunt
cat /etc/issue
Debian 4

6. Errare
humanum est...
rm /var/log/messages

7. perseverare
autem
diabolicum
chmod -R 777 ; su nobody ; rm -rf .

8. Alea iacta est
Hope driven release management.

9. Devs Versus Ops
D: Can you copy /ldap/dritchie/spes.war to prod? O: Open a Ticket.

10. O tempora
o mores!
Big efforts for small outputs

11. Spes ultima dea
Have you heard about this Puppet thing?

13. Repetitia iuvant
Machines automate better

14. Gratis
OpenSource from www.puppetlabs.com

15. Cui prodest?
Source: www.puppetlabs.com

16. Divide
et
impera
Many Puppet nodes One Puppet Master

17. package { 'openssh-server':
Status Quo }
ensure => present,
before => File['/etc/ssh/sshd_config'],
“A declarative language to describe file { '/etc/ssh/sshd_config':
system status” ensure => file,
mode => 600,
source => '/root/learning-manifests/sshd_config',
}
service { 'sshd':
ensure => running,
enable => true,
subscribe => File['/etc/ssh/sshd_config'],
}

18. Veni vidi vici
puppet apply

19. Deus
Ex
Machina
# This File is Managed by Puppet

20. Mater semper
certa est,
pater nunquam
O: Who changed that file?!
info: Filebucket[/var/lib/puppet/clientbucket]: Adding /etc/resolv.conf
(d7fbc1695489ce896d30b7b04d72887c)
info: //test/File[/etc/resolv.conf]: Filebucketed /etc/resolv.conf to main with sum
d7fbc1695489ce896d30b7b04d72887c
notice: //test/File[/etc/resolv.conf]/content: content changed '{md5}
d7fbc1695489ce896d30b7b04d72887c' to '{md5}
958836dd057fdbb33597d688cc6d28a2'

21. Ex novo
Provision from scratch.

22. Doctum
doces
Infrastructure as code

23. Scripta manent,
verba volant
svn ci -m “Ticket #777”

24. Factotum
P: I manage. Almost everything.

25. Ignorantia legis
non excusat
Once you Puppettize, you Puppettize

27. Sperimentazione iniziata nel 2010
Implementazione sistematica su nuovi server da Ottobre 2010
Conversione / Upgrade sistemi esistenti in corso.
Produzione a Rozzano - Sistemi gestiti: 167 (growing):
- Pagine Bianche
- Banners
- E-Commerce, Scioppy
- Tools PL, Iglu, Spysite, Routing, Fotocontest, Iglu ...
- Sistemi infrastrutturali (Dns, Syslog, Deploy, Ldap (soon) )
- Sistemi di monitoring (Nagios, Munin)
Disaster Recovery - Sistemi gestiti: 34:
- Pagine Gialle (Intera filiera)
- Sistemi infrastrutturali (Dns, Syslog, Deploy, Ldap)
- Sistemi di monitoring (Nagios, Munin)

28. # Base node
node basenode {
$dns_servers = ["192.168.39.42","192.168.39.43"]
$syslog_server = "syslog-1.pgol.com"
$type = "prod" # We assume that most of nodes are of prod(ution) type.
$users_auth = "ldap" # By default we want ldap auth
[...]
}
# Rozzano Production site
node rozzano inherits basenode {
$site = "rozzano"
$ntp_server = ["ntp1.pgol.com","ntp2.pgol.com"]
}
node rozzano-dmz inherits rozzano {
$zone = "dmz"
$users_auth = "local" # No ldap auth for users in DMZ servers
}
node rozzano-erog inherits rozzano {
$zone = "erog"
}
# Disaster recovery Site
node dr inherits basenode {
$site = "dr"
$dns_servers = ["192.168.50.10","192.168.50.11"]
[...]

29. # FRONTEND BIANCO # CLUSTER CASSANDRA DEV
node 'fep-pbit-1' inherits rozzano-erog { node 'dev-cassandra-1b' inherits rozzano-
include role_fep-pbit erog {
} ! $type = "dev"
! include role_cassandra
node 'fep-pbit-2' inherits rozzano-erog { }
include role_fep-pbit [...]
}
# CLUSTER CASSANDRA TEST
[...] node 'test-cassandra-1' inherits rozzano-
erog {
node 'test-pbit-1' inherits rozzano-erog { ! $type = "test"
$type = "test" ! include role_cassandra
include role_fep-pbit }
} [...]
[...] # CLUSTER CASSANDRA PROD
node 'cassandra-1' inherits rozzano-erog {
! include role_cassandra
}
[...]
node 'cassandra-5' inherits rozzano-erog {
! $has_batch = "true"
! include role_cassandra
}

30. class general {
include distro
include profile
include puppet
include motd
include users
include openssh
include hosts
include resolver
include monit
include sudo
include snmpd
include nrpe
include munin
include rsync
include basedirs
case $operatingsystem {
ubuntu,debian: {
# Moduli attualmente funzionanti solo su Ubuntu/Debian
include exim
include openntpd
include apt
include rsyslog
include unattended-upgrades
include hardware
}
centos,redhat: {
include yum
}
default: { }
}
}

31. class role_fep-pbit {
$role="fep-pbit"
include general
include nfs::client::fep-pbit
include apache
include apache::seat::fep-pbit
include php::pear
include php::dev
include php::oci8
include apache::spidertrap
apache::module { "rewrite": }
apache::module { "proxy": templatefile => "proxy.conf.erb" }
php::module { "gd": }
php::pear::module { "apc": }
php::pear::module { "XML_Serializer": use_package => "no" }
php::pecl::config { "http_proxy": value => "$proxy_server" }
[...]
# Monitoring
include monitor::seat::url_fep-pbit
# Deploy
puppi::project::files { "fep-pbit":
source => "http://deploy.${domain}/fep-pbit/deploylist.txt",
[...]
}
}

32. class apache::monitor {
# Port monitoring
monitor::port { "apache_${apache::params::protocol}_${apache::params::port}":
protocol => "${apache::params::protocol}",
port => "${apache::params::port}",
target => "${apache::params::monitor_target_real}",
enable => "${apache::params::monitor_port_enable}",
tool => "${monitor_tool}",
}
# Process monitoring
monitor::process { "apache_process":
process => "${apache::params::processname}",
service => "${apache::params::servicename}",
pidfile => "${apache::params::pidfile}",
enable => "${apache::params::monitor_process_enable}",
tool => "${monitor_tool}",
}
[...]
}

33. $monitor="yes"
$monitor_tool=["monit","nagios","puppi",”munin”]

34. class monitor::seat::url_fep-pbit {
# 1240
monitor::url { "Url-1240_PREFFISSI_INTERNAZIONALI":
url => "http://1240.paginebianche.it/execute.cgi?
ts=9&cb=18&btt=1&nz=CANADA&l=it",
port => '80',
target => "${fqdn}",
pattern => '01139',
enable => "true",
tool => "${monitor_tool}",
}
monitor::url { "Url-1240_RICERCA_LOCALITA_E_CAP_PER_PREFISSO":
url => "http://1240.paginebianche.it/execute.cgi?
ts=19&cb=18&btt=1&prefisso=0744&l=it",
port => '80',
target => "${fqdn}",
pattern => 'Terni',
enable => "true",
tool => "${monitor_tool}",
}
[...]
}

36. root@fep-pbit-1:~# puppi check
Host check: 50-Url-1240_PREFFISSI_INTERNAZIONALI [ OK ]
HTTP OK: HTTP/1.1 200 OK - 25285 bytes in 0.193 second response time |time=0.192856s;;;
0.000000 size=25285B;;;0
Host check: 50-Url-1240_RICERCA_LOCALITA_E_CAP_PER_PREFISSO[ OK ]
HTTP OK: HTTP/1.1 200 OK - 12417 bytes in 0.071 second response time |time=0.070648s;;;
0.000000 size=12417B;;;0
Host check: 50-Url-1240_RICERCA_LOCALITA_PER_CAP [ OK ]
HTTP OK: HTTP/1.1 200 OK - 12641 bytes in 0.048 second response time |time=0.048381s;;;
0.000000 size=12641B;;;0
[...]
Host check: 50-Mount__store_www.paginebianche.it_doc_root_g[ OK ]
/store/www.paginebianche.it/doc_root/gclight is mounted! Type is nfs
Host check: 50-apache_process [ OK ]
PROCS OK: 540 processes with command name 'apache2'
Host check: 50-apache_tcp_80 [ OK ]
TCP OK - 0.000 second response time on port 80|time=0.000250s;;;0.000000;10.000000
Host check: 50-openssh_process [ OK ]
PROCS OK: 3 processes with command name 'sshd'
Host check: 50-openssh_tcp_22 [ OK ]
TCP OK - 0.000 second response time on port 22|time=0.000151s;;;0.000000;10.000000

37. root@metaportali-mpc:~# puppi deploy configurator
Puppi setup: 00-configurator-RuntimeConfig-Initialization [ OK ]
Deploy: 10-configurator-Run_PRE-Checks [ OK ]
[...]
Deploy: 20-configurator-Retrieve_WAR [ OK ]
Deploy: 30-configurator-Backup_existing_WAR [ OK ]
Deploy: 36-configurator-Disable_extra_services [ OK ]
[...]
Deploy: 37-configurator-Check_undeploy [ OK ]
Deploy: 38-configurator-Service_stop [ OK ]
[...]
Deploy: 39-configurator-Run_Custom_PreDeploy_Script [ OK ]
Deploy: 40-configurator-Deploy_WAR [ OK ]
Deploy: 42-configurator-Service_start [ OK ]
[...]
Deploy: 43-configurator-Check_deploy [ OK ]
Deploy: 44-configurator-Enable_extra_services [ OK ]
[...]
Deploy: 80-configurator-Run_POST-Checks [ OK ]
[...]
Reporting: 20-configurator-Mail_Notification [ OK ]
REPORT FOR PUPPI - STATUS OK
Summary of operations is: /var/log/puppi/configurator/20110303-145104/summary
Details are in: /var/log/puppi/configurator/20110303-145104/
Temporary workdir has been: /tmp/puppi/configurator/ (Will be rewritten at the next puppi run)
Runtime config file is: /tmp/puppi/configurator/config
Files have been archived in: /var/lib/puppi/archive/configurator/20110303-145104

38. root@fep-pbit-1:~# puppi rollback fep-pbit
Puppi setup: 00-fep-pbit-RuntimeConfig-Initialization [ OK ]
Choose deploy to rollback:
total 24
drwxr-xr-x 2 root root 4096 2011-02-17 17:25 20110217-172418
drwxr-xr-x 2 root root 4096 2011-02-24 17:18 20110224-171656
drwxr-xr-x 2 root root 4096 2011-02-28 16:54 20110228-165323
drwxr-xr-x 2 root root 4096 2011-03-01 17:01 20110301-170009
drwxr-xr-x 2 root root 4096 2011-03-01 17:13 20110301-171339
drwxr-xr-x 2 root root 4096 2011-03-01 17:16 20110301-171607
lrwxrwxrwx 1 root root 47 2011-03-01 17:16 latest -> /var/lib/puppi/archive/fep-pbit/
20110301-171607
[...]

39. class role_mpc-metaportale {
$role="mpc-metaportale"
[...]
puppi::project::war { "configurator":
source => "http://deploy.pgol.com/mpc-metaportale/configurator.war",
user => "www-data",
init_script => "tomcat-mpc",
predeploy_customcommand => "rm -rf /store/tomcat/mpc/webapps/*",
predeploy_user => "root",
predeploy_priority => "39",
deploy_root => "/store/tomcat/mpc/webapps",
report_email => "release_engineering@seat.it,webdesign@paginegialle.it",
enable => "true",
disable_services => "monit puppet",
}
root@metaportali-mpc:~# puppi deploy configurator
} Puppi setup: 00-configurator-RuntimeConfig-Initialization [ OK ]
Deploy: 10-configurator-Run_PRE-Checks [ OK ]
[...]
Deploy: 20-configurator-Retrieve_WAR [ OK ]
Deploy: 30-configurator-Backup_existing_WAR [ OK ]
Deploy: 36-configurator-Disable_extra_services [ OK ]
[...]
Deploy: 37-configurator-Check_undeploy [ OK ]
Deploy: 38-configurator-Service_stop [ OK ]
[...]
Deploy: 39-configurator-Run_Custom_PreDeploy_Script [ OK ]
Deploy: 40-configurator-Deploy_WAR [ OK ]
Deploy: 42-configurator-Service_start [ OK ]
[...]
Deploy: 43-configurator-Check_deploy [ OK ]
Deploy: 44-configurator-Enable_extra_services [ OK ]
[...]
Deploy: 80-configurator-Run_POST-Checks [ OK ]
[...]
Reporting: 20-configurator-Mail_Notification [ OK ]
REPORT FOR PUPPI - STATUS OK
Summary of operations is: /var/log/puppi/configurator/20110303-145104/summary
[...]

41. Faster Setups
Do ut des Tested code
Quick Scalability Url based checks
Deployment Agility Site Aware configurations
Testing Environment Standardized deploy requests

42. In medio stat virtus
Share needs, constraints, knowledge and skills

43. Hic
manebimus
optime
Cooperation is fun

44. Full Infrastructure Automation
Unified Infrastructure Reporting Per aspera ad astra
Self Service Release Management
AutoTesting Release Workflow
Add here your wildest (IT) dream...

45. ad maiora
Graphics: www.tatlin.net