Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Performance Tuning Your Puppet Infrastructure - PuppetConf 2014

2 572 vues

Publié le

Performance Tuning Your Puppet Infrastructure - Nic Benders, New Relic

Publié dans : Technologie
  • Soyez le premier à commenter

Performance Tuning Your Puppet Infrastructure - PuppetConf 2014

  1. 1. Performance Tuning Your (Puppet) Infrastructure Nic Benders OWNING http://nicbenders.com/presentations/puppetconf-2014
  2. 2. I work at New Relic. ! This is a story about treating your infrastructure the same way you treat your applications. ! This is also a story about how we use New Relic, at New Relic. http://nicbenders.com/presentations/puppetconf-2014
  3. 3. Managing Our Applications • Source Control • Continuous Deployment • Performance Monitoring • Operational Analytics • Log Collection GitHub with Pull Requests Jenkins + Capistrano New Relic APM New Relic Insights Heka + ElasticSearch + Kibana
  4. 4. Managing Our Infrastructure • Source Control • Continuous Deployment • Performance Monitoring • Operational Analytics • Log Collection GitHub with Pull Requests Jenkins + Capistrano New Relic APM New Relic Insights Heka + ElasticSearch + Kibana
  5. 5. Source Control
  6. 6. Continuous Deployment
  7. 7. Performance Monitoring
  8. 8. DON’T PANIC • These example are for Puppet Enterprise 3.1, but this will work with Puppet Open Source, or with newer versions. You’ll just need to make a few changes here and there. • If you don’t have a New Relic account, you can sign-up for a free “Lite” account. • All of the examples are up in a more useful format at the URL below. (The URL will also be shown at the end of the presentation.) http://nicbenders.com/presentations/puppetconf-2014
  9. 9. Instrumenting the Puppet Master • Install newrelic_rpm using the Puppet’s Ruby interpreter • Modify the config.ru in the Puppet Master root dir • Create a newrelic.yml in the same directory # /var/opt/lib/pe-­‐puppetmaster/config.ru ! # Setup the New Relic Ruby Agent before anything else require 'rubygems' require 'newrelic_rpm' # END New Relic Ruby Agent setup # /var/opt/lib/pe-­‐puppetmaster/newrelic.yml ! common: &default_settings license_key: '000-­‐your-­‐license-­‐key-­‐here-­‐000' app_name: "Puppet Master" log_level: info log_file_path: '/var/log/pe-­‐puppet/' ssl: false capture_params: true transaction_tracer: enabled: true error_collector: enabled: true ignore_errors: "ActionController::RoutingError" production: <<: *default_settings monitor_mode: true
  10. 10. Getting good transaction names • Use set_transaction_name in a Rack Middleware to tell the Ruby Agent how we are going to name the “web transactions” for the Puppet Master • Use add_custom_parameters to pass the environment and other useful data along # # We don't want every transaction to just say "call", so define a # Transaction "namer" for the Ruby Agent based on the URL structure # class CoolNamer def initialize(app, opts = {}) @app = app end def call(env) req = Rack::Request.new(env) _, environment, kind_of_thing, resource = req.path.split('/',4) NewRelic::Agent.set_transaction_name(kind_of_thing, :category => :rack) NewRelic::Agent.add_custom_parameters(:puppet_env => environment) NewRelic::Agent.add_custom_parameters(:puppet_resource => resource) @app.call(env) end end use CoolNamer # END Transaction "namer"
  11. 11. Getting more detail • The Ruby Agent’s MethodTracer allows us to get details of what is going on inside a Transaction. • By hooking into Puppet’s built-in Profiler calls, we can quickly instrument all the most important things. • This example is for Puppet 3.3.1. In newer versions it is much easier. require "puppet/util/profiler" module Puppet::Util::Profiler include ::NewRelic::Agent::MethodTracer self.profile(message, &block) if /^Processed request def / === message # Top level trace, skip for now yield else metric_name = self.message_to_metric(message) trace_execution_scoped([metric_name]) do yield end end end def self.message_to_metric(message) message.chomp! message.gsub!(/^(Compiled catalog) for .*/, '1') message.gsub!(/^(Filtered result for catalog) .*/, '1') message.gsub!(/(d+) /, '') message.gsub!(/^(Evaluated resource) /, '1/') message.gsub!(/[.*]/,'') message.gsub!(/:s/,'/') message.gsub!(/^(Called) /, '1/') "Custom/Puppet/#{message}" end end
  12. 12. Instrumenting the Puppet DB • Download the Java Agent from your “Account Settings” page. • Create an /opt/puppet/newrelic dir and place the newrelic.jar and newrelic.yaml file into it. • Modify the newrelic.yaml • Modify JAVA_ARGS # /etc/syslog/pe-­‐puppetdb ! # Enable the New Relic Java Agent by adding this # to the bottom of the file JAVA_ARGS="-­‐javaagent:/opt/puppet/newrelic/newrelic.jar ${JAVA_ARGS}" # /opt/puppet/newrelic/newrelic.yml ! common: &default_settings license_key: '000-­‐your-­‐license-­‐key-­‐here-­‐000' ! app_name: 'Puppet DB' log_level: info log_file_path: /var/log/pe-­‐puppetdb/ ssl: false transaction_tracer: enabled: true transaction_threshold: apdex_f record_sql: obfuscated stack_trace_threshold: 0.2 explain_enabled: true explain_threshold: 0.2 browser_monitoring: auto_instrument: false production: <<: *default_settings
  13. 13. Operational Analytics
  14. 14. Log Collection
  15. 15. Puppet Agent already goes into Syslog • Agent syslogs have a ton of useful information. We just need to get it to a useful place. • Our application and syslog data is collected by a Heka process on each server. It then makes its way to ElasticSearch, where it can be queried by Kibana.
  16. 16. Send run reports to the same place • Add your log system as a Report processor! • We use https://github.com/ logstash/puppet-logstash-reporter • To get the data into Heka, we had to write some Lua, but it was pretty straightforward.
  17. 17. pe_puppet::master::config_hash: reports: 'http,puppetdb,logstash' ! logstash_reporter::logstash_host: 'our-­‐heka-­‐server' # Logstash-­‐style report processer through Heka class { 'logstash_reporter': logstash_port => '22253', logstash_host => hiera('logstash_reporter::logstash_host'), }
  18. 18. Managing Our Infrastructure • Source Control • Continuous Deployment • Performance Monitoring • Operational Analytics • Log Collection GitHub with Pull Requests Jenkins + Capistrano New Relic APM New Relic Insights Heka + ElasticSearch + Kibana
  19. 19. We’re Hiring Site Reliability Engineers MySQL Experts Engineering Managers Data Pipeline Engineers newrelic.com/about/careers
  20. 20. http://nicbenders.com/presentations/puppetconf-2014 nic@newrelic.com

×