Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Image Service Outage
postrotate  /bin/kill -HUP `cat /var/run/httpd.pid 2>/dev/null`  2> /dev/null || true
NEVER TEST INPRODUCTION!
It only takesone tinymistake
How Do You Enforce This?•   Documented standards and communicated best practices•   Robust testing workflow    •   Environm...
Testing Workflow
How We Use Environments•   Three environments: production, development, testing    •   Testing is unconstrained    •   Tes...
Working with Environments•    knife-flip by Etsy engineer Jon Cowie     (https://github.com/jonlives/knife-flip)    % knife ...
Keeping Environments in Sync•   knife-env-diff by Etsy engineer John Goulah    •   Get it at https://github.com/jgoulah/kn...
Introducing Knife Spork•   Knife plugin providing a testing/versioning workflow•   Authored by Jon Cowie•   Get it at https...
Spork Features•   Four stage process    •   Check: Look at versioning info for a cookbook    •   Bump: Automatically incre...
git:                              enabled: true                            irccat:                              enabled: t...
% knife spork check foodcriticChecking versions for cookbook foodcritic...Current local version: 0.0.4Remote versions (Max...
% knife spork bump foodcriticLoaded config file /home/pmcdonnell/git/chef-repo/config/spork-config.yml...Loaded config fil...
% knife spork upload foodcriticLoaded config file /home/pmcdonnell/git/chef-repo/config/spork-config.yml...Loaded config f...
% knife spork promote foodcritic --remotePulling latest changes from gitChecking that foodcritic version 0.0.5 exists on t...
WARNING: Youre about to promote changes to severalcookbooks:logrotate: = 0.1.24 changed to = 0.1.23foodcritic: = 0.0.4 cha...
Spork’s Logging Mechanisms •   Irccat: Logs to IRC channel (https://github.com/RJ/irccat)[11:35:33] <irccat> CHEF: pmcdonn...
Linting
Foodcritic•   A lint tool for Chef cookbooks written by Andrew Crump    (http://acrmp.github.com/foodcritic/)•   Comes wit...
Etsy’s Rules•       A work in progress, but newly open-sourced at        https://github.com/etsy/foodcritic-rules•       O...
Rule Resulting from Image Outage•   ETSY005 - Action :restart sent to a core service    •   Trippable services include htt...
Rule Resulting from Image Outage30 template "/etc/httpd/conf/httpd.conf" do31   source "httpd-conf.erb"32   owner "root"33...
Memcache Outage
02:27 < jallspaw> [Sat, 10 Jul 2010 01:45:01 +0000]INFO: Upgrading package[memcached] version from               1.4.2-1.f...
Don’t leave“known unknowns”lying in wait
Resulting Foodcritic Rule•   ETSY001 - Package or yum_package resource used with :upgrade action    •   Enforces always us...
Resulting Foodcritic Rule20 package "memcached" do21   action :upgrade22 end                      Changed to:20 package "m...
Reporting and Monitoring
Using Handlers •   Etsy’s handlers (https://github.com/etsy/chef-handlers)     •   Log failures to IRC[10:52:03] <irccat> ...
Graph with Graphite•   Metrics reporting made possible by knife-lastrun, authored by    John Goulah (https://github.com/jg...
% dsh -g all -c -M grep "Chef Run complete in" /var/log/chef/client.log | head -n 3 2>&1 | tee /tmp/tee && grepChef Run co...
Finding Run Time Outliers•   Knife doesn’t currently support Lucene’s NumericRangeQuery    •   Elapsed time is a floating p...
% knife search node elapsed:[200 TO 225] -alastrun.runtimes.elapsed4 items foundid:                         cent6-vmtempla...
% knife node lastrun sandboxmisc01.ny4.etsy.comStatus                    failedElapsed Time              211.78604Start Ti...
What Did Chef Just Do?•   chefrecentupdates by Etsy engineer Laurie Denness    (https://github.com/lozzd/ChefScripts)% che...
Preventative Measures
Knife Preflight•   By Jon Cowie (https://github.com/jonlives/knife-preflight)% knife preflight memcache::datacacheSearching ...
Continuous Chef•   Using Jenkins and base virtual machine images
“Out-of-Band” Management•   dsh (distributed shell) works even if Chef server is down    •   Etsy’s dsh groups are managed...
Configs Bundled with Packages•   Be careful with configs distributed with packages overwriting Chef    configs    •   They mu...
Jon will be at Velocity!•   Workshop: Michelin Starred Cooking with Chef    •   11:00am Monday, 06/25/2012    •   Topics  ...
We’re Hiring!        http://www.etsy.com/careers•   TONS of engineering positions open!•   Especially looking for a talent...
NEVER TEST INPRODUCTION!
NEVER TEST INPRODUCTION!
NEVER TEST INPRODUCTION!
NEVER TEST INPRODUCTION!
NEVER TEST INPRODUCTION!
NEVER TEST INPRODUCTION!
NEVER TEST INPRODUCTION!
Prochain SlideShare
Chargement dans…5
×

Publié le

NEVER TEST INPRODUCTION!

Publié dans : Technologie, Business

×