SlideShare une entreprise Scribd logo
1  sur  21
Télécharger pour lire hors ligne
Reducing Noisy Alerts from Nagios
                      Monitoring Casual Talk #1

                        Takumi SAKAMOTO
                             @takus




12年6月15日金曜日
Self Introduction

              •   Takumi SAKAMOTO
              •   @takus
              •   DeNA Co., Ltd. (2012.4 ~)
                  •   Operation Engineer ?




12年6月15日金曜日
Before Joining DeNA




12年6月15日金曜日
My365
              A Photo Sharing SNS



12年6月15日金曜日
12年6月15日金曜日
< 20 Servers




12年6月15日金曜日
After Joining DeNA




12年6月15日金曜日
Mobage




12年6月15日金曜日
> 1000 Servers




12年6月15日金曜日
What’s the Difference?




12年6月15日金曜日
Someone Said That ...

              •   What happens once in every million times
                  happens 3500 times per day




                                      http://blog.nomadscafe.jp/2011/05/post-12.html
12年6月15日金曜日
In the Context of Nagios ...




                                          (Photo : Postal Loathing by justin)
                               http://www.flickr.com/photos/justin/2412778/
12年6月15日金曜日
Too Many Alert Mails

              •   Problems
                  •   continue to call our mobile phone
                  •   sometimes hide more important alerts
                  •   burden mail systems




12年6月15日金曜日
How to Reduce
              Unnecessary Alerts



12年6月15日金曜日
#1: Defining Service Dependencies

              •   Approach
                  •   whatever you use for monitoring remote
                      host status, the status depend on what you
                      use for monitoring
                      •   e.g. SNMP, NRPE, SSH ...
                  •   define service dependencies between
                      parent service and child services



12年6月15日金曜日
Consider Simple Case 1

              •   Your nagios monitors remote hosts via SNMP
                  •   CPU, DISK, NTP, MEMORY
                  •   all services are OK

                                                       CPU

                                                       DISK
                      Nagios                SNMP
                                                       NTP

                                                      MEMORY

                                              Remote Host
12年6月15日金曜日
Consider Simple Case 2

              •   Nagios sometimes fails to check status by
                  SNMP because of high server load
              •   In this case, nagios evaluates all service status
                  are UNKNOWN and sends us 4 alert mails
                                                         CPU

                                             ???         DISK
                   Nagios                   SNMP
                                                         NTP

                                                        MEMORY

                                               Remote Host
12年6月15日金曜日
Consider Simple Case 3

              •   If many servers become over-loaded once,
                  nagios sends us a lot of noisy alert mails
                  •   because it is obvious that SNMP doesn’t
                      work well




12年6月15日金曜日
Defining SNMP Service Dependencies

              •    Nagios stop to send alert mail if SNMP
                   returns UNKNOWN
                  •   you will receive only a SNMP CRITICAL
                      alerts
              define servicedependency {
                dependent_host_name host1
                dependent_service_description CPU,DISK,MEMORY,NTP

                  host_name host1
                  service_description SNMP

                  notification_failure_criteria u
              }

12年6月15日金曜日
#2: Summarizing Similar Alerts


                                                               CPU

 Send Summary Alert
                      Summarizer
                                      テキスト
                                      テキスト
                                                               CPU




                                                               CPU




                            fluentdでNagiosアラートの集約

                            http://6pongi.wordpress.com/2012/06/08/fluentdnagios/
12年6月15日金曜日
What is your solution?




12年6月15日金曜日

Contenu connexe

En vedette

Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTExpeed Software
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsPixeldarts
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthThinkNow
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfmarketingartwork
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024Neil Kimberley
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)contently
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024Albert Qian
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsKurio // The Social Media Age(ncy)
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Search Engine Journal
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summarySpeakerHub
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next Tessa Mero
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentLily Ray
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best PracticesVit Horky
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project managementMindGenius
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...RachelPearson36
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Applitools
 

En vedette (20)

Everything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPTEverything You Need To Know About ChatGPT
Everything You Need To Know About ChatGPT
 
Product Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage EngineeringsProduct Design Trends in 2024 | Teenage Engineerings
Product Design Trends in 2024 | Teenage Engineerings
 
How Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental HealthHow Race, Age and Gender Shape Attitudes Towards Mental Health
How Race, Age and Gender Shape Attitudes Towards Mental Health
 
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdfAI Trends in Creative Operations 2024 by Artwork Flow.pdf
AI Trends in Creative Operations 2024 by Artwork Flow.pdf
 
Skeleton Culture Code
Skeleton Culture CodeSkeleton Culture Code
Skeleton Culture Code
 
PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024PEPSICO Presentation to CAGNY Conference Feb 2024
PEPSICO Presentation to CAGNY Conference Feb 2024
 
Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)Content Methodology: A Best Practices Report (Webinar)
Content Methodology: A Best Practices Report (Webinar)
 
How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 

Reducing Noisy Nagios Alerts

  • 1. Reducing Noisy Alerts from Nagios Monitoring Casual Talk #1 Takumi SAKAMOTO @takus 12年6月15日金曜日
  • 2. Self Introduction • Takumi SAKAMOTO • @takus • DeNA Co., Ltd. (2012.4 ~) • Operation Engineer ? 12年6月15日金曜日
  • 4. My365 A Photo Sharing SNS 12年6月15日金曜日
  • 11. Someone Said That ... • What happens once in every million times happens 3500 times per day http://blog.nomadscafe.jp/2011/05/post-12.html 12年6月15日金曜日
  • 12. In the Context of Nagios ... (Photo : Postal Loathing by justin) http://www.flickr.com/photos/justin/2412778/ 12年6月15日金曜日
  • 13. Too Many Alert Mails • Problems • continue to call our mobile phone • sometimes hide more important alerts • burden mail systems 12年6月15日金曜日
  • 14. How to Reduce Unnecessary Alerts 12年6月15日金曜日
  • 15. #1: Defining Service Dependencies • Approach • whatever you use for monitoring remote host status, the status depend on what you use for monitoring • e.g. SNMP, NRPE, SSH ... • define service dependencies between parent service and child services 12年6月15日金曜日
  • 16. Consider Simple Case 1 • Your nagios monitors remote hosts via SNMP • CPU, DISK, NTP, MEMORY • all services are OK CPU DISK Nagios SNMP NTP MEMORY Remote Host 12年6月15日金曜日
  • 17. Consider Simple Case 2 • Nagios sometimes fails to check status by SNMP because of high server load • In this case, nagios evaluates all service status are UNKNOWN and sends us 4 alert mails CPU ??? DISK Nagios SNMP NTP MEMORY Remote Host 12年6月15日金曜日
  • 18. Consider Simple Case 3 • If many servers become over-loaded once, nagios sends us a lot of noisy alert mails • because it is obvious that SNMP doesn’t work well 12年6月15日金曜日
  • 19. Defining SNMP Service Dependencies • Nagios stop to send alert mail if SNMP returns UNKNOWN • you will receive only a SNMP CRITICAL alerts define servicedependency { dependent_host_name host1 dependent_service_description CPU,DISK,MEMORY,NTP host_name host1 service_description SNMP notification_failure_criteria u } 12年6月15日金曜日
  • 20. #2: Summarizing Similar Alerts CPU Send Summary Alert Summarizer テキスト テキスト CPU CPU fluentdでNagiosアラートの集約 http://6pongi.wordpress.com/2012/06/08/fluentdnagios/ 12年6月15日金曜日
  • 21. What is your solution? 12年6月15日金曜日