Contenu connexe Similaire à thwackCamp 2013: Cut the Alert Noise: Best Practices to Avoid Common Pitfalls and Optimize Managing IT Alers (20) thwackCamp 2013: Cut the Alert Noise: Best Practices to Avoid Common Pitfalls and Optimize Managing IT Alers1. Cut the Alert Noise: Best Practices to Avoid
Common Pitfalls and Optimize Managing IT Alerts
SolarWinds® thwackCamp 2013
© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
2. Agenda
» What is Alert Central™ and how does it work?
» Solving common IT alert problems with Alert Central
Automatic assignment and escalation
Reassignment, moving alerts between team members/groups quickly
Integrating on call scheduling with alert management
Filtering out alerts that don’t need escalation
Preventing alerts from duplicate systems from being escalated
© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
3. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
How Alert Central Works
» Deployed as Virtual Appliance
» Alerts from IT systems company-wide are
consolidated in Alert Central (collected via e-mail
or direct integration with SolarWinds Orion family
products)
» Alert Central handles escalation and on-call
scheduling (with configurable policies) to ensure
alerts go to the appropriate person
» That person can then acknowledge and clear the
alert, reassign, or escalate it (via email or in
console)
4. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Automatic Assignment & Escalation
» Problem: get alerts ONLY to the people who can handle them.
» Problem: if someone’s not available, alerts can get stuck until they are.
» Problem: if everyone’s notified, it takes time to identify whether they care.
» Solution:
Create groups for each logical staff group in Alert Central.
Create escalation policies within each group to follow the group’s notification preferences.
Assign alerts to each group based on information in the alert.
5. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Moving alerts between team members/groups quickly
» Problem: With a mass email system, it’s easy to lose track of who is responsible or
miss that it’s you that got “assigned” the alert.
» Problem: When alerts need to go to another group, it can be unclear how to do so.
» Problem: Working remotely or from home, having to log in to deal with alerts can be
time consuming.
» Solution:
Automatic assignment makes sure only one person has the ball.
Reassign alerts via email or the console
Reassign alerts to either a specific user or a group, which will use the group’s configured
escalation policy automatically.
Each user can configure notification policies in Alert Central for their notification preferences.
6. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Integrating on call scheduling with alert management
» Problem: Maintaining an external on call calendar (on the whiteboard, in a
spreadsheet, in a shared calendar) can be hard to keep accurate
» Problem: Without on call integrated to alerting, either a shared device has to be
used or a person has to be available 24/7 to find the on call person
» Solution:
Use on call calendars within Alert Central to support rotations, regular schedules
Integrate on call directly with escalation policies – with fallback options
7. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Filtering out alerts that don’t need escalation
» Problem: Some alerts don’t need immediate action or are handled by regular
helpdesk/IT staff, but come in through the same email source
» Problem: Realistically, systems sometimes generate noise or invalid alerts, and it’s
faster to tune them out until a fix is made
» Solution:
Identify criteria for noise alerts based on the email/alert details
In Alert Central’s source configuration, choose the “Trash this alert” option for that alert criteria
8. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Preventing alerts from duplicate systems from being escalated
» Problem: Implementing new systems while leaving old systems running generates
some duplicate alerts
» Problem: Overlapping monitoring systems can generate similar alerts, but both
copies don’t need to be delivered
» Problem: In any case, it’s not possible to tune out an entire source, filtering needs to
be more sensitive
» Solution:
Identify data in the duplicate alert that doesn’t need to be delivered
Use “Trash this alert” to indicate the duplicate alerts should be skipped
As more cases come up, continue evolving the policy easily
9. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Resources
» Alert Central Getting Started Videos
Configuring Escalation Policies & On Call:
http://www.youtube.com/watch?feature=player_embedded&v=8h7nUEXJ3ws
Configuring Orion Alert Sources:
http://www.youtube.com/watch?feature=player_embedded&v=BJLK0IqDHII
Configuring Email Alert sources:
http://www.youtube.com/watch?feature=player_embedded&v=oMxN2oZZM4s
» Alert Central on thwack®
http://thwack.solarwinds.com/community/tools_tht/alert-central
» Alert Central Resource Library
Links to FAQs, downloads, and more: http://thwack.solarwinds.com/docs/DOC-170671
11. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Thank You!
The SOLARWINDS and SOLARWINDS & Design marks are the exclusive property of SolarWinds
Worldwide, LLC, are registered with the U.S. Patent and Trademark Office, and may be registered or
pending registration in other countries. All other SolarWinds trademarks, service marks, and logos
may be common law marks, registered or pending registration in the United States or in other
countries. All other trademarks mentioned herein are used for identification purposes only and
may be or are trademarks or registered trademarks of their respective companies.
Notes de l'éditeur Quick demo: group escalation policies; routing rules (which we’ll come back to later, so we don’t need to be too specific on). Quick demo: look at an alert in AC, show reassignment. Show user notification prefs. Quick demo: on call calendars, calendars in escalation policies Quick demo: alert source config with a trash option Quick demo: sourceconfig with more filtering rules