Monitoring must evolve from a reactive to proactive approach based on unusual behavior. This requires a DevOps approach including systems thinking, amplifying feedback loops, and continuous learning. A central event handler should collect all events and alerts to enable dashboards, trend analysis, and complex event correlation to identify issues and their root causes across related systems. Predictive analytics on the historical monitoring data can produce models to send early warnings when patterns indicate future problems.
Breaking the Kubernetes Kill Chain: Host Path Mount
Life of an event - A never ending tool chain
1. Copyright
Life of an Event
A never ending tool chain
26-03-2015
Arnold van Wijnbergen
2. Copyright
2
Arnold van Wijnbergen
@BSMConsultant / Arnold.van.Wijnbergen@gmail.com
•15 years working experience
with automating IT
•DevOps Evangelist with
great passion about
Architecture and
Automation Tooling
•Full time Tooling Geek
Automation
Monitoring
Event management
Deployment
#IlovIT ☁
4. Copyright
4
Do you recognize this situation?
Organization is having monitoring but
always missing the correct data and
triggers.
Outage top 10 week 12
8. Copyright
8
Next statements will place the
context to DevOps
By applying the three Ways
/Underpinning Principles of DevOps
(Gene Kim)
• Systems Thinking
• Amplify Feedback Loops
• Culture of Continual
Experimentation And Learning
10. Copyright
10
Statement 2
DEV BUILDS BUSINESS VALUE
OPS MUST SUPPORT BUSINESS VALUE
Development started with Agile
Scrum approach.
DevOps will fill in these gaps for
Operations with use of Continuous
Monitoring to support the
Continuous Delivery pipeline
12. Copyright
12
BUT
Statement 4
DEV DOES TESTING
OPS DOES MONITORING
Align both disciplines
Close your feedback loop here
Reuse code for Quality
Assurance and Continuous
Improvement…
13. Copyright
13
BUT
Statement 5
All resources within your solution
are involved. Operations must be
able to pinpoint the outage,
understand and improve the whole
solution to increase resilience
Monitoring is NOT only about
infrastructure and solving the problem
14. Copyright
14
Now that we know the
DevOps part of the story
How should we fill in the
Monitoring part of the story ?
16. Copyright
16
Just like a starry sky all events (like stars) are inter-related
to each other like (the constellation)
And sometimes a star falls down, which breaks the chain
17. Copyright
17
Now that we talked you through Space
we will take the last steps how to
implement Continuous Monitoring
18. Copyright
18
Introduce a central event
handler which …
• Will become your big data
lake
• Will contain events and
alerts ( actionable events )
• Is used for dash boarding
• Trend analyses over time
• Application logging should
contain ARM points or
UUID for traceability
• Can escalate events to
alerts over time by
complex correlation.
• Should feed your IT chain
dashboards for deeper
chain analytics.
• At least 30 days of raw
operational data
First step – Centralize your event data
Data Lake
Infra
Monitoring
User
Experience
Monitoring
Application
Monitoring
19. Copyright
19
IT Value Chain
Business Application Business Application
Load balancer Load balancer
CMS
CMDB
Populate chain
&
Application dependencies
Employee is boarding a new customer
Confirmation
by post fails
SYMPTOM
SYMPTOM
SYMPTOM
Print queue
offline
CAUSE
Get to know your IT
Value Chains by …
• Use CMS tools for
discovering your IT
landscape with their
dependencies and to
keep configuration
management up-to-
date.
• Giving the DevOps
teams the possibility
to see the business
impact, track down
all specific events,
deepdive and identify
the cause event and
other symptom
events within one or
more related
business
applications.
Second step – Introduce IT Chain Monitoring
20. Copyright
20
Introduce a analytics system
to ….
• Analyze all your
monitoring historical data,
like events, metrics and
health states for the
machine learning
algorithms.
• These pre-processing will
find patterns to
automatically produce a
model for post processing
on production data
streams.
• After one of the patterns is
matched a prediction by
an early warning alert will
send out.
Third step – Put Analytics on your monitoring
data
Predictive Analytics Engine
EventsMetricsHistorical
Data sources
Prediction
Pre-processing Patterns
Post-processingModel
Produce model
Algorithms
Input
Output
21. Copyright
21
Fourth step – Review your feedback loop for your implemented
Continuous Monitoring toolchain
IT FactoryIT Delivery
Production
environment
SYSTEM Actor
End users
PEOPLE Actor
IT Management
Support desk
PEOPLE Actor
IT Management Tool chain
SYSTEM Actor
DevOps Engineers
PEOPLE Actor
Delivery Pipeline
22. Copyright
22
The Life of an Event
CONCLUSION
EVERY EVENT HAS A PURPOSE FOR
EXISTENCE AND DURING HIS
LIFETIME MULTIPLE INTER-
RELATIONS WILL EXIST. THIS
BEHAVIOUR WILL HELP US TO LEARN
AND DETECT PATTERNS
AT LAST BE PREDICTIVE FOR THE
FUTURE