Splunk Webinar: Verwandeln Sie Daten in wertvolle Erkenntnisse - Machine Learning mit Splunk

1. Copyright © 2016 Splunk Inc.
Machine Learning
DACH Webinar, 26. August 2016
Philipp Drieger
Sales Engineer
SME Business Analytics | IoT | ML

2. Disclaimer
During the course of this presentation, we may make forward looking statements regarding future
events or the expected performance of the company. We caution you that such statements reflect our
current expectations and estimates based on factors currently known to us and that actual events or
results could differ materially. For important factors that may cause actual results to differ from those
contained in our forward-looking statements, please review our filings with the SEC. The forward-looking
statements made in the this presentation are being made as of the time and date of its live presentation.
If reviewed after its live presentation, this presentation may not contain current or accurate information.
We do not assume any obligation to update any forward looking statements we may make.
In addition, any information about our roadmap outlines our general product direction and is subject to
change at any time without notice. It is for informational purposes only and shall not, be incorporated
into any contract or other commitment. Splunk undertakes no obligation either to develop the features
or functionality described or to include any such feature or functionality in a future release.

3. Copyright © 2016 Splunk Inc.
Why do we need ML?

4. Copyright © 2016 Splunk Inc.
Historical Data Real-time Data Statistical Models
DB, HDFS, NoSQL, Splunk, etc Machine Learning
T – a few days T + a few days
Why is this so challenging using traditional methods?
• DATA IS STILL IN MOTION, still in a BUSINESS PROCESS.
• Enrich real-time MACHINE DATA with structured HISTORICAL DATA
• Make decisions IN REAL TIME using ALL THE DATA
Splunk
Security Operations Center
Network Operations Center
Business Operations Center

5. Copyright © 2016 Splunk Inc.
What is ML?

6. ML 101: What is it?
• Machine Learning (ML) is a process for generalizing from examples
– Examples = example or “training” data
– Generalizing = building “statistical models” to capture correlations
– Process = ML is never done, you must keep validating & refitting models
• Simple ML workflow:
– Explore data
– FIT models based on data
– APPLY models in production
– Keep validating models
Image source: http://phdp.github.io/posts/2013-07-05-dtl.html
“All models are wrong, but some are useful.”
- George Box

7. 3 Types of Machine Learning
1. Supervised Learning: generalizing from labeled data

8. 3 Types of Machine Learning
2. Unsupervised Learning: generalizing from unlabeled data
Clustering

9. 3 Types of Machine Learning
3. Reinforcement Learning: generalizing from rewards in time

10. Machine Learning Algorithms
Unsupervised Supervised
ContinuousCategorial

11. Machine Learning Algorithms
Unsupervised Supervised
ContinuousCategorial
Clustering:
• kmeans, cluster
• K-means
• DBSCAN
• Birch
• Spectral Clustering
Classification:
• Logistic Regression
• Support Vector Machine
• Naïve-Bayes (Gaussian, Bernoulli)
• RandomForestClassifier
• KNN, Trees
Regression:
• Linear Regression
• Polynomial Regression
• ElasticNet
• Ridge
• Lasso
• RandomForestRegr.
Dimensionality
reduction:
• PCA
• KernelPCA
Association Analysis:
• Apriori
• FP-Growth
Hidden Markov Model
predict
outliers
anomalies
anomalydetection
Vectorization:
• TFIDF
• Decision Trees
SPL command ML Toolkit App V1.3
Preprocessing:
• Std. Scaler
• Fieldselector

12. Copyright © 2016 Splunk Inc.
Operationalization
& Use Cases

13. Operationalize Machine Learning
• Reinforcement learning is key
• Reduce false positives:
– Recommend incidents to analyst
– Record their workflow
– Build models to learn
• Reward analysts for good analysis:
– Give bonuses for successful investigations
& detailed investigative reports
• Reward machines for good learning:
– Calibrate data, risk/rewards, analyst outcome

14. IT Ops: Predictive Maintenance
1. Get resource usage data (CPU, latency, outage reports)
2. Explore data, and fit predictive models on past / real-time data
3. Apply & validate models until predictions are accurate
4. Forecast resource saturation, demand & usage
5. Surface incidents to IT Ops, who INVESTIGATES & ACTS
Problem: Network outages and truck rolls cause big time & money expense
Solution: Build predictive model to forecast outage scenarios, act pre-emptively & learn
Operationalize

15. Security: Find Insider Threats
Problem: Security breaches cause big time & money expense
Solution: Build predictive model to forecast threat scenarios, act pre-emptively & learn
1. Get security data (data transfers, authentication, incidents)
2. Explore data, and fit predictive models on past / real-time data
3. Apply & validate models until predictions are accurate
4. Forecast abnormal behavior, risk scores & notable events
5. Surface incidents to Security Ops, who INVESTIGATES & ACTS
Operationalize

16. Business Analytics: Predict Customer Churn
Problem: Customer churn causes big time & money expense
Solution: Build predictive model to forecast possible churn, act pre-emptively & learn
1. Get customer data (set-top boxes, web logs, transaction history)
2. Explore data, and fit predictive models on past / real-time data
3. Apply & validate models until predictions are accurate
4. Forecast resource saturation, demand & usage
5. Surface incidents to Business Ops, who INVESTIGATES & ACTS
Operationalize

17. Summary: The ML Process
Problem: <Stuff in the world> causes big time & money expense
Solution: Build predictive model to forecast <possible incidents>, act pre-emptively & learn
1. Get all relevant data to problem
2. Explore data, and fit predictive models on past / real-time data
3. Apply & validate models until predictions are accurate
4. Forecast KPIs & metrics associated to use case
5. Surface incidents to X Ops, who INVESTIGATES & ACTS
Operationalize

18. Copyright © 2016 Splunk Inc.
Show me the ML!

19. ML Toolkit & Showcase App
• 20+ standard algorithms for model creation:
– Supervised: Logistic & Linear Regr., SVM, Random Forest
– Unsupervised: KMeans, DBSCAN, Spectral Clustering
• Leverages Python for Scientific Computing
Library and enables use of 300+ algorithms
• Modeling Assistants: Guide model building
and validation
• Showcases: Interactive examples for 20 IT,
security, business, IoT use cases
Extends Splunk platform functions and provides a guided modeling environment

20. Copyright © 2016 Splunk Inc.
ML Links and Resources

21. Links and Resources
• Download the Splunk App Machine Learning Toolkit and Showcase:
https://splunkbase.splunk.com/app/2890/
• Get started with the latest documentation:
http://docs.splunk.com/Documentation/MLApp/latest/User/About
• Want to dive deeper? Advanced Splunk education course “Splunk for Analytics and Data Science”:
https://www.splunk.com/view/SP-CAAAPCM
• Learn from your peers: 15+ machine learning related sessions at .conf
http://conf.splunk.com/sessions/2016-sessions.html
Lots of helpful resources at your finger tips:
Sept 26-29, 2016 | Walt Disney World Swan and Dolphin Resorts

22. Copyright © 2016 Splunk Inc.
Thanks – Q&A
philipp@splunk.com
Philipp Drieger
Sales Engineer
SME Business Analytics | IoT | ML