SlideShare a Scribd company logo
1 of 41
Security Log Mining Beyond Log Analysis Anton Chuvakin, Ph.D., GCIA, GCIH, GCFA Security Log Mining   Last presented on March 9, 2007 IT Underground Prague, Czech Republic
Goals ,[object Object],[object Object],[object Object]
Outline: Log Mining (LM) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Definitions ,[object Object],[object Object],[object Object]
Log Analysis: What ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis: Why ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis: Why NOT  or Log Analysis Challenges ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: How ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: When ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
KDD and DM ,[object Object],[object Object],[object Object],[object Object]
Brief on Some DM Techniques ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
KDL and LM ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Preliminary Requirements ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Data from DM Perspective ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Data from DM Perspective ,[object Object],[object Object],[object Object],[object Object]
Example: Jumbled Mess of SAP Application Logs ,[object Object],[object Object],[object Object],[object Object],[object Object]
What Do We “Mine” for? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Simple Example ,[object Object],[object Object],[object Object],[object Object]
Deeper into interesting - I ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Example 1:  Can You Guess What Happened?! Destination Port 1D Baseline
Example 2:  Can You Guess What Happened?!
Example 3:  Can You Guess What Happened?!
Deeper into interesting - II ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Example 4:  Can You Guess What Happened?!
Example 5:  Can You Guess What Happened?!
Deeper into interesting - III ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Example 6: Can You Guess What Happened?!
More Examples ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Simple Example Revisited
Example 7: Can You Guess What Happened?!
Example 8: Fun Port Metrics
Example 9:  Can You Guess What Happened?!
Real-life Usage ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
How YOU can do it - I? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
How YOU can do it - II? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
How YOU can do it - III? ,[object Object],[object Object],[object Object],[object Object]
How YOU can do it - IV? ,[object Object],[object Object],[object Object],[object Object]
Conclusion ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Take These Home with You!! ,[object Object],[object Object],[object Object]
[object Object]
Feedback? Q&A? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

More Related Content

What's hot

Threat Hunting with Splunk
Threat Hunting with SplunkThreat Hunting with Splunk
Threat Hunting with Splunk
Splunk
 
Fantastic Red Team Attacks and How to Find Them
Fantastic Red Team Attacks and How to Find ThemFantastic Red Team Attacks and How to Find Them
Fantastic Red Team Attacks and How to Find Them
Ross Wolf
 

What's hot (20)

Bsides 2019 - Intelligent Threat Hunting
Bsides 2019 - Intelligent Threat HuntingBsides 2019 - Intelligent Threat Hunting
Bsides 2019 - Intelligent Threat Hunting
 
Threat Hunting with Splunk
Threat Hunting with SplunkThreat Hunting with Splunk
Threat Hunting with Splunk
 
Malware forensics
Malware forensicsMalware forensics
Malware forensics
 
When Security Tools Fail You
When Security Tools Fail YouWhen Security Tools Fail You
When Security Tools Fail You
 
Incident response methodology
Incident response methodologyIncident response methodology
Incident response methodology
 
OSINT- Leveraging data into intelligence
OSINT- Leveraging data into intelligenceOSINT- Leveraging data into intelligence
OSINT- Leveraging data into intelligence
 
EENA 2021: Keynote – Open-Source Intelligence (OSINT) for emergency services ...
EENA 2021: Keynote – Open-Source Intelligence (OSINT) for emergency services ...EENA 2021: Keynote – Open-Source Intelligence (OSINT) for emergency services ...
EENA 2021: Keynote – Open-Source Intelligence (OSINT) for emergency services ...
 
Open Source Intelligence (OSINT)
Open Source Intelligence (OSINT)Open Source Intelligence (OSINT)
Open Source Intelligence (OSINT)
 
Fantastic Red Team Attacks and How to Find Them
Fantastic Red Team Attacks and How to Find ThemFantastic Red Team Attacks and How to Find Them
Fantastic Red Team Attacks and How to Find Them
 
Sigma and YARA Rules
Sigma and YARA RulesSigma and YARA Rules
Sigma and YARA Rules
 
Threat hunting 101 by Sandeep Singh
Threat hunting 101 by Sandeep SinghThreat hunting 101 by Sandeep Singh
Threat hunting 101 by Sandeep Singh
 
The Four Types of Threat Detection and Use Cases in Industrial Security
The Four Types of Threat Detection and Use Cases in Industrial SecurityThe Four Types of Threat Detection and Use Cases in Industrial Security
The Four Types of Threat Detection and Use Cases in Industrial Security
 
Threat Hunting with Splunk Hands-on
Threat Hunting with Splunk Hands-onThreat Hunting with Splunk Hands-on
Threat Hunting with Splunk Hands-on
 
Unsupervised Anomaly Detection with Isolation Forest - Elena Sharova
Unsupervised Anomaly Detection with Isolation Forest - Elena SharovaUnsupervised Anomaly Detection with Isolation Forest - Elena Sharova
Unsupervised Anomaly Detection with Isolation Forest - Elena Sharova
 
Windows Threat Hunting
Windows Threat HuntingWindows Threat Hunting
Windows Threat Hunting
 
Network forensics and investigating logs
Network forensics and investigating logsNetwork forensics and investigating logs
Network forensics and investigating logs
 
Cyber Threat hunting workshop
Cyber Threat hunting workshopCyber Threat hunting workshop
Cyber Threat hunting workshop
 
PHDays 2018 Threat Hunting Hands-On Lab
PHDays 2018 Threat Hunting Hands-On LabPHDays 2018 Threat Hunting Hands-On Lab
PHDays 2018 Threat Hunting Hands-On Lab
 
Tools for Open Source Intelligence (OSINT)
Tools for Open Source Intelligence (OSINT)Tools for Open Source Intelligence (OSINT)
Tools for Open Source Intelligence (OSINT)
 
OSINT
OSINTOSINT
OSINT
 

Similar to Log Mining: Beyond Log Analysis

Similar to Log Mining: Beyond Log Analysis (20)

Making Logs Sexy Again: Can We Finally Lose The Regexes?
Making Logs Sexy Again: Can We Finally Lose The Regexes?Making Logs Sexy Again: Can We Finally Lose The Regexes?
Making Logs Sexy Again: Can We Finally Lose The Regexes?
 
Baselining Logs
Baselining LogsBaselining Logs
Baselining Logs
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)
 
Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008
 
Logs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMALogs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMA
 
Log Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinLog Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton Chuvakin
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
 
Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)Evolution of Monitoring and Prometheus (Dublin 2018)
Evolution of Monitoring and Prometheus (Dublin 2018)
 
Log Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual UsesLog Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual Uses
 
Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?
 
Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'
 
NIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldNIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real World
 
Big Data made easy in the era of the Cloud - Demi Ben-Ari
Big Data made easy in the era of the Cloud - Demi Ben-AriBig Data made easy in the era of the Cloud - Demi Ben-Ari
Big Data made easy in the era of the Cloud - Demi Ben-Ari
 
AI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are DangerousAI & ML in Cyber Security - Why Algorithms Are Dangerous
AI & ML in Cyber Security - Why Algorithms Are Dangerous
 
Implementing and Running SIEM: Approaches and Lessons
Implementing  and Running SIEM: Approaches and LessonsImplementing  and Running SIEM: Approaches and Lessons
Implementing and Running SIEM: Approaches and Lessons
 
IT Operation Analytic for security- MiSSconf(sp1)
IT Operation Analytic for security- MiSSconf(sp1)IT Operation Analytic for security- MiSSconf(sp1)
IT Operation Analytic for security- MiSSconf(sp1)
 
Audit logs for Security and Compliance
Audit logs for Security and ComplianceAudit logs for Security and Compliance
Audit logs for Security and Compliance
 
Chaos Engineering Without Observability ... Is Just Chaos
Chaos Engineering Without Observability ... Is Just ChaosChaos Engineering Without Observability ... Is Just Chaos
Chaos Engineering Without Observability ... Is Just Chaos
 
Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinEnterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
 

More from Anton Chuvakin

More from Anton Chuvakin (20)

Future of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsFuture of SOC: More Security, Less Operations
Future of SOC: More Security, Less Operations
 
SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?
 
Meet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton ChuvakinMeet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton Chuvakin
 
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
 
SOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton ChuvakinSOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton Chuvakin
 
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 BoothHey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
 
20 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 202220 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 2022
 
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
 
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020  Groovy SOC Tunes aka Modern SOC TrendsSOCstock 2020  Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
 
SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC
 
Modern SOC Trends 2020
Modern SOC Trends 2020Modern SOC Trends 2020
Modern SOC Trends 2020
 
Anton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in BriefAnton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in Brief
 
Generic siem how_2017
Generic siem how_2017Generic siem how_2017
Generic siem how_2017
 
Tips on SIEM Ops 2015
Tips on SIEM Ops 2015Tips on SIEM Ops 2015
Tips on SIEM Ops 2015
 
Five SIEM Futures (2012)
Five SIEM Futures (2012)Five SIEM Futures (2012)
Five SIEM Futures (2012)
 
RSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics PresentationRSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics Presentation
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
 
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton ChuvakinPractical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
 
SIEM Primer:
SIEM Primer:SIEM Primer:
SIEM Primer:
 

Recently uploaded

Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
UK Journal
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
UXDXConf
 

Recently uploaded (20)

Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara LaskowskaPowerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4j
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 WarsawGoogle I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
The Metaverse: Are We There Yet?
The  Metaverse:    Are   We  There  Yet?The  Metaverse:    Are   We  There  Yet?
The Metaverse: Are We There Yet?
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
WebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM PerformanceWebAssembly is Key to Better LLM Performance
WebAssembly is Key to Better LLM Performance
 
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdfBreaking Down the Flutterwave Scandal What You Need to Know.pdf
Breaking Down the Flutterwave Scandal What You Need to Know.pdf
 
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoftOauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
 
Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024Long journey of Ruby Standard library at RubyKaigi 2024
Long journey of Ruby Standard library at RubyKaigi 2024
 
Syngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdfSyngulon - Selection technology May 2024.pdf
Syngulon - Selection technology May 2024.pdf
 
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptxWSO2CONMay2024OpenSourceConferenceDebrief.pptx
WSO2CONMay2024OpenSourceConferenceDebrief.pptx
 
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for SuccessStructuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
 
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
Integrating Telephony Systems with Salesforce: Insights and Considerations, B...
 

Log Mining: Beyond Log Analysis