SlideShare a Scribd company logo

Making Logs Sexy Again: Can We Finally Lose The Regexes

•
•
Anton Chuvakin
Anton Chuvakin

Making Logs Sexy Again: Can We Finally Lose The Regexes?, presented at DeepSec 2008 in Vienna, Austria

Technology
1 of 41
Making Logs Sexy Again: Can We Finally Lose The Regexes? Dr. Anton Chuvakin
Agenda ,[object Object],[object Object],[object Object],[object Object],[object Object],C O N F I D E N T I A L
Who is Anton? ,[object Object],[object Object]
First: WTH is Logs? ,[object Object],C O M P A N Y C O N F I D E N T I A L II. A huge messy pile of undocumented, unstructured or poorly structured “ stuff ” that might or might not mean anything – at all?
WTF?!! ,[object Object],[object Object],[object Object],[object Object]
Second: What Do We Do With Logs? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
[object Object],C O M P A N Y C O N F I D E N T I A L
Log Analysis Basics: Summary ,[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Manual ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Filtering ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Summary ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Search ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Correlation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Log Management “Grand Challenges?” ,[object Object],[object Object],[object Object],[object Object]
GC1 – Log Chaos ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Example Log Chaos - Login? <122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff:192.168.138.35 port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Failure Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0     Logon account :  POWERUSER    <57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-warning-00515: Admin User netscreen has logged on via Telnet from 10.14.98.55:39073 (2002-12-17 15:50:53)
GC2 – Secure and Reliable Log Collection ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GC3 - Log Parsing and Regexs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GC4 – Automated Meaning Extraction ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GC5 – “Fuzzy” Search ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
[object Object],C O M P A N Y C O N F I D E N T I A L
Handling The Challenges ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Log Mining ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Preliminary DATA Requirements ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What Do We “Mine” for? ,[object Object],[object Object],[object Object],[object Object]
Simple Example ,[object Object],[object Object],[object Object],[object Object]
Deeper into interesting - I ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Can You Guess What Happened?!
Deeper into interesting - II ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Where Is The DATA? ,[object Object],[object Object],[object Object],[object Object]
“ Search+” ,[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Text Mining Logs? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Keywords to Meaning? ,[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Log Visualization ,[object Object],[object Object],[object Object],[object Object]
Log Context Enrichment? C O M P A N Y C O N F I D E N T I A L
Finally, Can We Make Better Logs? CEE = Syntax + Vocabulary + Transport + Log Recommendations ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Conclusions and CALL TO ACTION! ,[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Thank You! ,[object Object],[object Object],[object Object],[object Object],[object Object]
Backup and Reference Slides C O M P A N Y C O N F I D E N T I A L
“ Top 11 Reasons People Hate Logs” ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Recommended

Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Anton Chuvakin
 
Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'Anton Chuvakin
 
Log Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisLog Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisAnton Chuvakin
 
Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?Anton Chuvakin
 
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...Anton Chuvakin
 
Log Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual UsesLog Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual UsesAnton Chuvakin
 
Choosing Your Log Management Approach: Buy, Build or Outsource
Choosing Your Log Management Approach: Buy, Build or OutsourceChoosing Your Log Management Approach: Buy, Build or Outsource
Choosing Your Log Management Approach: Buy, Build or OutsourceAnton Chuvakin
 
"Grand Challenges" of Log Management
"Grand Challenges" of Log Management"Grand Challenges" of Log Management
"Grand Challenges" of Log ManagementAnton Chuvakin
 

Recommended

Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Anton Chuvakin
 
Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'Anton Chuvakin
 
Log Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisLog Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisAnton Chuvakin
 
Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?Anton Chuvakin
 
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...Anton Chuvakin
 
Log Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual UsesLog Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual UsesAnton Chuvakin
 
Choosing Your Log Management Approach: Buy, Build or Outsource
Choosing Your Log Management Approach: Buy, Build or OutsourceChoosing Your Log Management Approach: Buy, Build or Outsource
Choosing Your Log Management Approach: Buy, Build or OutsourceAnton Chuvakin
 
"Grand Challenges" of Log Management
"Grand Challenges" of Log Management"Grand Challenges" of Log Management
"Grand Challenges" of Log ManagementAnton Chuvakin
 
Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinEnterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinAnton Chuvakin
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008guestc0c304
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Anton Chuvakin
 
Baselining Logs
Baselining LogsBaselining Logs
Baselining LogsAnton Chuvakin
 
Logs = Accountability
Logs = AccountabilityLogs = Accountability
Logs = AccountabilityAnton Chuvakin
 
Pivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsPivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsKiyoto Tamura
 
Power of logs: practices for network security
Power of logs: practices for network securityPower of logs: practices for network security
Power of logs: practices for network securityInformation Technology Society Nepal
 
Log Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinLog Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinAnton Chuvakin
 
Logs vs Insiders
Logs vs InsidersLogs vs Insiders
Logs vs InsidersAnton Chuvakin
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)Brian Brazil
 
Python for Data Logistics
Python for Data LogisticsPython for Data Logistics
Python for Data LogisticsKen Farmer
 
LogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationLogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationAnton Chuvakin
 
CSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinCSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinAnton Chuvakin
 
Logs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMALogs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMAAnton Chuvakin
 
NIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldNIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldAnton Chuvakin
 
Functional and non functional application logging
Functional and non functional application loggingFunctional and non functional application logging
Functional and non functional application loggingSander De Vos
 
Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Andrew McNicol
 
What is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupWhat is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupMaarten Balliauw
 
The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!Michele Leroux Bustamante
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Demi Ben-Ari
 
Future of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsFuture of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsAnton Chuvakin
 
SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?Anton Chuvakin
 

More Related Content

Similar to Making Logs Sexy Again: Can We Finally Lose The Regexes

Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinEnterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinAnton Chuvakin
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008guestc0c304
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Anton Chuvakin
 
Pivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsPivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsKiyoto Tamura
 
Log Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinLog Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinAnton Chuvakin
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)Brian Brazil
 
Python for Data Logistics
Python for Data LogisticsPython for Data Logistics
Python for Data LogisticsKen Farmer
 
LogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationLogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationAnton Chuvakin
 
CSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinCSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinAnton Chuvakin
 
Logs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMALogs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMAAnton Chuvakin
 
NIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldNIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldAnton Chuvakin
 
Functional and non functional application logging
Functional and non functional application loggingFunctional and non functional application logging
Functional and non functional application loggingSander De Vos
 
Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Andrew McNicol
 
What is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupWhat is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupMaarten Balliauw
 
The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!Michele Leroux Bustamante
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Demi Ben-Ari
 

Similar to Making Logs Sexy Again: Can We Finally Lose The Regexes (20)

Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinEnterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
 
Baselining Logs
Baselining LogsBaselining Logs
Baselining Logs
 
Logs = Accountability
Logs = AccountabilityLogs = Accountability
Logs = Accountability
 
Pivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsPivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logs
 
Power of logs: practices for network security
Power of logs: practices for network securityPower of logs: practices for network security
Power of logs: practices for network security
 
Log Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinLog Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton Chuvakin
 
Logs vs Insiders
Logs vs InsidersLogs vs Insiders
Logs vs Insiders
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)
 
Python for Data Logistics
Python for Data LogisticsPython for Data Logistics
Python for Data Logistics
 
LogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationLogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log Standardization
 
CSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinCSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
 
Logs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMALogs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMA
 
NIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldNIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real World
 
Functional and non functional application logging
Functional and non functional application loggingFunctional and non functional application logging
Functional and non functional application logging
 
Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016
 
What is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupWhat is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
 
The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
 

More from Anton Chuvakin

Future of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsFuture of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsAnton Chuvakin
 
SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?Anton Chuvakin
 
Meet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton ChuvakinMeet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton ChuvakinAnton Chuvakin
 
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...Anton Chuvakin
 
SOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton ChuvakinSOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton ChuvakinAnton Chuvakin
 
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 BoothHey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 BoothAnton Chuvakin
 
20 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 202220 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 2022Anton Chuvakin
 
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton ChuvakinAnton Chuvakin
 
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC TrendsSOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC TrendsAnton Chuvakin
 
SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC Anton Chuvakin
 
Modern SOC Trends 2020
Modern SOC Trends 2020Modern SOC Trends 2020
Modern SOC Trends 2020Anton Chuvakin
 
Anton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in BriefAnton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in BriefAnton Chuvakin
 
Five SIEM Futures (2012)
Five SIEM Futures (2012)Five SIEM Futures (2012)
Five SIEM Futures (2012)Anton Chuvakin
 
RSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics PresentationRSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics PresentationAnton Chuvakin
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinAnton Chuvakin
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinAnton Chuvakin
 
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton ChuvakinPractical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton ChuvakinAnton Chuvakin
 

More from Anton Chuvakin (20)

Future of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsFuture of SOC: More Security, Less Operations
Future of SOC: More Security, Less Operations
 
SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?
 
Meet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton ChuvakinMeet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton Chuvakin
 
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
 
SOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton ChuvakinSOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton Chuvakin
 
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 BoothHey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
 
20 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 202220 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 2022
 
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
 
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC TrendsSOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
 
SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC
 
Modern SOC Trends 2020
Modern SOC Trends 2020Modern SOC Trends 2020
Modern SOC Trends 2020
 
Anton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in BriefAnton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in Brief
 
Generic siem how_2017
Generic siem how_2017Generic siem how_2017
Generic siem how_2017
 
Tips on SIEM Ops 2015
Tips on SIEM Ops 2015Tips on SIEM Ops 2015
Tips on SIEM Ops 2015
 
Five SIEM Futures (2012)
Five SIEM Futures (2012)Five SIEM Futures (2012)
Five SIEM Futures (2012)
 
RSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics PresentationRSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics Presentation
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
 
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton ChuvakinPractical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
 
SIEM Primer:
SIEM Primer:SIEM Primer:
SIEM Primer:
 

Recently uploaded

Google AI Hackathon: LLM based Evaluator for RAG
Google AI Hackathon: LLM based Evaluator for RAGGoogle AI Hackathon: LLM based Evaluator for RAG
Google AI Hackathon: LLM based Evaluator for RAGSujit Pal
 
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Alan Dix
 
Maximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxMaximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxOnBoard
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Servicegiselly40
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...shyamraj55
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 

Recently uploaded (20)

Google AI Hackathon: LLM based Evaluator for RAG
Google AI Hackathon: LLM based Evaluator for RAGGoogle AI Hackathon: LLM based Evaluator for RAG
Google AI Hackathon: LLM based Evaluator for RAG
 
🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘🐬 The future of MySQL is Postgres 🐘
🐬 The future of MySQL is Postgres 🐘
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...Swan(sea) Song – personal research during my six years at Swansea ... and bey...
Swan(sea) Song – personal research during my six years at Swansea ... and bey...
 
Maximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptxMaximizing Board Effectiveness 2024 Webinar.pptx
Maximizing Board Effectiveness 2024 Webinar.pptx
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...
 
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
 
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure serviceWhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 

Making Logs Sexy Again: Can We Finally Lose The Regexes

  • 1. Making Logs Sexy Again: Can We Finally Lose The Regexes? Dr. Anton Chuvakin
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17. Example Log Chaos - Login? <122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff:192.168.138.35 port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Failure Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0     Logon account :  POWERUSER    <57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-warning-00515: Admin User netscreen has logged on via Telnet from 10.14.98.55:39073 (2002-12-17 15:50:53)
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29. Can You Guess What Happened?!
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36. Log Context Enrichment? C O M P A N Y C O N F I D E N T I A L
  • 37.
  • 38.
  • 39.
  • 40. Backup and Reference Slides C O M P A N Y C O N F I D E N T I A L
  • 41.

Editor's Notes

  1. “ Will be high-level, but the subject is definitely pretty DEEP… so DeepSec” Making Logs Sexy Again: Can We Finally Lose The Regexes?   As I was talking to one conference organizer a few years ago and he complained that there hasn’t been anything new in the field of automated log analysis for many years and that we have not progressed much since regular expressions (regexes) were used for extracting bits of useful information from logs. This talk is intended to change that! It will go through a set of approaches to “log problem”: some completely new (text mining of logs), some borrowed from other fields (Bayesian methods for log analysis) and show how people can use them to solve practical problems using these new methods. Only real-world-tested analysis methods and only those that can be used by the audience will be shown (and, of course, only the “0technically cool” and novel ones!) Example questions that will be answered: are logs data or text? Which data mining methods works really well on logs (which is still simple enough to be used by the non-Ph.Ds)? Can you make sense of all the keywords and key phrases in logs? Does Bayes works for logs? Can you predict when the system will crash based on logs?