SlideShare une entreprise Scribd logo

Using Hadoop to Detect Security Risks and Predict Breaches

•
•
DataWorks Summit
DataWorks Summit
TechnologieFormation
1  sur  26
Caspida – Karthik KannanCaspida Inc. Threat Detection Using Hadoop KARTHIK KANNAN FOUNDER, CMO
Caspida – Karthik Kannan Title  Using Hadoop and Machine Learning to Detect Security Risks and Vulnerabilities, and Predict Breaches in your Enterprise Environment
Caspida – Karthik Kannan Topics  Security challenges  Today’s approaches – limitations  Why is security a Big Data problem?  Hadoop and ML in other industries  Security – with Hadoop and ML  Some examples  Where do you go from here?
Caspida – Karthik Kannan Security Unique use case that applies horizontally  Incident analysis  Anomaly detection  Queries at scale  Predetermined metrics Needs to be dynamically self-learning
Caspida – Karthik Kannan Today's Security Challenges  Target credit card breaches  Snowden insider attack  RSA security breach  Twitter hacking
Caspida – Karthik Kannan CIO Survey : Top Concerns
Caspida – Karthik Kannan Verizon Data Breach Report 2014 DBIR data shows attackers are getting better and faster at what they do, more quickly than organizations can address the threats. http://www.verizonenterprise.com/DBIR/2014/?utm_source=ContextualAds&utm_medium=ResultLinks&utm_c ampaign=DBIR2014
Caspida – Karthik Kannan Market Data Courtesy: Mary Meeker, KPCB, Internet Trends Report, 2014
Caspida – Karthik Kannan Current Methods Fail Limited scale, manual, no dynamicity Signatures Rules Malware-Detection
Caspida – Karthik Kannan Why is Security a Big Data Problem?  Variety of security events − New sources, new relationships, new entities  Analysis sophistication − Dynamic correlations, sequences, non- contiguous patterns  Context – time − Months & years, not days Good reference/reading:
Caspida – Karthik Kannan The Right Tools for the Right Purpose Protecting the perimeter and defending against known attacks (signatures) Discovery unauthorized use of SaaS/cloud apps and policy enablement for Shadow IT SIEMs collect data, use extensive human-generated rules, rely on manual analysis and provide static alerts Firewall, IPS, malware, AV Cloud security SIEM Lacking dynamic, self-learning methods that are needed to detect sophisticated attacks?
Caspida – Karthik Kannan Mobile There is an App for Everything! SMSPhone MMS IM Mobile App Stores Mobile Device Mgmt (MDM) Mobile App Mgmt (MAM) Cloud SaaS Monitoring SaaS Encryption Web Mail CRM/ERP SaaS Apps (Salesforce, …) Custom Apps/TestDev Clouds Desktop Password Hashing Antivirus Anti-Malware SW OS security layering OS-level Sandboxing Disk Encryption Productivity Apps/Development/Test Security in the Technology Evolution Application-specific Attacks (Facebook wall, Browser) Attackers AttackTypes DDoS (Zombies etc.) Password Guessing Filesystems / DBs Misconfigurations Viruses Malware/Spyware Keyloggers Sniffing Governments Special Interest Groups Polymorphic APT Botnets Web App Attacks (XSS, etc.) Phishing Enterprise Firewalls Multi-Factor Authentication IDSAntivirus Malware Sandboxing Threat Feeds SIEMVPN Corporate Email Finance Apps Corporate Storage/Filers Collaboration Tools/ECM Cloud Apps Time2000 20131990 2010 AttackSophistication
Caspida – Karthik Kannan Stages of an attack Research Infiltrate Capture Exfiltrate Market- place 86% of enterprises focus on step 2 only Studies show that companies save up to $4M/year when they have security intelligence systems that focus on all stages 1 2 3 4 5
Caspida – Karthik Kannan ML + Statistical Models Visualization Models Data Lake Standard models: K- means, Random Forest, Nearest-neighbor, Gaussian, Bayesian etc. Custom models: user patterns/behavior, time- oriented, data attributes- specific, SaaS, mobile
Caspida – Karthik Kannan Algorithms  Time Series Analysis − Good when dealing with time series − Examples:  Linear Regression  Parametric (ARIMA/FARIMA)  Forecasting: Holt-Winters  Classification Models − Good to find which categories things are falling under − Examples:  Logistic Regression  Decision Trees  Decision Tables  Neural Networks  K-Nearest Neighbors  Ensemble Models (Random Forests)  Grouping Models − Used for finding global patterns at scale − Examples:  K-Means Clustering  Random graph walks  Inference Models − Important when trying to infer value of a feature from a context − Example  Association Rules  Bayesian Networks  Simplification Models − Important when we need to decrease number of features analyzed − Examples  Principal Component Analysis (PCA)  Low-Rank Approximation  Single-Value Decomposition (SVD)
Caspida – Karthik Kannan Data Sources: Information Value Pyramid Network Packets: L2-L4 Network Packets: L7 Generic System Logs Application Logs Lower Volume; Concentration of Information No need to decipher semantics of information Top-Down view with Correlation on important signals OS logs on system events, processes’ health Need additional deciphering of information High-Volume of Source Data Can capture malware code for analysis Problems with encrypted traffic High-Volume of Source Data Analysis only based on signatures and packet statistics
Caspida – Karthik Kannan Advanced Persistent Threat (APT) Kill Chain A handful set of users targeted by phishing attacks The user downloads the malware which finds a back door to access the system Attacker attempts to move other systems and accounts by elevating privileges accordingly Data is gathered from different systems and staged for exfiltration Data is sent out via multiple channels (encrypted over FTP, DNS back channels etc.) Lateral Movement Phishing and Zero Day Attack Back Door Data Gathering Exfiltrate
Caspida – Karthik Kannan Ideal Hadoop-based solution Data Sources Data Lake Data Science
Caspida – Karthik Kannan Machine Learning in Industries  eCommerce: identify shopper behavior and predict buying patterns, inventory planning, recommendations − AggregateKnowledge − RichRelevance − Amazon  AdTech: identify mobile/online users, model their preferences, and render appropriate advertisements to the right audience − AdMob (Google) − MoPub (Twitter) − Efficient Frontier (Adobe)
Caspida – Karthik Kannan Types of Security Analytics  Breach − Phishing attack − DDoS attack − Watering hole attack  Exploitation − Lateral movement − Domain account misuse  Exfiltration − Privileged data leakage − Anomalous login activity  Debilitation − App or DB server load/activity patterns − Web server patterns  Monitoring − Metrics management
Caspida – Karthik Kannan Data Sources & Analysis Source Information obtained 1 Web server Incoming, outgoing traffic, IP addresses, times, session durations 2 Domain controllers User IDs accessing specific IP addresses, times, durations 3 IAM servers Apps, servers, other protected services users are accessing, times, durations 4 Content servers Detailed transactional histories, customer account data, ACLs 5 Messaging server events Email stats, attachment info, external communications (IPs, frequencies) + correlations – across time and events to produce network of related users, apps, servers and other critical services that may be affected by threats + machine learning algorithms – dynamic models driving automatic insights into malicious, external, APT, SaaS, mobile or network threats in repeatable fashion + search/queries – to sharpen insights and threat intelligence by drilling down into desired dimension such as time window, geography, criticality etc.
Caspida – Karthik Kannan Anatomy of an attack IP Location 200.55.12.68 Brazil 58.202.85.1 China 220.12.98.41 US/SC 119.56.128.25 China … … IP Location 200.55.12.68 Brazil 58.202.85.1 China 220.12.98.41 US/SC 119.56.128.25 China … … UID2 UID1 UID3 UID5 UID4 UID6 UID7 UID9 UID1 UID8 Svr1 Svr2 App1 App2 DB1 FTP1 Identification of suspicious IP originations, destinations IP addresses, geo-spatial information collection Network of correlations for suspected IPs; which users are accessing them the most? Identification of suspicious users Correlations of suspected users with apps, databases and other sensitive services 1 2 3 456 Timeline of malicious behavior, e.g., sending emails or communicating with CnC Actions IP1 IP2 UID8 UID1 UID8 DB2
Caspida – Karthik Kannan Network traffic Behavioral threat models Network Traffic: PCAP, Netflow • Switches • Routers • Firewalls* • IPS’* • Web gateways* • Proxy server* • Any other network device* * optional Sources • Traffic monitoring & analysis: • Which IP is communicating with which external or internal destination • Traffic volume, frequency • Correlate with IAM (for user ID – IP mapping) • Max traffic contributors – users, apps, IP addresses • Correlate with Web server (for URL traffic analysis per user) • Correlate with Messaging server (for email source/recipient analysis) • Correlate with Firewall (for external traffic analysis per IP, user) • Correlate with App, DB servers (for internal app transactional analysis) • External threat (e.g., bad IP address list) feeds Threat Intelligence
Caspida – Karthik Kannan Examples  Ground-speed violations − detect user logins that are geographically spaced apart but fall within seconds/minutes of each other  Lateral movement − accounts moving from one server/device to another to explore and list content on each location before deciding which to exfiltrate  Domain admin creations − auto creation of admin accounts by spurious account; e.g., r00t, adm1n etc.
Caspida – Karthik Kannan Where do you start?  Need a data lake  Analytics: − ML − Statistical  Actions
Caspida – Karthik Kannan Thank you!

Recommandé

Burning Down the Haystack to Find the Needle: Security Analytics in Action
Burning Down the Haystack to Find the Needle: Security Analytics in ActionBurning Down the Haystack to Find the Needle: Security Analytics in Action
Burning Down the Haystack to Find the Needle: Security Analytics in ActionJosh Sokol
 
Confusion and deception new tools for data protection
Confusion and deception new tools for data protectionConfusion and deception new tools for data protection
Confusion and deception new tools for data protectionPriyanka Aash
 
Applied cognitive security complementing the security analyst
Applied cognitive security complementing the security analyst Applied cognitive security complementing the security analyst
Applied cognitive security complementing the security analyst Priyanka Aash
 
Advances in cloud scale machine learning for cyber-defense
Advances in cloud scale machine learning for cyber-defenseAdvances in cloud scale machine learning for cyber-defense
Advances in cloud scale machine learning for cyber-defensePriyanka Aash
 
Hadoop / Spark on Malware Expression
Hadoop / Spark on Malware ExpressionHadoop / Spark on Malware Expression
Hadoop / Spark on Malware ExpressionMapR Technologies
 
SPO2-T11_Automated-Prevention-of-Ransomware-with-Machine-Learning-and-GPOs
SPO2-T11_Automated-Prevention-of-Ransomware-with-Machine-Learning-and-GPOsSPO2-T11_Automated-Prevention-of-Ransomware-with-Machine-Learning-and-GPOs
SPO2-T11_Automated-Prevention-of-Ransomware-with-Machine-Learning-and-GPOsRod Soto
 
Machine learning cybersecurity boon or boondoggle
Machine learning cybersecurity boon or boondoggleMachine learning cybersecurity boon or boondoggle
Machine learning cybersecurity boon or boondogglePriyanka Aash
 
Applied machine learning defeating modern malicious documents
Applied machine learning defeating modern malicious documentsApplied machine learning defeating modern malicious documents
Applied machine learning defeating modern malicious documentsPriyanka Aash
 

Recommandé

Burning Down the Haystack to Find the Needle: Security Analytics in Action
Burning Down the Haystack to Find the Needle: Security Analytics in ActionBurning Down the Haystack to Find the Needle: Security Analytics in Action
Burning Down the Haystack to Find the Needle: Security Analytics in ActionJosh Sokol
 
Confusion and deception new tools for data protection
Confusion and deception new tools for data protectionConfusion and deception new tools for data protection
Confusion and deception new tools for data protectionPriyanka Aash
 
Applied cognitive security complementing the security analyst
Applied cognitive security complementing the security analyst Applied cognitive security complementing the security analyst
Applied cognitive security complementing the security analyst Priyanka Aash
 
Advances in cloud scale machine learning for cyber-defense
Advances in cloud scale machine learning for cyber-defenseAdvances in cloud scale machine learning for cyber-defense
Advances in cloud scale machine learning for cyber-defensePriyanka Aash
 
Hadoop / Spark on Malware Expression
Hadoop / Spark on Malware ExpressionHadoop / Spark on Malware Expression
Hadoop / Spark on Malware ExpressionMapR Technologies
 
SPO2-T11_Automated-Prevention-of-Ransomware-with-Machine-Learning-and-GPOs
SPO2-T11_Automated-Prevention-of-Ransomware-with-Machine-Learning-and-GPOsSPO2-T11_Automated-Prevention-of-Ransomware-with-Machine-Learning-and-GPOs
SPO2-T11_Automated-Prevention-of-Ransomware-with-Machine-Learning-and-GPOsRod Soto
 
Machine learning cybersecurity boon or boondoggle
Machine learning cybersecurity boon or boondoggleMachine learning cybersecurity boon or boondoggle
Machine learning cybersecurity boon or boondogglePriyanka Aash
 
Applied machine learning defeating modern malicious documents
Applied machine learning defeating modern malicious documentsApplied machine learning defeating modern malicious documents
Applied machine learning defeating modern malicious documentsPriyanka Aash
 
Cloudslam09:Building a Cloud Computing Analysis System for Intrusion Detection
Cloudslam09:Building a Cloud Computing Analysis System for Intrusion DetectionCloudslam09:Building a Cloud Computing Analysis System for Intrusion Detection
Cloudslam09:Building a Cloud Computing Analysis System for Intrusion DetectionWei-Yu Chen
 
Hunting: Defense Against The Dark Arts
Hunting: Defense Against The Dark ArtsHunting: Defense Against The Dark Arts
Hunting: Defense Against The Dark ArtsSpyglass Security
 
Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Pl...
Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Pl...Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Pl...
Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Pl...Carolyn Duby
 
Splunk Live! Utrecht 2016 - CERT EU
Splunk Live! Utrecht 2016 - CERT EUSplunk Live! Utrecht 2016 - CERT EU
Splunk Live! Utrecht 2016 - CERT EUSplunk
 
Network Security Data Visualization
Network Security Data VisualizationNetwork Security Data Visualization
Network Security Data Visualizationamiable_indian
 
Threat intel- -content-curation-organizing-the-path-to-successful-detection
Threat intel- -content-curation-organizing-the-path-to-successful-detectionThreat intel- -content-curation-organizing-the-path-to-successful-detection
Threat intel- -content-curation-organizing-the-path-to-successful-detectionPriyanka Aash
 
How to Hunt for Lateral Movement on Your Network
How to Hunt for Lateral Movement on Your NetworkHow to Hunt for Lateral Movement on Your Network
How to Hunt for Lateral Movement on Your NetworkSqrrl
 
Apache metron meetup presentation at capital one
Apache metron meetup presentation at capital oneApache metron meetup presentation at capital one
Apache metron meetup presentation at capital onegvetticaden
 
Soc 2030-socs-are-broken-lets-fix- them
Soc 2030-socs-are-broken-lets-fix- themSoc 2030-socs-are-broken-lets-fix- them
Soc 2030-socs-are-broken-lets-fix- themPriyanka Aash
 
Insights from-NSAs-cybersecurity-threat-operations-center
Insights from-NSAs-cybersecurity-threat-operations-centerInsights from-NSAs-cybersecurity-threat-operations-center
Insights from-NSAs-cybersecurity-threat-operations-centerPriyanka Aash
 
Cybersecurity - Jim Butterworth
Cybersecurity - Jim ButterworthCybersecurity - Jim Butterworth
Cybersecurity - Jim ButterworthTechBiz Forense Digital
 
Apache Metron: Community Driven Cyber Security
Apache Metron: Community Driven Cyber Security Apache Metron: Community Driven Cyber Security
Apache Metron: Community Driven Cyber Security DataWorks Summit/Hadoop Summit
 
Novetta Cyber Analytics
Novetta Cyber AnalyticsNovetta Cyber Analytics
Novetta Cyber AnalyticsNovetta
 
Insider Threat Visualization - HITB 2007, Kuala Lumpur
Insider Threat Visualization - HITB 2007, Kuala LumpurInsider Threat Visualization - HITB 2007, Kuala Lumpur
Insider Threat Visualization - HITB 2007, Kuala LumpurRaffael Marty
 
PHDays 2018 Threat Hunting Hands-On Lab
PHDays 2018 Threat Hunting Hands-On LabPHDays 2018 Threat Hunting Hands-On Lab
PHDays 2018 Threat Hunting Hands-On LabTeymur Kheirkhabarov
 
Incident Response: Validation, Containment & Forensics
Incident Response: Validation, Containment & Forensics Incident Response: Validation, Containment & Forensics
Incident Response: Validation, Containment & ForensicsPriyanka Aash
 
Attacks on Critical Infrastructure: Insights from the “Big Board”
Attacks on Critical Infrastructure: Insights from the “Big Board”Attacks on Critical Infrastructure: Insights from the “Big Board”
Attacks on Critical Infrastructure: Insights from the “Big Board”Priyanka Aash
 
DNS Measurement Activity on ITB 2010
DNS Measurement Activity on ITB 2010DNS Measurement Activity on ITB 2010
DNS Measurement Activity on ITB 2010Affan Basalamah
 
Threat-Based Adversary Emulation with MITRE ATT&CK
Threat-Based Adversary Emulation with MITRE ATT&CKThreat-Based Adversary Emulation with MITRE ATT&CK
Threat-Based Adversary Emulation with MITRE ATT&CKKatie Nickels
 
Hunting on the Cheap
Hunting on the CheapHunting on the Cheap
Hunting on the CheapEndgameInc
 
Performing Network & Security Analytics with Hadoop
Performing Network & Security Analytics with HadoopPerforming Network & Security Analytics with Hadoop
Performing Network & Security Analytics with HadoopDataWorks Summit
 
Cisco OpenSOC
Cisco OpenSOCCisco OpenSOC
Cisco OpenSOCJames Sirota
 

Contenu connexe

Tendances

Cloudslam09:Building a Cloud Computing Analysis System for Intrusion Detection
Cloudslam09:Building a Cloud Computing Analysis System for Intrusion DetectionCloudslam09:Building a Cloud Computing Analysis System for Intrusion Detection
Cloudslam09:Building a Cloud Computing Analysis System for Intrusion DetectionWei-Yu Chen
 
Hunting: Defense Against The Dark Arts
Hunting: Defense Against The Dark ArtsHunting: Defense Against The Dark Arts
Hunting: Defense Against The Dark ArtsSpyglass Security
 
Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Pl...
Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Pl...Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Pl...
Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Pl...Carolyn Duby
 
Splunk Live! Utrecht 2016 - CERT EU
Splunk Live! Utrecht 2016 - CERT EUSplunk Live! Utrecht 2016 - CERT EU
Splunk Live! Utrecht 2016 - CERT EUSplunk
 
Network Security Data Visualization
Network Security Data VisualizationNetwork Security Data Visualization
Network Security Data Visualizationamiable_indian
 
Threat intel- -content-curation-organizing-the-path-to-successful-detection
Threat intel- -content-curation-organizing-the-path-to-successful-detectionThreat intel- -content-curation-organizing-the-path-to-successful-detection
Threat intel- -content-curation-organizing-the-path-to-successful-detectionPriyanka Aash
 
How to Hunt for Lateral Movement on Your Network
How to Hunt for Lateral Movement on Your NetworkHow to Hunt for Lateral Movement on Your Network
How to Hunt for Lateral Movement on Your NetworkSqrrl
 
Apache metron meetup presentation at capital one
Apache metron meetup presentation at capital oneApache metron meetup presentation at capital one
Apache metron meetup presentation at capital onegvetticaden
 
Soc 2030-socs-are-broken-lets-fix- them
Soc 2030-socs-are-broken-lets-fix- themSoc 2030-socs-are-broken-lets-fix- them
Soc 2030-socs-are-broken-lets-fix- themPriyanka Aash
 
Insights from-NSAs-cybersecurity-threat-operations-center
Insights from-NSAs-cybersecurity-threat-operations-centerInsights from-NSAs-cybersecurity-threat-operations-center
Insights from-NSAs-cybersecurity-threat-operations-centerPriyanka Aash
 
Novetta Cyber Analytics
Novetta Cyber AnalyticsNovetta Cyber Analytics
Novetta Cyber AnalyticsNovetta
 
Insider Threat Visualization - HITB 2007, Kuala Lumpur
Insider Threat Visualization - HITB 2007, Kuala LumpurInsider Threat Visualization - HITB 2007, Kuala Lumpur
Insider Threat Visualization - HITB 2007, Kuala LumpurRaffael Marty
 
PHDays 2018 Threat Hunting Hands-On Lab
PHDays 2018 Threat Hunting Hands-On LabPHDays 2018 Threat Hunting Hands-On Lab
PHDays 2018 Threat Hunting Hands-On LabTeymur Kheirkhabarov
 
Incident Response: Validation, Containment & Forensics
Incident Response: Validation, Containment & Forensics Incident Response: Validation, Containment & Forensics
Incident Response: Validation, Containment & ForensicsPriyanka Aash
 
Attacks on Critical Infrastructure: Insights from the “Big Board”
Attacks on Critical Infrastructure: Insights from the “Big Board”Attacks on Critical Infrastructure: Insights from the “Big Board”
Attacks on Critical Infrastructure: Insights from the “Big Board”Priyanka Aash
 
DNS Measurement Activity on ITB 2010
DNS Measurement Activity on ITB 2010DNS Measurement Activity on ITB 2010
DNS Measurement Activity on ITB 2010Affan Basalamah
 
Threat-Based Adversary Emulation with MITRE ATT&CK
Threat-Based Adversary Emulation with MITRE ATT&CKThreat-Based Adversary Emulation with MITRE ATT&CK
Threat-Based Adversary Emulation with MITRE ATT&CKKatie Nickels
 
Hunting on the Cheap
Hunting on the CheapHunting on the Cheap
Hunting on the CheapEndgameInc
 

Tendances (20)

Cloudslam09:Building a Cloud Computing Analysis System for Intrusion Detection
Cloudslam09:Building a Cloud Computing Analysis System for Intrusion DetectionCloudslam09:Building a Cloud Computing Analysis System for Intrusion Detection
Cloudslam09:Building a Cloud Computing Analysis System for Intrusion Detection
 
Hunting: Defense Against The Dark Arts
Hunting: Defense Against The Dark ArtsHunting: Defense Against The Dark Arts
Hunting: Defense Against The Dark Arts
 
Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Pl...
Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Pl...Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Pl...
Providence Future of Data Meetup - Apache Metron Open Source Cybersecurity Pl...
 
Splunk Live! Utrecht 2016 - CERT EU
Splunk Live! Utrecht 2016 - CERT EUSplunk Live! Utrecht 2016 - CERT EU
Splunk Live! Utrecht 2016 - CERT EU
 
Network Security Data Visualization
Network Security Data VisualizationNetwork Security Data Visualization
Network Security Data Visualization
 
Threat intel- -content-curation-organizing-the-path-to-successful-detection
Threat intel- -content-curation-organizing-the-path-to-successful-detectionThreat intel- -content-curation-organizing-the-path-to-successful-detection
Threat intel- -content-curation-organizing-the-path-to-successful-detection
 
How to Hunt for Lateral Movement on Your Network
How to Hunt for Lateral Movement on Your NetworkHow to Hunt for Lateral Movement on Your Network
How to Hunt for Lateral Movement on Your Network
 
Apache metron meetup presentation at capital one
Apache metron meetup presentation at capital oneApache metron meetup presentation at capital one
Apache metron meetup presentation at capital one
 
Soc 2030-socs-are-broken-lets-fix- them
Soc 2030-socs-are-broken-lets-fix- themSoc 2030-socs-are-broken-lets-fix- them
Soc 2030-socs-are-broken-lets-fix- them
 
Insights from-NSAs-cybersecurity-threat-operations-center
Insights from-NSAs-cybersecurity-threat-operations-centerInsights from-NSAs-cybersecurity-threat-operations-center
Insights from-NSAs-cybersecurity-threat-operations-center
 
Cybersecurity - Jim Butterworth
Cybersecurity - Jim ButterworthCybersecurity - Jim Butterworth
Cybersecurity - Jim Butterworth
 
Apache Metron: Community Driven Cyber Security
Apache Metron: Community Driven Cyber Security Apache Metron: Community Driven Cyber Security
Apache Metron: Community Driven Cyber Security
 
Novetta Cyber Analytics
Novetta Cyber AnalyticsNovetta Cyber Analytics
Novetta Cyber Analytics
 
Insider Threat Visualization - HITB 2007, Kuala Lumpur
Insider Threat Visualization - HITB 2007, Kuala LumpurInsider Threat Visualization - HITB 2007, Kuala Lumpur
Insider Threat Visualization - HITB 2007, Kuala Lumpur
 
PHDays 2018 Threat Hunting Hands-On Lab
PHDays 2018 Threat Hunting Hands-On LabPHDays 2018 Threat Hunting Hands-On Lab
PHDays 2018 Threat Hunting Hands-On Lab
 
Incident Response: Validation, Containment & Forensics
Incident Response: Validation, Containment & Forensics Incident Response: Validation, Containment & Forensics
Incident Response: Validation, Containment & Forensics
 
Attacks on Critical Infrastructure: Insights from the “Big Board”
Attacks on Critical Infrastructure: Insights from the “Big Board”Attacks on Critical Infrastructure: Insights from the “Big Board”
Attacks on Critical Infrastructure: Insights from the “Big Board”
 
DNS Measurement Activity on ITB 2010
DNS Measurement Activity on ITB 2010DNS Measurement Activity on ITB 2010
DNS Measurement Activity on ITB 2010
 
Threat-Based Adversary Emulation with MITRE ATT&CK
Threat-Based Adversary Emulation with MITRE ATT&CKThreat-Based Adversary Emulation with MITRE ATT&CK
Threat-Based Adversary Emulation with MITRE ATT&CK
 
Hunting on the Cheap
Hunting on the CheapHunting on the Cheap
Hunting on the Cheap
 

En vedette

Performing Network & Security Analytics with Hadoop
Performing Network & Security Analytics with HadoopPerforming Network & Security Analytics with Hadoop
Performing Network & Security Analytics with HadoopDataWorks Summit
 
FINAL_SCFm50000_JonPapp_CAA_The_Practical_Benefits_of_a_Behavioral_Solution_f...
FINAL_SCFm50000_JonPapp_CAA_The_Practical_Benefits_of_a_Behavioral_Solution_f...FINAL_SCFm50000_JonPapp_CAA_The_Practical_Benefits_of_a_Behavioral_Solution_f...
FINAL_SCFm50000_JonPapp_CAA_The_Practical_Benefits_of_a_Behavioral_Solution_f...Jon Papp
 
Project report
Project reportProject report
Project reportArijit Roy
 
Agile Data: Building Hadoop Analytics Applications
Agile Data: Building Hadoop Analytics ApplicationsAgile Data: Building Hadoop Analytics Applications
Agile Data: Building Hadoop Analytics ApplicationsDataWorks Summit
 
One Flaw over the Cuckoo's Nest
One Flaw over the Cuckoo's NestOne Flaw over the Cuckoo's Nest
One Flaw over the Cuckoo's NestIñaki Rodríguez
 
5 Ways to Protect Your Healthcare Organization from a Ransomware Attack - HIM...
5 Ways to Protect Your Healthcare Organization from a Ransomware Attack - HIM...5 Ways to Protect Your Healthcare Organization from a Ransomware Attack - HIM...
5 Ways to Protect Your Healthcare Organization from a Ransomware Attack - HIM...ClearDATACloud
 
Big data y la inteligencia de negocios
Big data y la inteligencia de negociosBig data y la inteligencia de negocios
Big data y la inteligencia de negociosnnakasone
 
Integrated Tools in OSSIM
Integrated Tools in OSSIMIntegrated Tools in OSSIM
Integrated Tools in OSSIMAlienVault
 
How Apache Spark fits in the Big Data landscape
How Apache Spark fits in the Big Data landscapeHow Apache Spark fits in the Big Data landscape
How Apache Spark fits in the Big Data landscapePaco Nathan
 
5 Myths about Spark and Big Data by Nik Rouda
5 Myths about Spark and Big Data by Nik Rouda5 Myths about Spark and Big Data by Nik Rouda
5 Myths about Spark and Big Data by Nik RoudaSpark Summit
 
How Apache Spark fits into the Big Data landscape
How Apache Spark fits into the Big Data landscapeHow Apache Spark fits into the Big Data landscape
How Apache Spark fits into the Big Data landscapePaco Nathan
 
Madrid Spark Big Data Bluemix Meetup - Spark Versus Hadoop @ 100 TB Daytona G...
Madrid Spark Big Data Bluemix Meetup - Spark Versus Hadoop @ 100 TB Daytona G...Madrid Spark Big Data Bluemix Meetup - Spark Versus Hadoop @ 100 TB Daytona G...
Madrid Spark Big Data Bluemix Meetup - Spark Versus Hadoop @ 100 TB Daytona G...Chris Fregly
 
Hadoop, Spark and Big Data Summit presentation with SAP HANA Vora and a path ...
Hadoop, Spark and Big Data Summit presentation with SAP HANA Vora and a path ...Hadoop, Spark and Big Data Summit presentation with SAP HANA Vora and a path ...
Hadoop, Spark and Big Data Summit presentation with SAP HANA Vora and a path ...Ocean9, Inc.
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationRaffael Marty
 
Advanced persistent threat (apt)
Advanced persistent threat (apt)Advanced persistent threat (apt)
Advanced persistent threat (apt)mmubashirkhan
 
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...Databricks
 
Building Deep Learning Powered Big Data: Spark Summit East talk by Jiao Wang ...
Building Deep Learning Powered Big Data: Spark Summit East talk by Jiao Wang ...Building Deep Learning Powered Big Data: Spark Summit East talk by Jiao Wang ...
Building Deep Learning Powered Big Data: Spark Summit East talk by Jiao Wang ...Spark Summit
 

En vedette (20)

Performing Network & Security Analytics with Hadoop
Performing Network & Security Analytics with HadoopPerforming Network & Security Analytics with Hadoop
Performing Network & Security Analytics with Hadoop
 
Cisco OpenSOC
Cisco OpenSOCCisco OpenSOC
Cisco OpenSOC
 
FINAL_SCFm50000_JonPapp_CAA_The_Practical_Benefits_of_a_Behavioral_Solution_f...
FINAL_SCFm50000_JonPapp_CAA_The_Practical_Benefits_of_a_Behavioral_Solution_f...FINAL_SCFm50000_JonPapp_CAA_The_Practical_Benefits_of_a_Behavioral_Solution_f...
FINAL_SCFm50000_JonPapp_CAA_The_Practical_Benefits_of_a_Behavioral_Solution_f...
 
Project report
Project reportProject report
Project report
 
Agile Data: Building Hadoop Analytics Applications
Agile Data: Building Hadoop Analytics ApplicationsAgile Data: Building Hadoop Analytics Applications
Agile Data: Building Hadoop Analytics Applications
 
44Con Malware Workshop
44Con Malware Workshop44Con Malware Workshop
44Con Malware Workshop
 
One Flaw over the Cuckoo's Nest
One Flaw over the Cuckoo's NestOne Flaw over the Cuckoo's Nest
One Flaw over the Cuckoo's Nest
 
5 Ways to Protect Your Healthcare Organization from a Ransomware Attack - HIM...
5 Ways to Protect Your Healthcare Organization from a Ransomware Attack - HIM...5 Ways to Protect Your Healthcare Organization from a Ransomware Attack - HIM...
5 Ways to Protect Your Healthcare Organization from a Ransomware Attack - HIM...
 
Big data y la inteligencia de negocios
Big data y la inteligencia de negociosBig data y la inteligencia de negocios
Big data y la inteligencia de negocios
 
Integrated Tools in OSSIM
Integrated Tools in OSSIMIntegrated Tools in OSSIM
Integrated Tools in OSSIM
 
How Apache Spark fits in the Big Data landscape
How Apache Spark fits in the Big Data landscapeHow Apache Spark fits in the Big Data landscape
How Apache Spark fits in the Big Data landscape
 
5 Myths about Spark and Big Data by Nik Rouda
5 Myths about Spark and Big Data by Nik Rouda5 Myths about Spark and Big Data by Nik Rouda
5 Myths about Spark and Big Data by Nik Rouda
 
How Apache Spark fits into the Big Data landscape
How Apache Spark fits into the Big Data landscapeHow Apache Spark fits into the Big Data landscape
How Apache Spark fits into the Big Data landscape
 
Madrid Spark Big Data Bluemix Meetup - Spark Versus Hadoop @ 100 TB Daytona G...
Madrid Spark Big Data Bluemix Meetup - Spark Versus Hadoop @ 100 TB Daytona G...Madrid Spark Big Data Bluemix Meetup - Spark Versus Hadoop @ 100 TB Daytona G...
Madrid Spark Big Data Bluemix Meetup - Spark Versus Hadoop @ 100 TB Daytona G...
 
Hadoop, Spark and Big Data Summit presentation with SAP HANA Vora and a path ...
Hadoop, Spark and Big Data Summit presentation with SAP HANA Vora and a path ...Hadoop, Spark and Big Data Summit presentation with SAP HANA Vora and a path ...
Hadoop, Spark and Big Data Summit presentation with SAP HANA Vora and a path ...
 
Creating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & VisualizationCreating Your Own Threat Intel Through Hunting & Visualization
Creating Your Own Threat Intel Through Hunting & Visualization
 
Advanced persistent threat (apt)
Advanced persistent threat (apt)Advanced persistent threat (apt)
Advanced persistent threat (apt)
 
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
 
Building Deep Learning Powered Big Data: Spark Summit East talk by Jiao Wang ...
Building Deep Learning Powered Big Data: Spark Summit East talk by Jiao Wang ...Building Deep Learning Powered Big Data: Spark Summit East talk by Jiao Wang ...
Building Deep Learning Powered Big Data: Spark Summit East talk by Jiao Wang ...
 
Big Data Security and Governance
Big Data Security and GovernanceBig Data Security and Governance
Big Data Security and Governance
 

Similaire à Using Hadoop to Detect Security Risks and Predict Breaches

Criminal IP ASM | Threat Intelligence-based Automated Attack Surface Managem...
Criminal IP ASM | Threat Intelligence-based Automated Attack Surface Managem...Criminal IP ASM | Threat Intelligence-based Automated Attack Surface Managem...
Criminal IP ASM | Threat Intelligence-based Automated Attack Surface Managem...Criminal IP
 
Presentation data security solutions certified ibm business partner for ibm...
Presentation data security solutions certified ibm business partner for ibm...Presentation data security solutions certified ibm business partner for ibm...
Presentation data security solutions certified ibm business partner for ibm...xKinAnx
 
Data security solutions_Baltics_IBM_QRadar_SIEM_Use_Cases_28.01.2014
Data security solutions_Baltics_IBM_QRadar_SIEM_Use_Cases_28.01.2014Data security solutions_Baltics_IBM_QRadar_SIEM_Use_Cases_28.01.2014
Data security solutions_Baltics_IBM_QRadar_SIEM_Use_Cases_28.01.2014Andris Soroka
 
Using Event Processing to Enable Enterprise Security
Using Event Processing to Enable Enterprise SecurityUsing Event Processing to Enable Enterprise Security
Using Event Processing to Enable Enterprise SecurityTim Bass
 
How to protect your corporate from advanced attacks
How to protect your corporate from advanced attacksHow to protect your corporate from advanced attacks
How to protect your corporate from advanced attacksMicrosoft
 
Trends in network security feinstein - informatica64
Trends in network security feinstein - informatica64Trends in network security feinstein - informatica64
Trends in network security feinstein - informatica64Chema Alonso
 
Dhishant -Latest Resume
Dhishant -Latest ResumeDhishant -Latest Resume
Dhishant -Latest ResumeDhishant Abrol
 
DETECTE E INVESTIGUE LAS AMENAZAS AVANZADAS
DETECTE E INVESTIGUE LAS AMENAZAS AVANZADAS DETECTE E INVESTIGUE LAS AMENAZAS AVANZADAS
DETECTE E INVESTIGUE LAS AMENAZAS AVANZADAS Cristian Garcia G.
 
Splunk for Security Breakout Session
Splunk for Security Breakout SessionSplunk for Security Breakout Session
Splunk for Security Breakout SessionSplunk
 
Security Breakout Session
Security Breakout Session Security Breakout Session
Security Breakout Session Splunk
 
Insights Into Modern Day Threat Protection
Insights Into Modern Day Threat ProtectionInsights Into Modern Day Threat Protection
Insights Into Modern Day Threat ProtectionAbhinav Biswas
 
"Evolving Cybersecurity Strategies" - Identity is the new security boundary
"Evolving Cybersecurity Strategies" - Identity is the new security boundary"Evolving Cybersecurity Strategies" - Identity is the new security boundary
"Evolving Cybersecurity Strategies" - Identity is the new security boundaryDean Iacovelli
 
Leverage Big Data for Security Intelligence
Leverage Big Data for Security Intelligence Leverage Big Data for Security Intelligence
Leverage Big Data for Security Intelligence Stefaan Van daele
 
Operational Security Intelligence
Operational Security IntelligenceOperational Security Intelligence
Operational Security IntelligenceSplunk
 
So You Want a Job in Cybersecurity
So You Want a Job in CybersecuritySo You Want a Job in Cybersecurity
So You Want a Job in CybersecurityTeri Radichel
 
Road map for actionable threat intelligence
Road map for actionable threat intelligenceRoad map for actionable threat intelligence
Road map for actionable threat intelligenceabhisheksinghcs
 
Anatomy of a cyber attack
Anatomy of a cyber attackAnatomy of a cyber attack
Anatomy of a cyber attackMark Silver
 
Container Workload Security Solution Ideas by Mandy Sidana.pptx
Container Workload Security Solution Ideas by Mandy Sidana.pptxContainer Workload Security Solution Ideas by Mandy Sidana.pptx
Container Workload Security Solution Ideas by Mandy Sidana.pptxMandy Sidana
 

Similaire à Using Hadoop to Detect Security Risks and Predict Breaches (20)

Criminal IP ASM | Threat Intelligence-based Automated Attack Surface Managem...
Criminal IP ASM | Threat Intelligence-based Automated Attack Surface Managem...Criminal IP ASM | Threat Intelligence-based Automated Attack Surface Managem...
Criminal IP ASM | Threat Intelligence-based Automated Attack Surface Managem...
 
Presentation data security solutions certified ibm business partner for ibm...
Presentation data security solutions certified ibm business partner for ibm...Presentation data security solutions certified ibm business partner for ibm...
Presentation data security solutions certified ibm business partner for ibm...
 
Data security solutions_Baltics_IBM_QRadar_SIEM_Use_Cases_28.01.2014
Data security solutions_Baltics_IBM_QRadar_SIEM_Use_Cases_28.01.2014Data security solutions_Baltics_IBM_QRadar_SIEM_Use_Cases_28.01.2014
Data security solutions_Baltics_IBM_QRadar_SIEM_Use_Cases_28.01.2014
 
Using Event Processing to Enable Enterprise Security
Using Event Processing to Enable Enterprise SecurityUsing Event Processing to Enable Enterprise Security
Using Event Processing to Enable Enterprise Security
 
How to protect your corporate from advanced attacks
How to protect your corporate from advanced attacksHow to protect your corporate from advanced attacks
How to protect your corporate from advanced attacks
 
Trends in network security feinstein - informatica64
Trends in network security feinstein - informatica64Trends in network security feinstein - informatica64
Trends in network security feinstein - informatica64
 
Dhishant -Latest Resume
Dhishant -Latest ResumeDhishant -Latest Resume
Dhishant -Latest Resume
 
DETECTE E INVESTIGUE LAS AMENAZAS AVANZADAS
DETECTE E INVESTIGUE LAS AMENAZAS AVANZADAS DETECTE E INVESTIGUE LAS AMENAZAS AVANZADAS
DETECTE E INVESTIGUE LAS AMENAZAS AVANZADAS
 
Splunk for Security Breakout Session
Splunk for Security Breakout SessionSplunk for Security Breakout Session
Splunk for Security Breakout Session
 
Security Breakout Session
Security Breakout Session Security Breakout Session
Security Breakout Session
 
Insights Into Modern Day Threat Protection
Insights Into Modern Day Threat ProtectionInsights Into Modern Day Threat Protection
Insights Into Modern Day Threat Protection
 
"Evolving Cybersecurity Strategies" - Identity is the new security boundary
"Evolving Cybersecurity Strategies" - Identity is the new security boundary"Evolving Cybersecurity Strategies" - Identity is the new security boundary
"Evolving Cybersecurity Strategies" - Identity is the new security boundary
 
Leverage Big Data for Security Intelligence
Leverage Big Data for Security Intelligence Leverage Big Data for Security Intelligence
Leverage Big Data for Security Intelligence
 
Operational Security Intelligence
Operational Security IntelligenceOperational Security Intelligence
Operational Security Intelligence
 
So You Want a Job in Cybersecurity
So You Want a Job in CybersecuritySo You Want a Job in Cybersecurity
So You Want a Job in Cybersecurity
 
cybersecurity-careers.pdf
cybersecurity-careers.pdfcybersecurity-careers.pdf
cybersecurity-careers.pdf
 
Road map for actionable threat intelligence
Road map for actionable threat intelligenceRoad map for actionable threat intelligence
Road map for actionable threat intelligence
 
Anatomy of a cyber attack
Anatomy of a cyber attackAnatomy of a cyber attack
Anatomy of a cyber attack
 
IBM Security QRadar
IBM Security QRadar IBM Security QRadar
IBM Security QRadar
 
Container Workload Security Solution Ideas by Mandy Sidana.pptx
Container Workload Security Solution Ideas by Mandy Sidana.pptxContainer Workload Security Solution Ideas by Mandy Sidana.pptx
Container Workload Security Solution Ideas by Mandy Sidana.pptx
 

Plus de DataWorks Summit

Floating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache RatisFloating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache RatisDataWorks Summit
 
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFiTracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFiDataWorks Summit
 
HBase Tales From the Trenches - Short stories about most common HBase operati...
HBase Tales From the Trenches - Short stories about most common HBase operati...HBase Tales From the Trenches - Short stories about most common HBase operati...
HBase Tales From the Trenches - Short stories about most common HBase operati...DataWorks Summit
 
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...DataWorks Summit
 
Managing the Dewey Decimal System
Managing the Dewey Decimal SystemManaging the Dewey Decimal System
Managing the Dewey Decimal SystemDataWorks Summit
 
Practical NoSQL: Accumulo's dirlist Example
Practical NoSQL: Accumulo's dirlist ExamplePractical NoSQL: Accumulo's dirlist Example
Practical NoSQL: Accumulo's dirlist ExampleDataWorks Summit
 
HBase Global Indexing to support large-scale data ingestion at Uber
HBase Global Indexing to support large-scale data ingestion at UberHBase Global Indexing to support large-scale data ingestion at Uber
HBase Global Indexing to support large-scale data ingestion at UberDataWorks Summit
 
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Scaling Cloud-Scale Translytics Workloads with Omid and PhoenixScaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Scaling Cloud-Scale Translytics Workloads with Omid and PhoenixDataWorks Summit
 
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFiBuilding the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFiDataWorks Summit
 
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability ImprovementsSupporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability ImprovementsDataWorks Summit
 
Security Framework for Multitenant Architecture
Security Framework for Multitenant ArchitectureSecurity Framework for Multitenant Architecture
Security Framework for Multitenant ArchitectureDataWorks Summit
 
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto: Optimizing Performance of SQL-on-Anything EnginePresto: Optimizing Performance of SQL-on-Anything Engine
Presto: Optimizing Performance of SQL-on-Anything EngineDataWorks Summit
 
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...DataWorks Summit
 
Extending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google CloudExtending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google CloudDataWorks Summit
 
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Event-Driven Messaging and Actions using Apache Flink and Apache NiFiEvent-Driven Messaging and Actions using Apache Flink and Apache NiFi
Event-Driven Messaging and Actions using Apache Flink and Apache NiFiDataWorks Summit
 
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Securing Data in Hybrid on-premise and Cloud Environments using Apache RangerSecuring Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Securing Data in Hybrid on-premise and Cloud Environments using Apache RangerDataWorks Summit
 
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...DataWorks Summit
 
Computer Vision: Coming to a Store Near You
Computer Vision: Coming to a Store Near YouComputer Vision: Coming to a Store Near You
Computer Vision: Coming to a Store Near YouDataWorks Summit
 
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Big Data Genomics: Clustering Billions of DNA Sequences with Apache SparkBig Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Big Data Genomics: Clustering Billions of DNA Sequences with Apache SparkDataWorks Summit
 

Plus de DataWorks Summit (20)

Data Science Crash Course
Data Science Crash CourseData Science Crash Course
Data Science Crash Course
 
Floating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache RatisFloating on a RAFT: HBase Durability with Apache Ratis
Floating on a RAFT: HBase Durability with Apache Ratis
 
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFiTracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi
 
HBase Tales From the Trenches - Short stories about most common HBase operati...
HBase Tales From the Trenches - Short stories about most common HBase operati...HBase Tales From the Trenches - Short stories about most common HBase operati...
HBase Tales From the Trenches - Short stories about most common HBase operati...
 
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...
 
Managing the Dewey Decimal System
Managing the Dewey Decimal SystemManaging the Dewey Decimal System
Managing the Dewey Decimal System
 
Practical NoSQL: Accumulo's dirlist Example
Practical NoSQL: Accumulo's dirlist ExamplePractical NoSQL: Accumulo's dirlist Example
Practical NoSQL: Accumulo's dirlist Example
 
HBase Global Indexing to support large-scale data ingestion at Uber
HBase Global Indexing to support large-scale data ingestion at UberHBase Global Indexing to support large-scale data ingestion at Uber
HBase Global Indexing to support large-scale data ingestion at Uber
 
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Scaling Cloud-Scale Translytics Workloads with Omid and PhoenixScaling Cloud-Scale Translytics Workloads with Omid and Phoenix
Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix
 
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFiBuilding the High Speed Cybersecurity Data Pipeline Using Apache NiFi
Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi
 
Supporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability ImprovementsSupporting Apache HBase : Troubleshooting and Supportability Improvements
Supporting Apache HBase : Troubleshooting and Supportability Improvements
 
Security Framework for Multitenant Architecture
Security Framework for Multitenant ArchitectureSecurity Framework for Multitenant Architecture
Security Framework for Multitenant Architecture
 
Presto: Optimizing Performance of SQL-on-Anything Engine
Presto: Optimizing Performance of SQL-on-Anything EnginePresto: Optimizing Performance of SQL-on-Anything Engine
Presto: Optimizing Performance of SQL-on-Anything Engine
 
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...
 
Extending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google CloudExtending Twitter's Data Platform to Google Cloud
Extending Twitter's Data Platform to Google Cloud
 
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
Event-Driven Messaging and Actions using Apache Flink and Apache NiFiEvent-Driven Messaging and Actions using Apache Flink and Apache NiFi
Event-Driven Messaging and Actions using Apache Flink and Apache NiFi
 
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Securing Data in Hybrid on-premise and Cloud Environments using Apache RangerSecuring Data in Hybrid on-premise and Cloud Environments using Apache Ranger
Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger
 
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...
 
Computer Vision: Coming to a Store Near You
Computer Vision: Coming to a Store Near YouComputer Vision: Coming to a Store Near You
Computer Vision: Coming to a Store Near You
 
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Big Data Genomics: Clustering Billions of DNA Sequences with Apache SparkBig Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark
 

Dernier

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?Antenna Manufacturer Coco
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessPixlogix Infotech
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024The Digital Insurer
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024Results
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Servicegiselly40
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?Igalia
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls
 

Dernier (20)

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
Advantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your BusinessAdvantages of Hiring UIUX Design Service Providers for Your Business
Advantages of Hiring UIUX Design Service Providers for Your Business
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men08448380779 Call Girls In Greater Kailash - I Women Seeking Men
08448380779 Call Girls In Greater Kailash - I Women Seeking Men
 

Using Hadoop to Detect Security Risks and Predict Breaches

  • 1. Caspida – Karthik KannanCaspida Inc. Threat Detection Using Hadoop KARTHIK KANNAN FOUNDER, CMO
  • 2. Caspida – Karthik Kannan Title  Using Hadoop and Machine Learning to Detect Security Risks and Vulnerabilities, and Predict Breaches in your Enterprise Environment
  • 3. Caspida – Karthik Kannan Topics  Security challenges  Today’s approaches – limitations  Why is security a Big Data problem?  Hadoop and ML in other industries  Security – with Hadoop and ML  Some examples  Where do you go from here?
  • 4. Caspida – Karthik Kannan Security Unique use case that applies horizontally  Incident analysis  Anomaly detection  Queries at scale  Predetermined metrics Needs to be dynamically self-learning
  • 5. Caspida – Karthik Kannan Today's Security Challenges  Target credit card breaches  Snowden insider attack  RSA security breach  Twitter hacking
  • 6. Caspida – Karthik Kannan CIO Survey : Top Concerns
  • 7. Caspida – Karthik Kannan Verizon Data Breach Report 2014 DBIR data shows attackers are getting better and faster at what they do, more quickly than organizations can address the threats. http://www.verizonenterprise.com/DBIR/2014/?utm_source=ContextualAds&utm_medium=ResultLinks&utm_c ampaign=DBIR2014
  • 8. Caspida – Karthik Kannan Market Data Courtesy: Mary Meeker, KPCB, Internet Trends Report, 2014
  • 9. Caspida – Karthik Kannan Current Methods Fail Limited scale, manual, no dynamicity Signatures Rules Malware-Detection
  • 10. Caspida – Karthik Kannan Why is Security a Big Data Problem?  Variety of security events − New sources, new relationships, new entities  Analysis sophistication − Dynamic correlations, sequences, non- contiguous patterns  Context – time − Months & years, not days Good reference/reading:
  • 11. Caspida – Karthik Kannan The Right Tools for the Right Purpose Protecting the perimeter and defending against known attacks (signatures) Discovery unauthorized use of SaaS/cloud apps and policy enablement for Shadow IT SIEMs collect data, use extensive human-generated rules, rely on manual analysis and provide static alerts Firewall, IPS, malware, AV Cloud security SIEM Lacking dynamic, self-learning methods that are needed to detect sophisticated attacks?
  • 12. Caspida – Karthik Kannan Mobile There is an App for Everything! SMSPhone MMS IM Mobile App Stores Mobile Device Mgmt (MDM) Mobile App Mgmt (MAM) Cloud SaaS Monitoring SaaS Encryption Web Mail CRM/ERP SaaS Apps (Salesforce, …) Custom Apps/TestDev Clouds Desktop Password Hashing Antivirus Anti-Malware SW OS security layering OS-level Sandboxing Disk Encryption Productivity Apps/Development/Test Security in the Technology Evolution Application-specific Attacks (Facebook wall, Browser) Attackers AttackTypes DDoS (Zombies etc.) Password Guessing Filesystems / DBs Misconfigurations Viruses Malware/Spyware Keyloggers Sniffing Governments Special Interest Groups Polymorphic APT Botnets Web App Attacks (XSS, etc.) Phishing Enterprise Firewalls Multi-Factor Authentication IDSAntivirus Malware Sandboxing Threat Feeds SIEMVPN Corporate Email Finance Apps Corporate Storage/Filers Collaboration Tools/ECM Cloud Apps Time2000 20131990 2010 AttackSophistication
  • 13. Caspida – Karthik Kannan Stages of an attack Research Infiltrate Capture Exfiltrate Market- place 86% of enterprises focus on step 2 only Studies show that companies save up to $4M/year when they have security intelligence systems that focus on all stages 1 2 3 4 5
  • 14. Caspida – Karthik Kannan ML + Statistical Models Visualization Models Data Lake Standard models: K- means, Random Forest, Nearest-neighbor, Gaussian, Bayesian etc. Custom models: user patterns/behavior, time- oriented, data attributes- specific, SaaS, mobile
  • 15. Caspida – Karthik Kannan Algorithms  Time Series Analysis − Good when dealing with time series − Examples:  Linear Regression  Parametric (ARIMA/FARIMA)  Forecasting: Holt-Winters  Classification Models − Good to find which categories things are falling under − Examples:  Logistic Regression  Decision Trees  Decision Tables  Neural Networks  K-Nearest Neighbors  Ensemble Models (Random Forests)  Grouping Models − Used for finding global patterns at scale − Examples:  K-Means Clustering  Random graph walks  Inference Models − Important when trying to infer value of a feature from a context − Example  Association Rules  Bayesian Networks  Simplification Models − Important when we need to decrease number of features analyzed − Examples  Principal Component Analysis (PCA)  Low-Rank Approximation  Single-Value Decomposition (SVD)
  • 16. Caspida – Karthik Kannan Data Sources: Information Value Pyramid Network Packets: L2-L4 Network Packets: L7 Generic System Logs Application Logs Lower Volume; Concentration of Information No need to decipher semantics of information Top-Down view with Correlation on important signals OS logs on system events, processes’ health Need additional deciphering of information High-Volume of Source Data Can capture malware code for analysis Problems with encrypted traffic High-Volume of Source Data Analysis only based on signatures and packet statistics
  • 17. Caspida – Karthik Kannan Advanced Persistent Threat (APT) Kill Chain A handful set of users targeted by phishing attacks The user downloads the malware which finds a back door to access the system Attacker attempts to move other systems and accounts by elevating privileges accordingly Data is gathered from different systems and staged for exfiltration Data is sent out via multiple channels (encrypted over FTP, DNS back channels etc.) Lateral Movement Phishing and Zero Day Attack Back Door Data Gathering Exfiltrate
  • 18. Caspida – Karthik Kannan Ideal Hadoop-based solution Data Sources Data Lake Data Science
  • 19. Caspida – Karthik Kannan Machine Learning in Industries  eCommerce: identify shopper behavior and predict buying patterns, inventory planning, recommendations − AggregateKnowledge − RichRelevance − Amazon  AdTech: identify mobile/online users, model their preferences, and render appropriate advertisements to the right audience − AdMob (Google) − MoPub (Twitter) − Efficient Frontier (Adobe)
  • 20. Caspida – Karthik Kannan Types of Security Analytics  Breach − Phishing attack − DDoS attack − Watering hole attack  Exploitation − Lateral movement − Domain account misuse  Exfiltration − Privileged data leakage − Anomalous login activity  Debilitation − App or DB server load/activity patterns − Web server patterns  Monitoring − Metrics management
  • 21. Caspida – Karthik Kannan Data Sources & Analysis Source Information obtained 1 Web server Incoming, outgoing traffic, IP addresses, times, session durations 2 Domain controllers User IDs accessing specific IP addresses, times, durations 3 IAM servers Apps, servers, other protected services users are accessing, times, durations 4 Content servers Detailed transactional histories, customer account data, ACLs 5 Messaging server events Email stats, attachment info, external communications (IPs, frequencies) + correlations – across time and events to produce network of related users, apps, servers and other critical services that may be affected by threats + machine learning algorithms – dynamic models driving automatic insights into malicious, external, APT, SaaS, mobile or network threats in repeatable fashion + search/queries – to sharpen insights and threat intelligence by drilling down into desired dimension such as time window, geography, criticality etc.
  • 22. Caspida – Karthik Kannan Anatomy of an attack IP Location 200.55.12.68 Brazil 58.202.85.1 China 220.12.98.41 US/SC 119.56.128.25 China … … IP Location 200.55.12.68 Brazil 58.202.85.1 China 220.12.98.41 US/SC 119.56.128.25 China … … UID2 UID1 UID3 UID5 UID4 UID6 UID7 UID9 UID1 UID8 Svr1 Svr2 App1 App2 DB1 FTP1 Identification of suspicious IP originations, destinations IP addresses, geo-spatial information collection Network of correlations for suspected IPs; which users are accessing them the most? Identification of suspicious users Correlations of suspected users with apps, databases and other sensitive services 1 2 3 456 Timeline of malicious behavior, e.g., sending emails or communicating with CnC Actions IP1 IP2 UID8 UID1 UID8 DB2
  • 23. Caspida – Karthik Kannan Network traffic Behavioral threat models Network Traffic: PCAP, Netflow • Switches • Routers • Firewalls* • IPS’* • Web gateways* • Proxy server* • Any other network device* * optional Sources • Traffic monitoring & analysis: • Which IP is communicating with which external or internal destination • Traffic volume, frequency • Correlate with IAM (for user ID – IP mapping) • Max traffic contributors – users, apps, IP addresses • Correlate with Web server (for URL traffic analysis per user) • Correlate with Messaging server (for email source/recipient analysis) • Correlate with Firewall (for external traffic analysis per IP, user) • Correlate with App, DB servers (for internal app transactional analysis) • External threat (e.g., bad IP address list) feeds Threat Intelligence
  • 24. Caspida – Karthik Kannan Examples  Ground-speed violations − detect user logins that are geographically spaced apart but fall within seconds/minutes of each other  Lateral movement − accounts moving from one server/device to another to explore and list content on each location before deciding which to exfiltrate  Domain admin creations − auto creation of admin accounts by spurious account; e.g., r00t, adm1n etc.
  • 25. Caspida – Karthik Kannan Where do you start?  Need a data lake  Analytics: − ML − Statistical  Actions
  • 26. Caspida – Karthik Kannan Thank you!