Data-Driven Security Analytics Using a Data Lake

1© Copyright 2014 EMC Corporation. All rights reserved.© Copyright 2014 EMC Corporation. All rights reserved.
Principal Data Scientist, Pivotal
Derek Lin

Agenda
• Information Security Analytics & Use Cases
• Data Lake
• Data Science: Extracting Values from Data Lake
• Demo

Information Security Analytics
Landscape
Application Areas

CIO Survey: Top Concerns
Sources: Barclays September 2013 CIO Survey, KPMG January 2014 CIO/CFO Survey
54% What to Collect
85% How to Analyze

A Big Data Analytics Response
• More sophisticated adversaries and sophisticated methods.
• Limited human capacity combined with massive amounts of events
– 40% of all survey respondents are overwhelmed with the security data
they already collect
– 35% have insufficient time or expertise to analyze what they collect
• Security tools, tactics and defenses becoming outdated:
– Content is static and not as dynamic as the threat landscape
– Segregated by too many point products, tool interfaces, disparate data
sets
1 EMA, The Rise of Data-Driven Security, Crawford, Aug 2012
Survey Sample Size = 200

Enterprise Information Security
Analytics
Insider
Threat
Asset Risk
Malware
Threat

APT Kill Chain
Advanced Persistent Threat (APT)
A handful of users
are targeted by two
phishing attacks:
one user opens
Zero day payload
(CVE-02011-0609)
The user machine
is accessed
remotely by
Poison Ivy tool
Attacker elevates
access to important
user, service and
admin accounts,
and specific
systems
Data is acquired from
target servers and
staged for exfiltration
Data is exfiltrated via
encrypted files over
ftp to external,
compromised machine
at a hosting provider
Phishing and
Zero Day Attack
Back Door
Lateral
Movement
Data
Gathering
Exfiltrate
1 2 3 4 5

Anatomy of an Attack | Anatomy of a Response

Technology Landscape in the Kill Chain
Perimeter
penetration
Malware
beacon
Lateral
movement
Staging &
exfiltration
Single source
real-time
Proactive, limited-sources
rule-based methods
over short-range
Proactive, multi-sources
data-driven methods
over long-range
Reactive, manual
post-incident response
Host/Network Analysis & Search/Indexing
Data-lake enabled analytics
IDS/IPS
Anti-virus
SIEM
DLP

Analytics Opportunities in Threat Defense
• Malware discovery
– Host infection detection
– Malware Command & Control
beaconing activity detection
• Perimeter penetration
prevention/detection
– Anomalous VPN login detection
– Denial of service attack
mitigation
– Local IP black list construction
– Watering hole attack detection
– Chat room monitoring
– Phishing attack detection
– Web server attack detection
• In-network anomaly detection
– Anomalous resource access
detection
– Critical server activity
monitoring
• IR efficiency improvement
– Semi-automated analysis
• SIEM efficiency improvement
– Threat feed normalization
– Alert prioritization
Malware
Threat

Information Security Analytics Areas
Insider
Threat
Asset Risk
Malware
Threat

Malicious Insider Threat
http://www.logrhythm.com/Portals/0/resources/LogRhythm_Survey_15.42014.pdf
What do you think is the biggest threat to your
organization’s confidential data?
Does your organization have any systems in place to
stop employees accessing confidential information or
taking data?

Analytics Opportunities in Identity, Access,
and Management
• Anomalous user to resource access detection
– User-resource access data
• Role and activity auditing
+ Role and provisioning data
• Privilege escalation auditing
+ Privilege escalation data
• IT support personnel auditing
- Support ticket data
- Command activity data Insider
Threat

Information Security Analytics Areas
Insider
Threat
Asset Risk
Malware
Threat

Information Asset Management

Analytics Opportunities in Asset Management
• Document categorization for risk labeling
– Unstructured data
– User access data
• Asset risk profiling
– Vulnerability scanner data
– User access data
Asset
Risk

Data Lake
Needs and Trend
Analytics Support

Life Before Data Lake
Departmental
Warehouse
Enterprise
Apps
Reporting
Non-Agile Models
Spread
marts
Prioritized
Operational Processes
Errant data and marts
Departmental
Warehouse
Siloed
Analytics
Data
Sources
Non-Prioritized
Data Provisioning
Static schemas
grow over time

Impact of Status Quo
• High-value data is hard to reach and leverage
• Data Scientists are last in line for data
– Queued after prioritized operational processes
• Data is moving in batches from Warehouse(s) to
Data Scientists’ desktops
– In-memory analytical work (w/ R, SAS, SPSS, Excel)
– Sampled, driving model accuracies down
• There is a “cottage industry” of analytics, rather
than centrally-managed harnessing of analytics
– Non-standardized initiatives
– Frequently, not-aligned with corporate business goals
Slow
“time-to-
insight”
&
reduced
business impact

Data-Driven Digital Media Analytics
Targeting &
Retention
Social Media
Analysis
Campaign
optimization
0
Transaction
History
Purchases
Clickstream
Customer
Data
Unified data supporting re-usable predictive models
GB
TB
PB
Data
Size

Data-Driven Financial Protection Analytics
TB
Data
Size
Web
Optimization
Fraud
Detection
Product
Recommendation
ATM
Member Data
Transactional Log
Firewall
Clickstream
Phone Channel
GB

Data-Driven IT Operation Analytics
GB
TB/
PB
Data
Size
Failure
Prediction
Root Cause
Analysis
Project Risk
Forecasting
Server or VM
Performance Metrics
CMDB
Configuration
Setting
Alerts & Incident
Server logs
Network
Performance Metrics

Data-Driven Security Analytics
TB/
PB
Data
Size
Insider Threat
Detection
Malware
Detection
DDoS Mitigation
AD/Auth
Asset/Role
Netflow
DNS/Firewall/Proxy
Critical Server
Packet Capture
GB
Defense with breadth in variety and depth in time

Pivotal Business Data Lake Architecture
Centralized Management
System monitoring System management
Unified Data Management Tier
Data mgmt.
services
MDM
RDM
Audit and
policy mgmt.
Processing Tier
Workflow Management
In-memory
MPP database
Existing Sources
Unified Sources Flexible Actions
Real-time
ingestion
Micro batch
ingestion
Batch
ingestion
Real-time
insights
Interactive
insights
Batch
insights
HDFS
New Data Sources

Pivotal Business Data Lake Architecture
Centralized Management
Unified Data Management Tier
Data Dispatch
MDM
RDM
Data Dispatch
Processing Tier
Spring XD
Pivotal GemFire XD
HAWQ
Unified Sources Flexible Actions
Clickstream
Sensor Data
Weblogs
Network Data
CRM Data
ERP Data
Pivotal
GemFire
Pivotal
RabbitMQ
Redis
Pivotal CFPivotal HD
Command Center
Existing SourcesNew Data Sources

Data Lake: More Than a Data Repository
To iterate and
experiment to
fail fast for fast
cycle of value
generation
Analytics
Support
Fast
Query
Data
Store

Data Science Tools
Commercial Open Source (or Free)
PL/R, PL/Python PL/Java

MADlib In-Database Functions
Predictive Modeling Library
Linear Systems
• Sparse and Dense Solvers
Matrix Factorization
• Single Value Decomposition
(SVD)
• Low-Rank
Generalized Linear Models
• Linear Regression
• Logistic Regression
• Multinomial Logistic Regression
• Cox Proportional Hazards
• Regression
• Elastic Net Regularization
• Sandwich Estimators (Huber
white, clustered, marginal
effects)
Machine Learning Algorithms
• Principal Component Analysis (PCA)
• Association Rules (Affinity Analysis,
Market Basket)
• Topic Modeling (Parallel LDA)
• Decision Trees
• Ensemble Learners (Random Forests)
• Support Vector Machines
• Conditional Random Field (CRF)
• Clustering (K-means)
• Cross Validation
Descriptive Statistics
Sketch-based Estimators
• CountMin (Cormode-
Muthukrishnan)
• FM (Flajolet-Martin)
• MFV (Most Frequent
Values)
Correlation
Summary
Support Modules
Array Operations
Sparse Vectors
Random Sampling
Probability Functions

Data Science: Extracting Value from
Data Lake
Technology & Tools
People

Evolution of Data Analytics in Security
BI and
Compliance-
driven
Investigation-
driven
Behavior-
metrics driven
Data-science
driven
Data goes in,
hard to extract
value
Fast queries
over large data
Single source
metrics, simple
correlation, rule-
based, high false
positive
Leverage full
contextual info,
multi-source,
automatic, for low
false positives

What is Data Science?
The use of statistical and machine learning techniques
on big multi-structured data in a distributed computing
environment
to identify correlations and causal relationships
classify and predict events
identify patterns and anomalies
and infer probabilities, interest and sentiment.

Data Science: The Next Security Frontier
• Beyond signatures
• Beyond simple metrics for
thresholding
• Beyond manual engineering
of rules
• Monitor each and every
entity in its environmental
context with 360° view
over long time window with
advanced mathematics

Pivotal Network Intelligence
Demo

What is a
“Data
Scientist”?
ProgrammingSkills
Mathematical/Statistical Skills

One Team
Member
ProgrammingSkills

Another
Team
Member
ProgrammingSkills

Yet
Another
ProgrammingSkills

Together
ProgrammingSkills

Take-Home Messages
Information
Security = Big
Data Problem
Data Lake is
more than a
data store
Data Science drives
value from Information
Security Data Lake

Data-Driven Security Analytics Using a Data Lake

Data-Driven Security Analytics Using a Data Lake

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Data-Driven Security Analytics Using a Data Lake

Similar to Data-Driven Security Analytics Using a Data Lake (20)

More from EMC

More from EMC (20)

Recently uploaded

Recently uploaded (20)

Data-Driven Security Analytics Using a Data Lake