Streaming Cyber Security into Graph: Accelerating Data into DataStax Graph and Blazegraph

Spotting the trends by looking
at the big picture
Streaming Cyber Security into Graph:
Accelerating Data into Datastax Graph
and Blazegraph

Silicon Valley
• Digital Experiences
• Artificial Intelligence
• Platforms & Systems
Washington DC
• Security
Dublin
• Artificial Intelligence
Sophia Antipolis
• Industry Innovation (FS & Resources)
Beijing
• Industrial Internet
Bangalore
• Software Engineering
Tel-Aviv
• Security
For more than 20 years, Accenture Labs has served as the tip of the spear for technology innovation at Accenture.
Over the last 5 years Accenture Labs has:
• Supported 300+ client engagements and hosted 1100+ client workshops
• Published 200+ thought leadership pieces, filed 110+ patent applications, and garnered 350+ Tier-1 media hits
Expanding Global Presence
2Copyright © 2016 Accenture All rights reserved.

Security Data Science is Hard
Once the security community moves beyond the mantras “encrypt
everything” and “secure the perimeter,” it can begin developing intelligent
prioritization and response plans to various kinds of breaches – with a strong
focus on integrity.
http://www.wired.com/2015/12/the-cia-secret-to-cybersecurity-that-no-one-seems-to-get/
Right now, financial services reports it takes an average of 98 days to detect an
Advance Threat but retailers say it can be about seven months.

Security Data Science is Hard
The challenge lies in efficiently scaling these technologies for practical
deployment, and making them reliable for large networks. This is where the
security community should focus its efforts.
http://www.wired.com/2015/12/the-cia-secret-to-cybersecurity-that-no-one-seems-to-get/
Right now, financial services reports it takes an average of 98 days to detect an
Advance Threat but retailers say it can be about seven months.

Research Hypotheses - Architecting the Next Generation Cyber Hunting
Cyber security is a big data problem, the volume and velocity of data from devices requires a
new approach that combines all data sources to allow for more in intelligent/advanced cyber
security hunting through analytics and exploration at scale across enterprise data.
Visualization will be a key part of cyber hunting because our human eyes and brains are
really good at detecting changes — what’s wrong or different — enabling us to follow the
threat.
Indication of compromise needs to evolve as attacks are becoming more sophisticated,
subtle, and hidden in the massive volume and velocity of data. Combining machine learning,
graph analysis, applied statistics, and deep learning is essential to reduce false positives,
detect threats faster, and empower cyber analyst to be more efficient.
Proprietary and Confidential Property of Accenture

EnableIncubateDiscover
Intellectual
asset
licensing
Joint Ventures
Products in-
sourced for scale
up
Intellectual assets
insourced for
development
Insourced
ideas &
technologies
Out to
Market
Scale
ASGARD
ASGARD Rethinking Cyber Security Analytics Hunting
Streaming
Storage
Analytics
Visualization
Interaction

Innovation Cycle
Architecture
Data
Visualization
Analytics
DATA SCIENCE ARCHITECTURE
Customize,
create, and iterate

STINGER
Project ASGARD – Advanced Security Graph Analytics for Real-time Defense
Building a data driven platform to advance cyber defense beyond any one traditional technology

Accenture Labs ASGARD V1 Platform
Ingest
Event
Processing
Storage
Notebooks
Query Layer
Data
Sources
Visualizations
SQL
Streaming
py

Big Data Cyber Defense is Hard… Really Hard
Cost
Efficiencies
Lack of Agile
Model
Development
Threats
disguised as
legitimate
Interconnected
Data Problem
Expanding
Attack
Surfaces
Out of Order
Events
Ongoing
Privacy
Concerns
Multi-Model
Approach

Cost
Efficiencies
Lack of Agile
Model
Development
Threats
disguised as
legitimate
Interconnected
Data Problem
Expanding
Attack
Surfaces
Out of Order
Events
Ongoing
Privacy
Concerns
Multi-Model
Approach

Cost
Efficiencies
Lack of Agile
Model
Development
Multi-Model
Approach
Threats
disguised as
legitimate
Interconnected
Data Problem
Expanding
Attack
Surfaces
Out of Order
Events
Ongoing
Privacy
Concerns

vs
Big Data Solution
• 10 node cluster - ~$60k in hardware
• Spark 1.6.0
• Query was done when data was available as a Pandas Dataframe
Production SIEM of Fortune 500 Enterprise
Data
• 450+ columns
• ~250 million events per day
SIEM
Benchmark
Cost
Efficiencies

Typical Scenario Time Period SIEM Big Data Speed Up
1 Show all network communication from one host
(IP) to multiple hosts (IPs)
1 Day 3h 20m 13s 1m 44s 114 Times Faster
1 Week Not Feasible* 4m 05s
2 Retrieve failed logon attempts in Active Directory 1 Day 18m 26s 1m 37s 10 Times Faster
1 Week 2h 13m 45s 3m 10s 41 Times Faster
3 Search for Malware (exe) in Symantec logs 1 Day 3h 24m 36s 1m 37s 125 Times Faster
1 Week Not Feasible* 3m 22s
4 View all proxy logs for a for specific domain 1 Day 4h 30m 13s 2m 54s 92 Times Faster
1 Week Not Feasible* 1m 09s**
Notes:
* Client team was unable to run the benchmarks without splitting the query by time units, and allocating more
resources to run it; they estimate it would take 20+ hours to complete
** Due to over 1.6 million results, the number of fields returned was reduced from 466 to 10 key fields resulting in 5x
speed-up over returning all fields; however, the other columns are still searchable and available within this time
Benchmark

Multi-Model Approach
Multi-Model
Approach

Multi-Model
Approach
No Silver Bullet!!!

Cyber Security is a Connected Data Problem
User:
Bob
User:
Jane
IP:
10.0.0.1
IP:
10.1.0.1
IP:
10.0.0.2
Assigned_IPHostname:
Comp_1
Hostname:
Comp_2
Auth_Success
Communicates_With
Auth_Success
Associated_With
User:
John
Hostname:
Comp_3
Assigned_IP
Malware
Sig:
Package_1
Detected_On
User:
Fred
Auth_Failure
Interconnected
Data Problem

Why Graph Analysis
Graphs represent Cyber Security Data Well
Traversals Faster Than SQL Joins (Efficiency)
More effective at detecting certain types of threat than
other analytical methods for example:
• Fast-flux Botnet Detection
• Lateral Movement within Networks
• Low and slow port scans
• Attack Surface Management Risk
Infection
Signatures
IP Address Users
IP Address

Graph Analysis is Computationally Expensive
On extremely large graphs, Graph Analytics computations can be costly
• Cyber Security needs near real-time graph analytics measures
• Many Graph analytics computations scale nicely to GPUs
Micro-batching data to GPUs gives us regular updates of graph features
CPU Graph Analytics tools GPU Analytics Accelerators
cuSTINGER nvGraph
GraphX
Out of Order Data made prior graph analysis not as accurate
(Parquet Limitations) Out of Order
Events

Why GPU Accelerated Graph Analytics
Use CPU cluster resources for random ad-hoc analysis, streaming, etc…
Repetitive Tasks should be more optimized (Cyber Security is very Repetitive)
GPU scale better with a smaller footprint; more green
Future R&D Goals more Ensemble analytical methods on GPU
• Time series => graph analysis => graph feature time series

Ingest
Event
Processing
Storage
Notebooks
Query Layer
Data
Sources
Visualizations
SQL
Streaming
py
Accenture Labs ASGARD Platform v2

Ingest
Event
Processing
Storage
Notebooks
Query Layer
Data
Sources
Visualizations
SQL
Streaming
py
GPU Layer
DASL

DataStax Enterprise Graph
TitanDB
Graph
+ =

DSE Graph with Spark and Gremlin
• OLAP
• See what every user did
• OLTP
• See what a specific user(s) did
Graph

• OLAP
• OLTP
Graph

DSE Graph Deployment
Configuration
• 32GB of memory per node
• 4 cores per node
45 Nodes

Spark Streaming v. Flink Deployment
Configuration
• 4GB of memory per executor
• 1 core per executor
18 Executors
1 Master
+

Spark Streaming v. Flink Schema
USER
Up to 3 Reads
for Existing
Vertices

USER
g.V().hasLabel(“label”).has(“key”, “value").tryNext()
.orElseGet{g.addV(“label”).property(“key”, “value”).next()}
1 Read for
Existing Vertex
Up to 3 Reads
for Existing
Vertices

IP
USER MSG
g.V().hasLabel(“label”).has(“key”, “value").tryNext()
.orElseGet{g.addV(“label”).property(“key”, “value”).next()}
Up to 3 Writes
to Add New
Vertexes
2 Writes to Add
New Vertexes

6 New Edges
Written for
Each Event
IP
USER MSG

Spark Streaming => DSE Graph
kafkaDirectStream
map
data transformations
foreachPartition
Initialize DSE
Initialize Semaphore
Check Semaphore
Close DSE
foreach
async graph queries
Initialize Accumulators

Flink => DSE Graph
KafkaConsumer
map
data transformations
DataSink
open
Initialize DSE
Initialize Semaphore
Initialize
Accumulators
close
Check Semaphore
Close DSE
invoke
async graph queries

Post-QueryQueryPre-Query
Asynchronous Graph Queries
Structured
Data
Create Graph
Query
Acquire
Semaphore
Permit
Execute Query
Asynchronously
Success
Callback
Failure
Callback
Release
Semaphore
Permit
Increment
Success
Accumulator
Increment
Failure
Accumulator

Spark Streaming v. Flink Benchmark
1 25 50 100 200 300 1000
Spark v1.6.2 79.3 11.8 9.5 9.6 10.5 13.6 10.3
Flink v1.1.1 77.3 10.0 9.4 8.8 10.1 10.0 10.4
0
2
4
6
8
10
12
14
16
18
20
RunTime(min)
DSE Graph Insertion with Semaphores
~9.7
million
events

Spark Streaming v. Flink Benchmark
1 25 50 100 200 300 1000
Spark v1.6.2 79.3 11.8 9.5 9.6 10.5 13.6 10.3
Flink v1.1.1 77.3 10.0 9.4 8.8 10.1 10.0 10.4
0
2
4
6
8
10
12
14
16
18
20
RunTime(min)
DSE Graph Insertion with Semaphores FAILURESSLOW
~9.7
million
events

DSE Graph Query Execution
Asynchronous Synchronous
*

DSE Graph Troubleshooting
Use DataStax studio and profiling to build your insertions / lookups
• DataStax studio profiling is much nicer to work with than the gremlin shell
• Ensure you’re hitting indexes wherever possible
• Scanning thousands of vertices / edges kills performance

DSE Graph Troubleshooting
Monitor CPU and Disk I/O
• OpsCenter gives a great view on resource usage
• Our workload was CPU heavy, and at too high of a CPU load we saw reduced throughput
and/or failed queries

Streaming DSE Graph Limitations and Lessons Learned
DSE Graph does not yet support prepared statements or batches
Spark & Flink Accumulators are non-atomic within executors
Flink Kafka Consumer pushes offsets to Zookeeper
• To read from the beginning of the stream set "auto.offset.reset” to ”smallest”, and set
"group.id” to a random value
Spark Kafka Consumer doesn’t push offsets to Zookeeper
• Treat offset data as time series and store data in Cassandra

• OLAP
• OLTP
Graph

cuSTINGER nvGraph
GraphX

Blazegraph
+ =

Blazegraph DASL
+ =
DASL

Out of Order Data - The Problem
Date Events
Prior to 8/6/16 9,411
8/6/16 1
8/7/16 56
8/8/16 4,484
8/9/16 83,465
8/10/16 498,093
8/11/16 163,179,386
8/12/16 35,569,614
Total Events 199,344,510

Date Events
Prior to 8/6/16 9,411
8/6/16 1
8/7/16 56
8/8/16 4,484
8/9/16 83,465
8/10/16 498,093
8/11/16 163,179,386
8/12/16 35,569,614
Received On

Date Events
Prior to 8/6/16 9,411
8/6/16 1
8/7/16 56
8/8/16 4,484
8/9/16 83,465
8/10/16 498,093
8/11/16 163,179,386
8/12/16 35,569,614
Received On
Build a Daily
Parquet
One Day of Data
Spread Across
Several Days
Need to Scan
Week of Data
for One Day

Out of Order Data – The Solution
Apache Kudu
• Columnar storage manager developed by Cloudera for the Hadoop ecosystem
• Supports
• Fast inserts/updates
• Efficient columnar scans
Benefits
• Allows us to insert data in real time
• Allows us to scan a true day of data more efficiently than Parquet
Development Time
• Started with Kudu 0.10 now at 1.0.0
• 2 weeks to get up and running

Where Does Kudu Fit?
ScanSpeed
Random Access Speed

Kudu - Deployment and Data Size
18 Tablet Servers
1 Master
+
Configuration
• 16GB of memory
• 8GB of block caching*
• Co-located with Spark/HDFS
Data
• 7 days of events
• 1.11 Billion rows
• 70 columns
• ~155GB raw

Kudu - Schema
Encoding
• BIT_SHUFFLE encoding on all INTEGER, DOUBLE, FLOAT columns
• DICT_ENCODING encoding on all STRING columns*
Compression
• DEFAULT_COMPRESSION set to NO_COMPRESSION for testing
• Successfully done experiments with SNAPPY_COMPRESSION reducing storage costs
Partitioning
• Hash bucketing on 5 PRIMARY KEYS
• YEAR, MONTH, DAY combination to reduce tablets scanned
• HOUR further distribute a day of data and allow for search
• EVENT_ID unique identifier to distribute data evenly
• 144 tablets per day; plan to test more partitioning in the future

Kudu - Upsert Performance with 1440 Tablets
0
100000
200000
300000
400000
500000
600000
700000
800000
900000
UpsertsPerSecond
Upsert/second
Expon. (Upsert/second)
Sampled every
10 seconds
with Impala
1400 Spark Tasks
160 Executors
~90 mins

Query Scenarios Size
1 Filter on year, month, day for one
source IP to three destination IPs
( s = IP AND ( d = IP OR
d = IP OR d = IP ))
330
2 Filter on year, month, day for failed
logins of a particular user
( et = FL AND user = Bob )
180
3 Filter on year, month, day for specific
firewall server events
( dv = FW AND dcs = Server )
150
4 Filter on year, month, day for all
traffic to a specific domain
( hst = Host )
3k
Kudu v. Parquet Performance in Spark
61.8
55.3 55.2 55.0
7.9
5.0 2.9 2.4
0
10
20
30
40
50
60
70
Scenario 1 Scenario 2 Scenario 3 Scenario 4
Avg.QueryTime(s)
Parquet Kudu

Kudu - Impact of Operators within Spark
Supported Predicate Pushdown
• =, <=, >=, BETWEEN
• Scans tablets within Kudu = FAST results
Unsupported Predicate Pushdown
• NULL, NOT NULL, <>, OR, LIKE, IN
• Optimizer tries to use any supported predicates to limit table scans
• Then pushes all data to Spark Task for comparison
• Data movement slows down query response time
AS OF v1.0.0
Source: https://github.com/cloudera/kudu/blob/master/docs/developing.adoc

1 Filter on year, month, day for subnet
( s LIKE IP% AND ( d = IP OR
d = IP OR d = IP ))
18k
logins of admin users
( et = FL AND user LIKE Ad% )
500
3 Filter on year, month, day for any
( dv = FW AND
dcs LIKE %Server% )
260k
traffic to domain & subdomains
( hst LIKE %Host )
6k
Testing Limitations of Kudu in Spark
61.8 55.3 55.2 55.0
7.9 5.0 2.9 2.4
224.1
132.0
174.8
194.5
0
50
100
150
200
250
Avg.QueryTime(s)
Parquet Kudu = Kudu LIKE

Testing Limitations of Kudu in Spark
61.8 55.3 55.2 55.0
7.9 5.0 2.9 2.4
224.1
132.0
174.8
194.5
0
50
100
150
200
250
Avg.QueryTime(s)
Parquet Kudu = Kudu LIKE
2.4x - 3.6x longer than Parquet for Kudu with LIKE
1 Filter on year, month, day for subnet
( s LIKE IP% AND ( d = IP OR
d = IP OR d = IP ))
18k
logins of admin users
( et = FL AND user LIKE Ad% )
500
3 Filter on year, month, day for any
( dv = FW AND
dcs LIKE %Server% )
260k
traffic to domain & subdomains
( hst LIKE %Host )
6k

Kudu Brings Big Data to GPU Analytics
cuSTINGER
&
Kudu provides
Spark Data Frames
for Blazegraph DASL
Kudu has low-level
C++ APIs
Support micro-
batching use case
for updates to GPUs

Ingest
Event
Processing
Storage
Notebooks
Query Layer
Data
Sources
Visualizations
SQL
Streaming
py
GPU Layer
cuSTINGER

More Lessons Learned
Initial thought Alluxio serves as caching layer
• Spark support was lacking
• Little performance improvement vs. parquet on HDFS
• Kudu better choice for our needs
KUDU-1651
• All NULL values in dictionary encoded blocks for Kudu 1.0.0 throws error
• Fixed in master branch!
Cannot change partitioning
• Planned to be supported in future releases

Future Research Work
Tackling
• Streaming Analytics (out-of-order)
• Complex Event Processing
• Managing Graph Insertions and Deletions in real time
GPU analytics
• Unified memory and page faulting on the new NVIDIA Pascal
GPUs solves a previous problem with maximum graph size
• NVIDIA DGX-1
Deep Learning

Thanks!
Cloudera
• Dan Burkert
• Todd Lipcon
Datastax
• Rob Murphy
• Jonathan Shook

Questions? @datametrician
Josh Patterson
Principal Data Scientist
Keith Kraus
Associate Principal Engineer
Mike Wendt
Principal Engineer
@mike_wendt
@keithjkraus

Streaming Cyber Security into Graph: Accelerating Data into DataStax Graph and Blazegraph

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Streaming Cyber Security into Graph: Accelerating Data into DataStax Graph and Blazegraph

Similaire à Streaming Cyber Security into Graph: Accelerating Data into DataStax Graph and Blazegraph (20)

Dernier

Dernier (20)

Streaming Cyber Security into Graph: Accelerating Data into DataStax Graph and Blazegraph