Apache Metron: Community Driven Cyber Security

Apache Metron:
Community Driven Cyber Security
Ned Shawa & Laurence Da Luz
Hadoop Summit Melbourne - 2016

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
 Apache Metron Introduction
 User Personas & Key Functional Themes
 Capabilities and Architecture
 Building a Use Case in Metron
 Questions

Apache Metron Introduction

Apache Metron Vision
“Apache Metron is a Security Data
Analytics Platform (SDAP). As a next
generation security analytics
framework, it is designed to
consume and monitor network traffic
and machine data within an
enterprise environment. Apache
Metron is extensible and is designed
to work at a massive scale. It is not a
SIEM but rather the next evolution of
a SIEM.”

Cyber Security – Today’s Enterprise Threat
 Organizations have recently become targets of complex
cyber security breeches that could have been easily
prevented
 Cyber attacks continuously become more advanced and
go un-detected using traditional IT security policies and
procedures
 Cyber Security attacks have increased in visibility and
targeted organizations with millions of customers – costing
millions in privacy damages
 Recent cyber security attacks have been led by highly
skilled technical organizations where the attack could have
been prevented by known solutions
62 % - Increase in Cyber Security
Breaches since 2013
8 months – Average time an
advanced security breach goes
unnoticed
3 Trillion – Total cost of Cyber
Security breaches
1 in 3 – Security professionals are
not familiar with cyber security
threats
2014 ISACA

Apache Metron – Community Driven Cyber Security
Security
Data Lake
Enriched 360
Correlated
Searchable
Discoverable
Threat
Intelligence
3rd Party Feeds
Static Rules
ML Models
IOC Sharing
Pluggable
Framework
Parsers
Enrichers
Threat Intel
UI Widgets
Security
Application
PCAP Replay
Evidence Store
Hunting Platform
Apache
Metron
Hortonworks and the Apache Metron Community are
focused on delivering the next generation cyber security
platform to enable organizations to enrich and analyze all
data within their enterprise

Apache Metron – How We Got Here

Apache Metron – Who’s Involved

Telemetry
Parsers
TELEMETRYINGESTBUFFER
Enrichment
Indexers &
Writers
Telemetry
Parsers
Real-Time Processing
Security Engine
Threat Intel Alert Triage
DATASERVICES&INTEGRATIONLAYER
Apache Metron – Capabilities Overview
Real-Time Security Stream Processing Pipeline
Performant
Network
Ingest
Probes
Real-Time
Enrich/
Threat Intel
Streams
Telemetry
Data Collectors
/ Other..

User Personas & Functional Themes

Metron User Personas

Metron’s Key Functional Themes
Platform
Work done to harden the platform for performance, scale, extensibility and
maintainability. This also includes capabilities around provisioning, managing
and monitoring the application.
Set of Data Sources that Metron provides capabilities to stream, ingest
and parse into the platform.
A set of Storm Topologies to perform various actions in real-time including:
normalization of telemetry data, enrichment, cross reference with threat intel
feeds, alerting, indexing, and persisting into Historical stores
Data Collection
Data Processing
Data/Integration Services
Portals/UI Set of portal, dashboard and user interfaces for the different personas.

Data CollectionSource Systems Message Queue
Stream Process and
Enrichment
Data Access
Network Traffic
SSH
System Log
HTTP(S)
File System
email Flume
PCAP
NiFi
Flume
Kafka
NiFi processor
NiFi processor
NiFi processor
NiFi processor
NiFi processor
NiFi processor
PCAP Topic
Email Topic
SSH Topic
SysLog Topic
HTTP Topic
DPI Topic
Flume
Storm & Spark
PCAP Topology
Email Topology
SSH Topology
SysLog Topology
HTTP Topology
DPI Topology
Hive
Solr
HBase
Raw Data
(Historical)
Data Index
PCAP Data
Ability to ingest and process over 1.2 million events per second
Apache Metron Logical Architecture

Capabilities and Architecture
Apache Metron 0.2

Metron 0.2 Streaming and Enrichment

Metron 0.2 Data Ingestion Architecture

Key Points:
• Each New Telemetry Data Source will have its own Parser Topology
• Two types of Parsers available in TP2: Grok and Java
Metron 0.2 Parsing / Normalization Topology

Metron 0.2 Parser Types
 Metron parser:
– Input: Read native format data from Kafka topic
– Output: Normalized Metron JSON Object
 Grok Parser
– Suitable for structured or semi-structured logs
– Regex-like syntax with pre-defined mappings (less effort to implement)
– Good for lower volumes of data
 Java Parser
– Requires custom code (more effort to implement)
– Good for higher volumes of data

Metron 0.2 Enrichment Topology

Metron 0.2 Enrichment Topology
Enrich Add additional information
to raw source during streaming
In-built Geo enrichment (IP to
coordinates + City/State/Country)
Streaming Allows ML models to
score in real-time instead of batch
Threat Intel Flag alerts against
intel feed & determine triage

Stellar Framework
 What is it?
– Powerful framework that provides a custom DSL that is used across different Metron components for querying,
transformation and configuring rules.
 Why do we Need it?
– For a variety of components we have the need to determine if a condition is true and if so perform some action.
– For those purposes, this framework provides the DSL to create those conditions and execute a set of action.
 How is Stellar Used within Metron today?
1. Filtering, transformations and validations in parser topologies
2. Threat Triage - allocating scores to certain rules based on conditions
3. PCAP CLI – Query for pcap data

What does Stellar consist of?
 Referencing Fields in the enriched JSON
 Simple boolean operations: and, not, or
 Simple comparison operations <, >, <=, >=
 Determining whether a field exists (via exists)
 The ability to have parenthesis to make order of
operations explicit
E.g.:
IN_SUBNET( ip, '192.168.0.0/24') or ip in [ '10.0.0.1’,'10.0.0.2' ] or exists(is_local)
 A fixed set of functions which take strings and
return boolean inlcuding:
– IN_SUBNET, IS_EMPTY, STARTS_WITH, ENDS_WITH,
REGEXP_MATCH, IS_IP, IS_DOMAIN, IS_EMAIL, IS_URL,
IS_DATE, IS_INTEGER
 A fixed set of transformation functions
including:
– TO_LOWER, TO_UPPER, TO_INTEGER, TO_DOUBLE, TRIM,
JOIN, SPLIT, GET_FIRST, GET_LAST, GET, MAP_GET,
DOMAIN_TO_TLD, DOMAIN_REMOVE_TLD, URL_TO_HOST,
URL_TO_PROTOCOL, URL_TO_PORT, URL_TO_PATH,
TO_EPOCH_TIMESTAMP
E.g.:
IN_SUBNET( ip, '192.168.0.0/24') or ip in [ '10.0.0.1', '10.0.0.2' ] or exists(is_local)
Query Language Transformation Language

Metron 0.2 Metron JSON Object
 Numerous sensors log in different formats. The parser should normalize at least the following
subset of fields to the following Metron JSON naming conventions:

Metron 0.2 Metron UI with Kibana 4

Building a Use Case in Metron
Squid Logs (Metron Reference App)

Metron Reference Application Squid Sensor
 What is the Reference App?
– A use case that showcases the following:
1. How to add telemetry events from a new data source (Squid) which covers parsing, filtering, transforms and validates
2. How to see the new Events in the Metron UI
3. How to enrich the telemetry events
4. How to do threat intel cross reference checks against event
5. How to raise alerts
6. How to persist (index, long term storage) the events
 Why do we need it?
– Similar to the famous java pet store app, it provides an app that is constantly updated to showcase new features.
 What are the updates to the Metron Reference App with TP2?
– Using Stellar framework to filter, transform and validate events
– How to work with the New Metron UI to display new events
– Using Stellar framework to do threat triage
– Streaming enrichments
 How do you consume it?
https://cwiki.apache.org/confluence/display/METRON/Metron+Reference+Application

Use Case Setup
• Scenario
• Customer Foo has installed Metron TP2 and they are using the out of the box data sources (PCAP, YAF/Netflow,
Snort and Bro). They love Metron!
• But now they want to add new data source the the platform: squid proxy logs.
• Customer Foo’s requirements are the following
1. Need to ingest the proxy events from Squid logs in real-time
2. The proxy logs have to be parsed into a standardized JSON structure that Metron can understand
3. In real-time, the squid proxy event needs to be enriched with domain/whois information (domain, cert,
country, company)
4. In real-time, the domain of the proxy event must be checked against for threat intel feeds
5. If there is a threat intel hit, an alert needs to be raised
6. The system should provide the ability to configure rules via a custom DSL to prioritize/score different types of
alerts
7. The end user must be able to see the new telemetry events and the alerts from the new data source

Metron 0.2 Squid Use Case

Metron 0.2 Squid Use Case
Step 1b NiFi TailFile
Step 1a Create Topic Step 2 Define Parser
Step 3 Enrichment Config
Step 4 Configure Alerts
Step 5 Create Dashboard
Configuration Driven

• What is Squid?
• Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times
by caching and reusing frequently-requested web pages
• What does a Squid Access Log look like?
• When you make an outbound http connection to https://www.cnn.com, the following entry is added to a file called access.log:
1461576382.642 161 98.220.218.158 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html
Unix Epoch Time
IP of host where
connection was made
Domain name of the
outbound connection
Squid & its Telemetry Event

• What is Squid?
• Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times
by caching and reusing frequently-requested web pages
• What does a Squid Access Log look like?
• When you make an outbound http connection to https://www.cnn.com, the following entry is added to a file called access.log:
Unix Epoch Time
IP of host where
connection was made
Domain name of the
outbound connection
Convert from Unix
Epoch to Timestamp
Asset enrichment to enrich IP
(hostname, type of device)
WHOIS enrichment to look
up domain name information
Threat Intel to cross-reference IP
with intel feed to see if there is a hit
Index the event into Elastic and
persist in HDFS (Security Data Vault)
What Metron does to the
Squid telemetry in real-time
Squid & its Telemetry Event

Step 1 Telemetry Ingest
Step 1a Create Topic in Kafka Step 1b NiFi TailFile
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh
--zookeeper $ZOOKEEPER_HOST:2181
--create --topic squid
--partitions 1 --replication-factor 1
Ingest Squid logs into
squid Kafka topic via NiFi

Step 2 Configuring the Squid Parser
Defining the Grok Filter for the Squid data
• Grok vs Java  no custom code
• Suitable for structured or semi-structured logs
• Pre-defined mappings
• Regex-based
TIMESTAMP_ISO8601
NUMBER
WORD
HOSTNAME
IP
USERNAME
SQUID_DELIMITED %{NUMBER:timestamp}.*%{INT:elapsed} %{IP:ip_src_address}
%{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method}
%{NOTSPACE:url}.*%{IP:ip_dst_addr}
Squid Grok Filter:
Pre-defined Grok mappings for
IP address and url are used

Step 2 Configuring the Squid Parser
Squid Parser and Transform Configuration
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "squid",
"parserConfig": {
"grokPath": "/apps/metron/patterns/squid",
"patternLabel": "SQUID_DELIMITED",
"timestampField": "timestamp"
},
"fieldTransformations" : [
{
"transformation" : "MTL"
,"output" : [ "full_hostname", "domain_without_subdomains" ]
,"config" : {
"full_hostname" : "URL_TO_HOST(url)"
,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
}
}
]
}
Kafka Topic
Filter Location
Stellar Transformation Language
Create 2 additional fields:
applying USL_TO_HOST and
DOMAIN_REMOVE_SUBDOMAINS
DOMAIN_TO_TLD (domain)
DOMAIN_REMOVE_TLD(domain)
URL_TO_HOST(url)
URL_TO_PROTOCOL(url)
…
Parser Configurations
Field Transformations
• Configuration stored in
ZooKeeper
• Configure parser and field
transformations

Data Ingestion Checkpoint / Tracing an event
Raw Source Data  Metron JSON Object
• Numerous sensor logs in different formats
• The parser normalizes a subset of fields
• Data is parsed into the Metron JSON
Object
1462366408966.966 161 127.0.0.1
TCP_MISS/200 105413 GET
tp://www.cnn.com/ - DIRECT/199.27.79.73
text/html
Metron Storm Parsing

Step 3 Configure Real-time Enrichment
Enriching events with WHOIS information
• Enrichment reference data stored in HBase
• Configuration stored in ZooKeeper
• WHOIS data bulk loaded using Metron framework
• Sample WHOIS data used:
Metron Enrichment
Store
(HBase/)
Enrichment Loader
Framework
Bulk Load Polling
Enrichment Source
Storm
Bolt
Cache
Metron Streaming Messages Enriched Metron Streaming Messages
google.com, "Google Inc.", "US", "Dns Admin",874306800000
work.net, "", "US", "PERFECT PRIVACY, LLC",788706000000
capitalone.com, "Capital One Services, Inc.", "US", "Domain
Manager",795081600000
cisco.com, "Cisco Technology Inc.", "US", "Info Sec",547988400000
cnn.com, "Turner Broadcasting System, Inc.", "US", "Domain Name
Manager",748695600000
Bulk Load or Streaming

{
"zkQuorum" : "$ZOOKEEPER_HOST:2181"
,"sensorToFieldList" : {
"squid" : {
"type" : "ENRICHMENT"
,"fieldToEnrichmentTypes" : {
"domain_without_subdomains" : [ "whois" ]
}
}
}
}
{
"config" : {
"columns" : {
"domain" : 0
,"owner" : 1
,"home_country" : 2
,"registrar": 3
,"domain_created_timestamp": 4
}
,"indicator_column" : "domain"
,"type" : "whois"
,"separator" : ","
}
,"extractor" : "CSV"
}
Step 3 Configure Real-time Enrichment
Extractor Configuration Enrichment Configuration
Metron Enrichment Bulk Loader Utility
Map Columns to
enrichment data
source
Identify column
to match on
Configure field to
enrichment type
mapping

Data Enrichment Checkpoint / Tracing an event
Metron JSON Object  Enriched Metron object
• Enrichment data is added to the Metron
JSON Object
Owner
Data Enrichment
Each event is enriched with WHOIS
data data based on domain mapping
Country
Registrar

Threat Intel Store
(HBase)
Threat Intel Loader
Framework
Bulk Load Polling
Storm
Bolt
Cache
Metron Streaming Messages
(Enriched)
Enriched Metron Streaming Messages
(Enriched) + Threat Intel Hits
Threat Intel Feed Source
(Optional) Threat
Intel Aggregator
• Threat Intel feeds are either bulk loaded or
streamed
• Similar to enrichment framework
• Mapping to flag out any matches between the
Threat Feed and Streaming data
• is_alert flag=true is generated on matches
Alerts via Threat Intel Feeds
URL_TO_HOST(url)
…
• Metron ‘Threat Triage’
• Define rules based on incoming data
• Use any field within the rules (newly enriched
fields)
• Label alert severity levels based on rule
conditions
Alert severity via Defined Rules
Step 4 Configure Threat Intel and Alerting

{
"config" : {
"columns" : {
"domain" : 0
,"source" : 1
}
,"indicator_column" : "domain"
,"type" : "zeusList"
,"separator" : ","
}
,"extractor" : "CSV"
}
• Domain is mapped against this Threat Intel Feed
• Alerts generated when a match is hit
• Zeus malware tracker list used
• Feed Bulk Loaded:
domain,source
• Sample threat intel data:
Threat Intel Feed Mapping
URL_TO_HOST(url)
…
Step 4a Configure Threat Intel and Alerting
malware_intel_feed.csv
039b1ee.netsolhost.com,abuse.ch
03bbec4.netsolhost.com,abuse.ch
0if1nl6.org,abuse.ch
0x.x.gg,abuse.ch
1st.technology,abuse.ch
76tguy6hh6tgftrt7tg.su,abuse.ch
agiftcard724.com,abuse.ch
…
Identify column
mappings for the
threat Intel feed
Specify column
to match on
{
"zkQuorum" :
"$ZOOKEEPER_HOST:2181"
,"sensorToFieldList" : {
"squid" : {
"type" : "THREAT_INTEL"
,"fieldToEnrichmentTypes" : {
"domain_without_subdomains" :
[ "zeusList" ]
}
}}}
Metron Threat Intel Bulk Loader Utility
Configure field to
threat Intel
mapping

 Requirement For Scoring a Specific Type Threat Intel Alert:
– Rule 1: If the threat intel enrichment came from threat intel feed called zeusList is alerted, then
we want to consider that an alert of score of 5
– Rule 2: If the url is neither a .com nor a .net, then we want to consider that alert a score of 10
Step 4b Alert Triage (Scoring Alerts)
Rule 2 If url is not a .com
OR .net. Score = 10
Rule 1 If threat intel exists
in the Zeus list. Score = 5
Aggregator defined for when
multiple conditions are met

Visualize Enriched Data and Alerts
(Example) Trend of Metron generated alerts for
data categorized by the alert risk level
Drill down into Squid data events
• Kibana Driven Dashboards
• List and Visualize Squid Data
List of Metron generated alerts ordered by risk
level - record level drill down
Step 5 Enhance the Metron UI

Metron Default Dashboard Kibana 4
• Displaying network data
collected from the
Metron probes
• In-built geo enrichment
for default sensors feed
the map view

Key Takeaways…
• Easy Extensibility - The ability to add new data source without writing
any code and in an easy manner!!
• Repeatable Pattern - The reference application represents a repeatable
pattern that you can apply to most data sources
• Configuration Drive - End to end framework to build real-time
enrichment and alerting data pipelines

Apache Metron: Community Driven Cyber Security

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (14)

Similaire à Apache Metron: Community Driven Cyber Security

Similaire à Apache Metron: Community Driven Cyber Security (20)

Plus de DataWorks Summit/Hadoop Summit

Plus de DataWorks Summit/Hadoop Summit (20)

Dernier

Dernier (20)

Apache Metron: Community Driven Cyber Security