Apache Metron Meetup & Code Lab: Introduction to Metron Architecture

Apache Metron
Meetup & Code Lab
George Vetticaden
Principal Architect @ Hortonworks
Apache Metron Committer
James Sirota
Engineering Lead & Chief Data Scientist @ Hortonworks
Apache Metron Committer

Part 1 – Overview of Apache Metron
• Challenges with Today’s Security Tools to Combat Cyber Attacks
• Introduction to Apache Metron
• Metron Architecture
• Personas and Core Themes
• Why Apache Metron?
Part 2 – Code Lab: Adding a Net New Data Telemetry Data Source into Metron
• Setting up the Use Case for the Code Lab: Tracing a Squid Telemetry through the platform
• Get your Metron vagrant VM started
• Use Case 1: Adding a net new telemetry data source to Metron
• Use Case 2: Enriching Telemetry Data
• Use Case 3: Adding/Enriching/Validating with Threat Intel Feeds
• Use Case 4: Setting up your IDE and writing Tests
Agenda

The Good Guys
Security
Practitioner
I have too many tools I need to learn
I don’t have a centralized view of my data
My tools are too expensive
I can’t find enough talent
I can’t keep relying on static rules
I need to discover bad stuff quicker
Most of my alerts are false positives
I have too many manual tasks
SOC
Manager
Threat landscape too dynamic
More assets/users to manage
Attack surface increases
Legacy techniques don’t work anymore
Metron will make it easier and faster to find
the real issues I need to act on
Metron is a more cost effective way for my team
to deal with the fast moving threat landscape

The Bad Guys
Advanced
Persistent
Threat
Script
Kiddie
My techniques are predictable and known
My attack vectors are also known
You are not the only person I’ve attacked
I brag about what I did or will do
I set off a large number of alerts
I fumble around a lot
I am very unique in a way I do things
I live on your network for about 300 days
I know what I am after and I look for it, slowly
Your rules will not detect me, I am too smart
I impersonate a legitimate user, but I don’t act like one
Metron can take everything that is known
about me and check for it in real time
Metron can model historical behavior of whoever I am
impersonating and flag me as I try to deviate

Problems With Existing Tools
Security
Information
Management
System
I am prohibitively expensive
I have vendor lock-in
I can’t deal with big data
I am not open
I am not extensible enough
Legacy
Point
Tools
I was built for 1995
I am super specialized
I don’t scale horizontally
I have a proprietary format
You need a PhD to operate me
Behavioral
Analytics
Tools
I am mostly vapor ware
I was built by a small startup
I was modeled after a data set from 1999
I spam you with false positives

Apache Metron Vision
“Apache Metron is a Security Data
Analytics Platform (SDAP). As a
next generation security analytics
framework, it is designed to
consume and monitor network
traffic and machine data within an
enterprise. Apache Metron is
extensible and is designed to work
at a massive scale. It is not a SIEM
but rather the next evolution of a
SIEM.”
Apache Metron provides the following capabilities:
 Extensible spouts and parsers for attaching Apache
Metron to monitor any telemetry source
 Extensible enrichment framework for any telemetry
stream
 Hadoop-backed storage for telemetry stream with a
customizable retention time
 Automated real-time index for telemetry streams
enabling real-time search
 Telemetry correlation and SQL query capability for data
stored in Hadoop backed by Hive
 ODBC/JDBC compatibility and integration with existing
analytics tools

Challenges that Apache Metron Solves
60%: Percent of breaches that
happened in minutes
8 months: Average time an
advanced security breach goes
unnoticed
$400 million in estimated
financial loss in 2015
70%-90%: Percentage of
malware in breach unique to
organization
2015 Verizon Data Breach Investigations Report
• Too expensive to keep data for enough time to
understand history
• Not enough of the right data to provide
context
• Too expensive to collect all the desired data to
understand context
• Not sure if can detect a targeted event.
• Too many events to review in timely manner
• Not enough staff to review events in a timely
manner

Real-time Processing Engine
PCAP
NETFLOW
DPI
IDS
AV
EMAIL
FIREWALL
HOST LOGS
PARSE
NORMALIZE
TAG
VALIDATE
PROCESS
USER
ASSET
GEO
WHOIS
CONN
ENRICH
STIX
Flat Files
Aggregators
Model As A
Service
Cloud
Services
LABEL
PCAP
Store
ALERT
PERSIST
Alert
Security Data
Vault
Apache Metron Logical Architecture
Network
Tap
Custom Metron UI/Portals
Real-Time
Search
Interactive
Dashboards
Data
Modelling
Integration
Layer
PCAP
Replay
Security
Layer
Data & Integration Services
Apache Metron

Sensor
A
Sensor
B
Sensor
N
Topic A
Topic B
Topic (N)
Apache
Kafka
PCAP
PCAP
Probe
Physical Architecture
Normalizing
Topology A
Normalizing
Topology B
Normalizing
Topology N
Apache
Storm
Native Format
Native Format
Native Format
PCAP on HDFS Metron PCAP
Service
PCAP
Topology
Enrich
Normalized
Metron
Format Enrichment/
Threat Intel
Topology
Out to Index + HDFS

Topic A
Normalizing
Topology A
Sensor
A
Native Format
Apache
Kafka
Apache
Storm
Kafka
Spout
Parser
Kafka
Bolt
Enriched
Metron JSON
Parsing/Normalization Topology
Key Points:
• Each New Telemetry Data Source will have its own Parser Topology
• Two types of Parsers available: Grok and Java

2 Types of Parsers
Parser Type Description Telemetry Type
Grok • A grok is a collection of named regular expressions.
• Provides a declarative way to write new parsers
without any code
• A parser takes an input, which is usually a byte
array coming from the Kafka Spout, and turns it into
a Metron JSON Object.
• The Grok parser does this by utilizing the Grok
library inside of the Parser Kafka Bolt Adapter.
• Use this parser when
telemetry is simple to parse
or low in volume
Java • Java based approach to writing a custom parsers • Use this parser when
telemetry is complex to
parse or high volume

Metron JSON Object
• Numerous sensors log in different formats. The parser should normalize at least the
following subset of fields to the following Metron JSON naming conventions:

Enrich
ment
Bolt(a)
Enrich
ment
Bolt(n)
Threat
Intel
Joiner
Message
Splitter:
Enrichment
Enrich
ment
Joiner
Message
Splitter:
Threat Intel
Model
Bolt
(n)
Threat
Intel
Bolt
(n)
Metron
Enrichment
Loader
Framework
Metron Threat
Loader
Framework
Data
Store
Fast
Cach
e
Fast
Cach
e
Fast
Cach
e
Fast
Cach
e
Data
Store
Enrichment
Topology
Apache
Kafka
Enriched
Writer
Bolt
= Message Stream
Apache Storm
= Enrichment Stream
Enrichment Topology

Metron’s Key Functional Themes
Platform
Work done to harden the platform for performance, scale, extensibility
and maintainability. This also includes capabilities around
provisioning, managing and monitoring the application.
Set of Data Sources that Metron provides capabilities to stream,
ingest and parse into the platform.
A set of Storm Topologies to perform various actions in real-time
including: normalization of telemetry data, enrichment, cross
reference with threat intel feeds, alerting, indexing, and persisting into
Historical stores
Data Collection
Data Processing
UI Set of portal, dashboard and user interfaces for the different
personas.

Target Personas and Themes for Apache Metron 0.1
T
e
c
h
P
r
e
v
i
e
w
1
-
I
n
t
r
o
Theme: Platform Theme: Data Collection
Theme: Data Processing Theme: UI
Security Platform
Engineer
Security Platform
Engineer
Security Platform
Engineer
SOC Investigator Security Platform
Engineer SOC Investigator
Forensic Investigator SOC Investigator
SOC Analyst SOC
Manager

• Fully automated vagrant install of Metron on a single VM
• Fully automated install of Metron on multi-node HDP cluster via Ansible scripts, Ambari
blueprints and APIs including:
• Multi-node Elastic Search Cluster
• Metron-UI Web Application
• Deployment of the Metron Storm Topology
• Deployment of telemetry sensors: PCAP, Bro, YAF(Netflow), Snort
• OpenSOC redesign (new topology structure, extensible enrichments, threat intel, data
loads, configs, ease of adding new topologies)
Platform
Data Collection
• Ingestion of the following data sources: PCAP via pycapa or C++ DPDK probe, Bro,
Netflow via YAF, Snort
• Parsers for the following data sources: PCAP, Bro, Netflow & Snort
Data Processing
• Support for the following enrichment services: Geo, WhoIs, Host
• Threat Intelligence Message enrichment - Enrich messages with fields that mat the
threat intelligence data in HBase
• Support for the following persistence services: HDFS, HBase and Elastic Search
• Indexing events and Alerts into Elastic Search cluster
• Support for Soltra(CIF) Threat Aggregator Services via STIX and Taxii Feed
• Ability to replay PCAP files for Testing
UI
• Metron Investigator UI to search across indexed events and alerts for SOC Analyst &
Investigators
• Histogram Panels for each of the data sources (YAF, Bro, Snort)
• Table Views for Alerts (YAF, Bro, Snort)
• Customize new panels with different data sources and different panel types.
Key Features of Apache Metron 0.1

Why Metron? SOC Analyst Perspective
Looking through
alerts
25%
Collecting contextual
data
25%
Formulating a
Hypothesis
5%
Investigate
20%
Remediate
15%
Update Workflow
5%
Wrte Report
5%
ANALYST WORKFLOW • Alerts Relevancy Engine
• Smarter ML alerts
• Centralized Alerts Console
• Enriched with threat intel data
• Fully enriched messages
• Single pane of glass UI
• Centralized real-time search
• All logs in one place
• Granular access to PCAP
• Replay old PCAP against new signatures
• Tag behavior for modelling by data scientists
• Raw messages used as evidentiary store
• Mine investigation history
• Asset inventory as an enrichment
• User identity as an enrichment
• Workflow engine
• Ticket clustering
Everything you need to know in one place

Why Metron? Data Scientist Perspective
Formulating a
Hypothesis
5%
Finding Data
20%
Cleaning Data
20%
Munging Data
20%
Visualizing Data
20%
Modelling Data
10%
Validating Model
5%
DATA SCIENCE WORKFLOW
• All my data is in the same place
• Data exposed through a variety of APIs
• Standard Access Control Policies
• Quickly see what I have
• Metron normalizes objects
• Partial schema validation on ingest
• Tagging on ingest
• Automatic data enrichment
• Automatic application of class labels
• Common Metron Objects
• Massively parallel computation framework
• Reusable Zeppelin Dashboards
• Real-time search + UI
• Integration with Python/R
• Integration with analytics tools
Reducing time from hypothesis to model

Use Case Setup
• Scenario
• Customer Foo has installed Metron TP1 and they are using the out of the box data sources (PCAP,
YAF/Netflow, Snort and Bro). They love Metron!
• But now they want to add new data source the the platform: squid proxy logs.
• Customer Foo’s requirements are the following
1. Need to ingest the proxy events from Squid logs in real-time
2. The proxy logs has to be parsed into a standardized JSON structure that Metron can understand
3. In real-time, the squid proxy event needs to be enriched with domain/whois information (domain,
cert, country, company)
4. In real-time, the domain of the proxy event must be checked against for threat intel feeds
5. If there is a threat intel hit, an alert needs to be raised
6. The end user must be able to see the new telemetry events and the alerts from the new data
source

Squid & its Telemetry Event
• What is Squid?
• Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and
improves response times by caching and reusing frequently-requested web pages
• What does a Squid Access Log look like?
• When you make an outbound http connection to https://www.cnn.com, the following entry gets added to a file
called access.log:
Unix Epoch Time
IP of host where connection was made.
The domain name
of the outbound connection
1461576382.642 161 98.220.218.158 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html

What Metron does to the Squid Telemetry Event in Real-time
Convert from
Unix Epoch to
Timestamp
Use Metron’s asset enrichment to enrich
that IP (hostname, type of device)
Use Metron’s WhoIs enrichment
To look up domain name information (e.g:
Use the Metron’s Threat Intel Services
to cross-reference the IP with threat intel
feed to see if there is a hit
Index the event into Elastic
and persist into HDFS
(Security Data Vault)

Real-time Processing Engine
Squid Logs
PARSE
NORMALIZE
TAG
VALIDATE
PROCESS
USER
ASSET
GEO
WHOIS
CONN
ENRICH
STIX
Flat Files
Aggregators
Model As A
Service
Cloud
Services
LABEL
PCAP
Store
ALERT
PERSIST
Alert
Security Data
Vault
Real-Time
Search
Interactive
Dashboards
Data
Modelling
Integration
Layer
PCAP
Replay
Security
Layer
Data & Integration Services
Tracing the Squid Event across the Platform
Custom Metron UI/Portals

Step 1: Telemetry Ingest (Tracing an Event)

Step 2 – Process/Parse (Tracing an Event)

Step 3 – Enrich (Tracing an Event)

Enriching Data Architecture
Metron Enrichment
Store
(HBase/)
Enrichment Loader
Framework
Bulk Load Polling
Enrichment Source
Storm
Bolt
Cache
Metron Streaming Messages Enriched Metron Streaming Messages

Step 4 – Label/Threat Intel (Tracing an Event)
Threat Intel Store
(HBase)
Threat Intel Loader
Framework
Bulk Load Polling
Storm
Bolt
Cache
Metron Streaming Messages
(Enriched)
Enriched Metron Streaming Messages
(Enriched) + Threat Intel Hits
Threat Intel Feed Source
(Optional) Threat
Intel Aggregator

High level Steps – How to Add the New Telemetry
1. Create new Kafka topic for the new telemetry source called “squid”
2. Create and validate a grok statement file that parses the squid event log into a format that Metron can understand
3. Store that grok statement in HDFS
4. Create a new flux configuration for the new Squid parser Storm Topology.
5. Update Zookeeper with configuration to mark what fields in the telemetry to enrich and what fields to cross-
reference with threat intel feeds.
6. Move the flux configuration to the host where you will deploy the topology.
7. Deploy the new squid Storm parser topology using the new flux configuration
8. Load WhoIs enrichment data and configure enrichment mapping
9. Load Threat Intel data and configure threat intel matching mapping
10. Use Apache Nifi to capture the squid events and push them into Metron
11. Create a new Panel in Kibana and see the telemetry events
Key Points
Easy Extensibility – The ability to add new data source without writing any code and in an easy mann
Repeatable Pattern - The following represents a repeatable pattern that you can apply to most data s

Apache Metron Meetup & Code Lab: Introduction to Metron Architecture

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (18)

Similaire à Apache Metron Meetup & Code Lab: Introduction to Metron Architecture

Similaire à Apache Metron Meetup & Code Lab: Introduction to Metron Architecture (20)

Plus de Hortonworks

Plus de Hortonworks (20)

Dernier

Dernier (20)

Apache Metron Meetup & Code Lab: Introduction to Metron Architecture