Contenu connexe
Similaire à Tracing your security telemetry with Apache Metron (20)
Plus de DataWorks Summit/Hadoop Summit (20)
Tracing your security telemetry with Apache Metron
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What is Apache Metron?
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What Apache Metron Does?
“Apache Metron provides a scalable advanced
security analytics framework built with the Hadoop
Community evolving from the Cisco OpenSOC
Project.
A cyber security application framework that provides
organizations the ability to detect cyber anomalies
and enable organizations to rapidly respond to
identified anomalies.”
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Metron Timeline
Sep 2014 •OpenSOC Beta
June 2015
•OpenSOC Community Edition
Dec 2015
•Metron enters Apache Incubator
April 2016
•Apache Metron 0.1
Now
•Working towards 0.2 release
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Who is Metron for?
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Streaming Parsing and Enrichment
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Metron’s parsing bolt can be configured two ways
– And outputs JSON
Grok Parser
– Less work to implement
– Regex-like syntax
– Good for lower volumes of data
Java Parser
– More work to implement
– Good for higher volumes of data
Parsing
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enrichment / Threat Intel
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Add additional information to raw source during streaming
Adding it during streaming allows ML models to score in real time instead of
batch
Primarily stored in HBase
Several enrichments
– GeoIP
– Host
– Threat Intelligence
Enrichment
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Occurs in the same Storm topology as enrichment
Very similar process and flow
Use a threat feed aggregator!
– Soltra adapter is provided to read feed and stream into HBase
– Flat File loader and Stix bulk loader available without threat feed aggregator
Threat Intel
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Field Description
ip_src_addr Octet source IP
ip_dest_addr Octet destination IP
ip_src_port Integer source port
ip_dest_port Integer destination port
protocol String protocol (e.g. TCP)
timestamp Sensor epoch timestamp
source.type yaf, snort, etc.
start_time Metron epoch timestamp
end_time Metron epoch timestamp
Metron JSON
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Standalone Storm topology
Reads from Kafka
Writes packets to HDFS
Kibana panel forwards request to REST PCAP service
– MR Job launched
– Delivers results back to Kibana
PCAP
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Tracing a Source Through Metron
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Caching proxy
– Mostly useful as a source of easy to get and easily readable logs
Squid
1467125585.752 5288 127.0.0.1 TCP_MISS/200 32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html
Time Elapsed Remote Host Code/Statu
s
Bytes Metho
d
URL rfc931 Peer Status/ Peer Host Type
1467125585.752 5288 127.0.0.1 TCP_MISS/2
00
32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid - Grok
Time Elapsed Remote Host Code/Statu
s
Bytes Metho
d
URL rfc931 Peer Status/ Peer Host Type
1467125585.752 5288 127.0.0.1 TCP_MISS/2
00
32250 GET https://news.ycombinator.com/ - DIRECT/104.20.43.44 text/html
SQUID_DELIMITED %{NUMBER:timestamp}%{SPACE:UNWANTED}
%{INT:elapsed}%{SPACE:UNWANTED}%{IPV4:ip_src_addr} %{WORD:action}/%{NUMBER:code}
%{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url} -
%{WORD:UNWANTED}/%{IPV4:ip_dst_addr} %{WORD:UNWANTED}/%{WORD:UNWANTED}
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid – Topology Definition
{ "parserClassName": "org.apache.metron.parsers.GrokParser", "sensorTopic": "squid", "pars
erConfig":
{ "grokPath": "/apps/metron/patterns/squid", "patternLabel": "SQUID_DELIMITED", "tim
estampField": "timestamp" },
"fieldTransformations" : [
{
"transformation" : "MTL" ,"output" : [ "full_hostname",
"domain_without_subdomains" ] ,"config" : { "full_hostname" :
"URL_TO_HOST(url)" ,"domain_without_subdomains" :
"DOMAIN_REMOVE_SUBDOMAINS(full_hostname)" } } ] }
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid – Topology Result
- 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Loading some WHOIS derived data.
– Not directly making WHOIS query, just using a CSV containing a few rows of data.
Squid – Enrichment Definition
{
"zkQuorum" : ”localhost:2181"
,"sensorToFieldList" : {
"squid" : {
"type" : "ENRICHMENT"
,"fieldToEnrichmentTypes" : {
"domain_without_subdomains" : [ "whois" ]
}
}
}
}
- 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid – Enrichment Result
- 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Loading a list of malicious domains
– ZeuS tracker
Squid – Enrichment Definition
{
"zkQuorum": "localhost:2181",
"sensorToFieldList": {
"squid": {
"type": "THREAT_INTEL",
"fieldToEnrichmentTypes": {
"url": ["zeusList”]
}
}
}
}
- 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Squid – Threat Intel Result
- 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions?
Justin Leet
Systems Architect
jleet@hortonworks.com
justinjleet@gmail.com