This presentation is how to build and use the Collective Intelligence Framework to start to collect threat intelligence from open-sources and use that to protect your network. This presentation was given for Hacker Hotshots on 11/27/2013.
How to Build the Collective Intelligence Framework - Hacker Hotshots 11/27/2013
1. Hacker Hotshots – 11/27/2013
How to Build the
Collective Intelligence Framework
And Start to Protect Your Network
John Bambenek
Chief Forensic Examiner, Bambenek Consulting
jcb@bambenekconsulting.com
2. Problem
Lots of people product lots of data,
blacklists, indicators of badness out there.
They all have their own formats and means
of distribution.
How to take multiple datasets, normalize
them and take action?
“Does anyone know anything about X?”
3. Solution: Collective Intelligence
Framework
Developed by REN-ISAC
http://code.google.com/p/collective-intelligence-
framework/
Does not generate data, simply takes sources
normalizes it and then outputs by given types
Not really a data sharing tool
Up to user to assess confidence in the data
Limited in the types of data it can handle
6. CIF Architecture
By default, everything lives in /opt/cif
Configs in /opt/cif/etc/*.cfg (CIF processes
all files ending in .cfg)
cif_smrt – queries the feeds
cif_feed – generates feeds by assessment
cif – command-line client tool
cif_crontool – used for querying all feeds
automatically
7. Requirements to Install
For a “real” instance, you would need some disk
(250 GB – 500 GB) and RAM (16 GB)
Disk is driven by how long you want to keep old data
Memory is only needing while parsing data
CIF can be placed in a virtual infrastructure easily
Can install it on most everything, Debian/Ubuntu
easiest mostly because the instructions are available
and clear
Ubuntu 12 probably best, 13 has some undocumented
changes that need to be made
Some kernel tweaking is needed
8. CIF Queries
Generally an analyst investigating will use
queries to see what is in the database.
cif –q <IP ADDRESS|DOMAIN
NAME|MD5>
Will include search records in the response
(unless suppressed)
Exact matching only (can’t search for part
of a URL… yet)
9. CIF Queries
CIF also ships a browser plugin which is
a little easier for analysis
Use cif_apikeys –l to get your key, find
your amazon IP and configure it now
Can query specific items or feeds
10. CIF feeds
cif –q feed/assessment –p output type [-z 0]
(-z 0 will prevent truncating URLs)
cif –q infrastructure/scan
Try with a lower confidence level
cif –q url/phishing –c 45
Not all output plugins work for all feeds
Full list at: http://code.google.com/p/collective-
intelligence-framework/wiki/API_FeedTypes_v1
11. CIF output types
bindzone
Bind zone configuration
bro
bro (network monitor)
csv
comma separated value
html
Html-ized table
iptables
iptables drop rules
json
json
pcapfilter
pcap filter (i.e. tcpdump)
snort
snort alert rules
table
ascii table (default)
12. CIF Output
There are dozens of sources (many don’t have
configs in CIF), but you can integrate them all
into CIF and/or a feed.
What to do with this now?
Snort Rules
Feed to web proxy to block/alert
Send to border device to blacklist IPs
Set up a sinkhole
You can also put your own data into CIF for
later research