2. Who is this session for?
Security
Practitioners
Security Architects
Security execs
Compliance/Audit
3. Agenda
Splunk for Security (20 min)
• Demonstration of Splunk Enterprise (10 min)
• Demonstration of the Splunk App for Enterprise Security (20 min)
• Q&A
•
4. Advanced Threats in the Headlines
Cyber Criminals
Nation States
Insider Threats
“160 million credit cards later, cutting edge hacking ring cracked”
– NBC News, July 2013
“Banks Seek U.S. Help on Iran Cyber attacks”
– Wall Street Journal, Jan 2013
“Verizon: Most Intellectual Property Theft Involves Company Insiders”
– Dark Reading, Oct 2012
5. Advanced Threats Are Hard to Detect
100%
243
Valid credentials
were used
Median # of days
before detection
40
63%
Average # of systems
accessed
Of victims were notified
by external entity
Source: Mandiant M-Trends Report 2012 and 2013
5
6. All Data is Security Relevant = Big Data
Databases
Email
Web
Desktops
Servers DHCP/ DNS Network
Flows
Traditional SIEM
Custom
Hypervisor Badges Firewall Authentication Vulnerability
Apps
Scans
Storage
Mobile
Data Loss
Intrusion
Detection Prevention
AntiMalware
Service
Desk
Call
Industrial
Control Records
7. Limitations of Existing SIEMs
Traditional SIEM
• Limits view of security threats. Difficult to collect all data
sources; requires costly, custom collectors for DB schema.
• Inflexible search/reporting hampers investigations and
threat detection
• Scale/speed issues impede ability to do big data analytics
• Difficult to deploy and manage; often multiple products
7
8. Solution: Splunk, the Engine For Machine Data
GPS, RFID, Hypervisor, Web
Servers, Email, Messaging,
Clickstreams, Mobile,
Telephony, IVR, Databases
Ad hoc
search
Monitor
and alert
Custom
dashboards
Report
and
analyze
Developer
Platform
Real-time
Machine Data
Sensors, Telematics,
Storage, Servers,
Security
devices, Desktops, CDRs
Splunk storage
8
Other Big Data stores
12. Splunk Security Use Cases
A Security Intelligence Platform
Splunk Can Complement OR Replace Existing SIEMs
Incident
Investigations /
Forensics
Security /
Compliance
Reporting
Real-Time
Monitoring of
Known Threats
Real-Time
Monitoring of
Unknown Threats
13. Use Case 1 - Incident Investigation/Forensics
January
•
May be a “cold case” investigation requiring
machine data going back months
March
Often initiated by alert in another product
•
February
•
Need all the original data in one place and a
fast way to search it to answer:
–
What happened and was it a false positive?
–
How did the threat get in, where have they
gone, and did they steal any data?
–
•
client=unknown[
99.120.205.249]
<160>Jan
2616:27
(cJFFNMS
Has this occurred elsewhere in the past?
truncating
integer value >
32 bits
<46>Jan
ASCII from
client=unknow
n
Take results and turn them into a real-time
search/alert if needed
DHCPACK
=ASCII
from
host=85.19
6.82.110
13
April
14. Case #2 – Security/Compliance Reporting
Many types of visualizations
Easy to create in Splunk
– Ad-hoc auditor reports
– New incident list
– Historical reports
– SOC/NOC dashboards
– Executive/auditor dashboards
14
15. Case #3 – Real-time Monitoring of Known Threats
Sources
Windows
Authentication
Endpoint
Security
Example Correlation – Data Loss
20130806041221.000000Caption=ACME-2975EBAdministrator Description=Built-in account for administering
the computer/domainDomain=ACME-2975EB InstallDate=NULLLocalAccount = IP: 10.11.36.20
Default Admin Account
TrueName=Administrator SID =S-1-5-21-1715567821-926492609-725345543 500SIDType=1
Source IP
Status=Degradedwmi_ type=UserAccounts
Aug 08 06:09:13 acmesep01.acmetech.com Aug 09 06:17:24 SymantecServer acmesep01: Virus found,Computer
name: ACME-002,Source: Real Time Scan,Risk name: Hackertool.rootkit,Occurrences: 1,C:/Documents and
Settings/smithe/Local Settings/Temp/evil.tmp,"""",Actual action: Quarantined,Requested action: Cleaned, time:
Malware Found
Source IP
2009-01-23 03:19:12,Inserted: 2009-01-23 03:20:12,End: 2009-01-23 03:19:12,Domain: Default,Group: My
CompanyACME Remote,Server: acmesep01,User: smithe,Source computer: ,Source IP: 10.11.36.20
Aug 08 08:26:54 snort.acmetech.com {TCP} 10.11.36.20:5072 -> 10.11.36.26:443 itsec snort[18774]:
[1:100000:3] [Classification: Potential Corporate Privacy Violation] Credit Card Number Detected in Clear Text
Source IP
[Priority: 2]:
Data Loss
Intrusion
Detection
All three occurring within a 24-hour period
Time Range
15
16. Case #4 – Real-time Monitoring of Unknown Threats
Sources
Example Correlation - Spearphishing
User Name
2013-08-09T12:40:25.475Z,,exch-hub-den-01,,exch-mbx-cup00,,,STOREDRIVER,DELIVER,79426,<20130809050115.18154.11234@acme.com>,johndoe@acme.com,,685191,1
,,, hacker@neverseenbefore.com , Please open this attachment with payroll information,, ,2013-0809T22:40:24.975Z
Email Server
Rarely seen email domain
Rarely 16:21:38 10.11.36.29 98483 148 TCP_HIT 200 200 0 622 - - OBSERVED GET
2013-08-09visited web site
www.neverbeenseenbefore.com HTTP/1.1 0 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
2.0.50727; InfoPath.1; MS-RTC LM 8; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; ) User John Doe,"
User Name
Web Proxy
Endpoint
Logs
User Name
08/09/2013 16:23:51.0128event_status="(0)The operation completed successfully. "pid=1300
process_image="John DoeDeviceHarddiskVolume1WindowsSystem32neverseenbefore.exe“ registry_type
="CreateKey"key_path="REGISTRYMACHINESOFTWAREMicrosoftWindows NTCurrentVersion Printers
PrintProviders John Doe-PCPrinters{} NeverSeenbefore" data_type"" Rarely seen service
All three occurring within a 24-hour period
Time Range
16
17. Case #4 – More Examples
Attack
Phase
What Threat is Doing
What to Look For
Lateral
movement
Creating new admin accounts
Account creation without corresponding IT
service desk ticket
Data
gathering
Stealing credentials
For single employee: Badges in at one
location, then logs in countries away
Data
gathering
Gathering confidential data for
theft
Employee makes standard deviations more
data requests from file server with
confidential data than normal
Exfiltration Exfiltration of info
Standard deviations larger traffic flows (incl
DNS) from a host to a given IP
17
Data
Source
AD/
Service
Desk logs
Badge/
VPN/
Auth
OS
NetFlow
19. Splunk Key Differentiators
•
•
•
•
•
•
•
•
Splunk
Traditional SIEM
Single product, UI, data store
Software-only; install on commodity hardware
Quick deployment + ease-of-use = fast time-to-value
All original machine data is indexed and searchable
Big data architecture enables strong scale and speed
Flexible search and reporting enables better/faster threat
investigations and detection, incl finding outliers/anomalies
Open platform with API, SDKs, Apps
Use cases outside security lead to cross-dept collaboration
and increased ROI
19
20. Splunk Security Intelligence Platform
80+ security apps
Splunk App for Enterprise Security
Palo Alto
Networks
Cisco Security
Suite
F5 Security
FireEye
NetFlow Logic
Active
Directory
Juniper
20
Blue Coat
Proxy SG
Sourcefire
OSSEC
21. Next Steps
Splunk
Traditional SIEM
• Info, data sheets, white papers, recorded demos at:
Splunk.com > Solutions > Security
Splunk.com > Solutions > Compliance
• Try Splunk for free!
Download Splunk at www.splunk.com
Go to Splunk.com > Community > Documentation > Search Tutorial
In 30 minutes will have imported data, run searches, created reports
Free Apps at Splunk.com > Community > Apps
• Contact sales team at Splunk.com > About Us > Contact
21
The number of threats is increasing and also becoming more advanced. Today’s advanced threats are stealthy and sophisticated and evade detection from traditional, point security products that look for specific threat signatures. Above are 3 types of advanced threats. They are good at stealing confidential data, wherther it be credit cards or IP, and many of their victims unfortunately end up in the headlines.FYI these advanced threats are also commonly called APTs, or Advanced Persistent Threats.
APT are hard to detect because they are not signature-based and hide behind legitimate credentialed activity to evade detection from traditional, point security products. Every year companies like Mandiant produce reports that describe the trends identified based on the breach investigation work that they do as part of their consulting practices. There are a couple metrics that I found interesting reading their recent reports.100% is often via stealing password hashes or using keyloggers. Often they steal admin-level credentials so they can access many other systems and not be detected.The 40 implies that even if you see malware in one place, you need to look much further as there are likely multiple infected machines and backdoors243 days shows how they can evade detection for months at a time. They move slow and low and do not set off alarms from point, signature-based security products like anti-malware solutions.63% of victims were notified by an external entity. Notification usually starts with customer complaints like bank account drained or credit card maxed out. Often FBI informs them.
Key part of IT security is protecting confidential data. Which means detecting advanced threats, like cybercriminals or malicious insiders, before they can steal your data. To detect or investigatethem, you need non-security and security data because advanced threats avoid detection from signature-based security products; the fingerprints of an advanced threat often are in the “non-security” data. Most traditional SIEMs just focus on gathering signature-based threats which do *not* have the fingerprints of advanced threats.Also the above scenario is worse if there is no SIEM. Instead point UIs and grep are used and aggregating data is very manual and time consuming.
Here is why SIEMs are not catching the advanced threats and not generating value in general. (A later slide has more detail on Splunk points of differentiation. )For point 1, traditional SIEMs suffer from:Collectors or backend database require data reduction or normalization – hampers security use casesLittle support for custom data sourcesBrittle collectors break when data format changesClock icon indicates slow searches and long time to deploy. Dollar sign is the expensive costs behind the long deployment (lots of prof serv) and multiple products.
Splunk can ingest any type of machine data, from any source in real time. These are listed here on the left and are flowing into Splunk for indexing. Once indexed, users can perform the use cases on the top right on the data. They can search through the data, monitor the data and be alerted in real-time if scheduled search parameters are met. The raw data can be aggregated in seconds for custom reports and dashboards. Also Splunk is a platform that developers can build on. It uses a well documented Rest API and several SDKs so developers and external; applications can directly access and act on the data within Splunk. Lastly, besides indexing raw data into its flat file data store, Splunk can also retreive and index data that resides in other data stores such as a SQL database or Hadoop.
Over 2500 security/compliance customers worldwide. Customers cover all sizes and verticals, and are all over the world. While not listed here, hundreds of SMBs and individuals also use for security/compliance.
SC magazine award was determined by the readers of SC Mag who are IT Security professionals. We beat out:HP for ArcSight ExpressIBM Software Group for QRadar SIEMLogRhythm for LogRhythmNetIQ for NetIQ Sentinel 7SolarWinds for SolarWinds Log & Event Manager (LEM)
SC magazine award was determined by the readers of SC Mag who are IT Security professionals. We beat out:HP for ArcSight ExpressIBM Software Group for QRadar SIEMLogRhythm for LogRhythmNetIQ for NetIQ Sentinel 7SolarWinds for SolarWinds Log & Event Manager (LEM)
make sure to stress SIP and we can meet their needs w/4 use cases. We are more than a SIEM in that we are much more flexible and also can be used for use cases outside of security
Use case 1. Alert from a point product UI or traditional SIEM. Pin board image on right indicates “cold case/CSI” sort of investigation that Splunk can enable. (FYI - papers on the pin board image do not tell a “real” investigation story so do not try to read all the images on the pin board). From a forensics perspective, things like endpoint OS logs or packet captures can be put into Splunk at the time of the investigation to get deeper into the details.With exiting SIEMs they struggle with incident investigations because they cannot:Retain all the original unmodified data (because the normalize/reduce it)Often it is hard to pivot among the data b/c it is in different data stores (logger/SIEM/Hadoop/etc) with no common UIQuickly return back search results (because their DB causes scale/speed issues)Limited flexibility/ability to do external lookups
Use case 2 (a slide with building images)(*) Ad-hoc reports for security investigation, executives, auditors. Show me all internal machines connecting back to a known, bad external IP over the last 30 days.(*) Daily list of new security events that IT Security reviews each morning. They use this to decide what incidents to look into that day. Malware discovered, data loss events, etc.(*) Reports that count up top 10 sort of lists showing top malware infections, IDS attacks, failed logins by user, etc. (*) Security or Operations Network Center might have world map overlayed with security incidents, or maybe yellow/red/green dials showing the overall threat levels.Executives may want to see trendlines showing threats over time to see that risk is decreasing.This is a sample dashboard built by a customer
Use case 3. It is about about taking thousands of security events that are low severity in isolation and connecting the dots in an automated, policy-driven manner to see when a combination of seemingly low severity events, when correlated, is actually a high-severity incident that needs immediate attention.There are hundreds of possible cross-product correlations. One is above and tells the story of a data loss event being detected by signature-based security productsFor a specific internal IP address running Windows, someone logs into it using a default administrative user name “Administrator” which is not good. All users should have a unique user name (not root or Administrator) so you know exactly who is doing what in the IT environment. The OS logs see this log in.Endpoint-based anti-malware sees known, bad malware running on that machine. Malware means “malicious software” and is a red flag because it may lead to data being stolen by a hackerA data loss prevention tool (in this case the Snort Intrusion Detection Prevention product) sees unencrypted credit card numbers leaving the organization from the above machine. This data loss of credit cards is a major red flag.Why these 3 events are bad: These 3 events happening on the same machine in a short time period indicates a hacker inappropriately logged into the machine, probably using stolen credentials, then put malware on the machine, perhaps a backdoor to remotely connect back to the machine later, then exfiltrated stolen credit cards from the machine. The credit cards may have then been used for illegal purposes which ultimately may have resulted in the costs of re-issuing credit cards, bad publicity, unhappy customers taking their business elsewhere, customer lawsuits, fines for PCI non-compliance, etc.Splunk can correlate on all these 3 events happening on the same machine and within a short time period. It has connected the dots to find the proverbial needle in the haystack. Splunk can detect and/or alert on these sorts of correlations in real-time or on a scheduled bases.Two other sample correlations:Firewall on an internal PC indicates the PC is being port scanned from an internal IP addressNetwork-based firewall indicates it is being port scanned from the same internal IP addressImportant settings have been changed on the suspicious internal machineWhy: The machine associated with the IP address may have been compromised by a threat which is doing internal reconnaissanceVulnerability scanner shows that an internal server has an unpatched OSIntrusion Detection System sees an external attack on that specific server that exploits the vulnerability in the OS Why: The server is likely to be successfully compromised
Use case 4. Like prior slide so see notes at the top. But in this case the events being correlated on all are from “non-security” data sources. This is because the threats are “unknown” to traditional security products because no signature exists for them. Each of these events in isolation would raise no alarms. Only when combined can see you see that they are risky because they represent outliers/anomolies that could be advanced threats like a sophisticated cybercriminal or a nation-state.There are hundreds of possible cross-product correlations. One is above and tells the story of a spearphising attack done in order to obtain and steal confidential data. More sample correlations are on the next slide. In this scenario above Splunk is keeping track of all the external email domains that are sending emails into the company, all external web sites being visited by internal employees, and all the services and executables running on internal machines. It can automatically count up things like # of emails received by each external domain, # of times employees visit external web domains, etc, to see the rarely seen items that are outliers. In the above scenario:Splunk sees an email reach an internal employee from an external email domain that has never/rarely been seen beforeThat same employee then visits a web site that is never/rarely visited by internal employeesA service starts up on the employees machine that is never/rarely seen in the organizationWhy these 3 events are bad: These 3 events happening on the same machine in a short time period indicates a hacker has performed a spearphising attack. They sent a realistic looking email to an internal employee that compelled the employee to click on a link or open an executable. This resulted in the web site dropping malware on the machine.Splunk can correlate on all these 3 events happening on the same machine and within a short time period. It has connected the dots to find the proverbial needle in the haystack. Splunk can detect and/or alert on these sorts of correlations in real-time or on a scheduled bases.
More on use case 4. Nothing on this slide and next would generate an alert in a traditional security tool like anti-malware or firewall. All of these “what to look for” can be automated, real-time searches. These are just a few examples of how to detect what may be an advanced threat.#2 Haversine algorithm used to calculate distance
Customers start by using Splunk Enterprise to address one specific solution area. Then they leverage it and their machine data to solve other pressing problems over time.Consequently, Splunk Enterprise has many critical uses across IT and the business: Application Management: provide end-to-end visibility across distributed infrastructures; troubleshoot across application environments; monitor for performance degradation; trace transactions across distributed systems and infrastructure.Development: accelerate development and test cycles; support advanced development methodologies like agile, continuous; integrate enterprise applications with SDKs and a robust API; build enterprise applications that leverage Splunk software.Infrastructure and Operations Management: proactively monitor across IT silos to ensure uptime; rapidly pinpoint and resolve problems; report on SLAs/track SLAs of service providers.Security and Compliance: provide rapid incident response, real-time correlation and in-depth monitoring across data sources; statistical analysis for advance pattern detection and threat defense.Web and Business Analytics: gain visibility and intelligence on customers, services and transactions; identify trends and patterns in real time; fully understand the impact of new product features on back-end services.
1 solution for Splunk for Security, but 3 offerings. At bottom is Splunk Enterprise, our core product. Every Splunk deployment includes this as this is where the core indexing and searching resides. Many customers build their own searches/reports/dashboards on top of it. On top of it, optional Apps can be installed. Apps are basically a collection of reports, dashboards, and searches purpose-built for a specific use case or product. Can be built by Splunk, customer, partners and all but a few are free on Splunkbase. Apps are great for customers who want out-of-the-box content and do want to have to build it themselves, and want to extend point solutions. One key App is the Splunk-built Enterprise Security app with the arrow pointing at it. It is basically an out-of-the-box SIEM with reports, dashboards, correlation rules, and workflow for security use cases. (It does have a cost though) Besides this app there are over 80 security-centric free Apps on Splunkbase. These are offering 3.The majority of Splunk security customers do Splunk Enterprise and the free apps. Also customers do leverage the API and SDKs that come with Splunk to further extend the platform.
You need to look for abnormal events in normal user activity data. SIEM not built for this and the architecture doesn’t support it. Data reduction modelTypical funnel representation of:Limited data collectionData subset used for securitySubset may fire one of 200+ rulesLinear approach – no going back in time1. Traditional SIEM / log management sources – almost always not all your sources are supported. Custom application data almost always left out2. Data sent to a log managed solution for scale and filtering only to ‘security related” events. Nice to have information level data from security devices and data sent by applications left out.3. Security data sent to the SIEM – only that which fires a rule responded to.4.Reinforces silos between applications, operations and security. This means investigations limited to only what you expect to see – you never know what you don’t know! Splunk – a data inclusion model – ALL data is security relevantIf a correlation search produces a positive result – investigations aren’t artificially limited. Investigations can include seeing the affect of a security event all the way to the application – up to the second. Pattern and threshold based rather than rules based – watch for a pattern of activity get an alert before and after it hits a thresholdMassive scale can offer broader view of attack vectors and surfacesOperations and security data seen in the same system – breaks down silos/barriers
