More Related Content Similar to Big Data For Threat Detection & Response (20) More from Harry McLaren (20) Big Data For Threat Detection & Response1. BIG DATA FOR THREAT
DETECTION & RESPONSE
Harry McLaren – Managing Consultant at ECS
Sam Farmer – Security Operations Specialist
2. WHO AM I?
HARRY MCLAREN
•Alumnus of Edinburgh Napier (Now a Mentor)
•Managing Security Consultant at ECS
• Big Data Consultancy (Splunk)
• Building SOC Technology (SIEM)
Copyright © - ECS 2018
4. AGENDA
• Introduction & Agenda
• Security Operations Overview
• Challenge: Monitoring, Detection & Hunting
• Solution 1: Big Data, Splunk & Heterogeneous Data
• Example: Example of Advanced Threat Activity
• Solution 2: SIEM, Platform Evolution & Frameworks
• Successful SIEM Deployments & Operation
• Splunk User Group & Questions
Copyright © - ECS 2018
6. ADVANCED THREATS ARE HARD TO FIND
• Human directed
• Goal-oriented
• Dynamic (adjust to changes)
• Coordinated
• Multiple tools & activities
• New evasion techniques
• Fusion of people, process,
& technology
• Contextual and behavioral
• Rapid learning and response
• Share info & collaborate
• Analyze all data for relevance
• Leverage IOC & Threat Intel
Threat
Attack Approach Security Approach
Technology
People
Process
Copyright © - ECS & Splunk 2018
7. ADVANCED THREATS ARE HARD TO FIND
• Human directed
• Goal-oriented
• Dynamic (adjust to changes)
• Coordinated
• Multiple tools & activities
• New evasion techniques
Threat
Attack Approach Security Approach
Technology
People
Process
Analytics-driven Security
Connecting Data
and People
Risk-Based Context and
Intelligence
Copyright © - ECS & Splunk 2018
8. ADVANCED THREATS ARE HARD TO FIND
▶ Continuously Protect the
business against:
• Data Breaches
• Malware
• Fraud
• IP Theft
▶ Comply with audit requirements
▶ Provide enterprise Visibility
▶ 70% to 90% improvement with
detection and research of events
▶ 70% to 95% reduction in security
incident investigation
▶ 10% to 30% reduction in risks
associated with data breaches,
fraud and IP theft
▶ 70% to 90% reduction in
compliance labor
Top Goals Top Splunk Benefits
Copyright © - ECS & Splunk 2018
9. ADVANCED THREATS ARE HARD TO FIND
Servers
Storage
DesktopsEmail Web
Transaction
Records
Network
Flows
DHCP/ DNS
Hypervisor Custom Apps
Physical
Access
Badges
Threat Intelligence
Mobile
CMDB
Intrusion
Detection
Firewall
Data Loss
Prevention
Anti-Malware
Vulnerability
Scans
Traditional
Authentication
Copyright © - ECS & Splunk 2018
10. SOLUTION: SPLUNK, THE ENGINE
FOR MACHINE DATA
Custom
Dashboards
Report &
Analyze
Monitor
& Alert
Developer
Platform
Ad-hoc
Search
References – Coded fields, mappings, aliases
Dynamic information – Stored in non-traditional formats
Environmental context – Human maintained files, documents
System/application – Available only using application request
Intelligence/analytics – Indicators, anomaly, research, white/blacklist
Real-Time
Machine Data
On-Premises
Private Cloud
Public
Cloud
Storage
Online
Shopping Cart
Telecoms
Desktops
Security
Web
Services
Networks
Containers
Web
Clickstreams
RFID
Smartphones
and Devices
Servers
Messaging
GPS
Location
Packaged
Applications
Custom
Applications
Online
Services
DatabasesCall Detail
Records
Energy Meters
Firewall
Intrusion
Prevention
Copyright © - ECS & Splunk 2018
11. EXAMPLE OF ADVANCED THREAT ACTIVITIES
.pdf executes & unpacks malware
overwriting and running “allowed” programs
Threat
Intelligence
Auth - User Roles
Host
Activity/Security
Network
Activity/Security
Transaction Gain Access
to System
Create Additional
Environment
Conduct
Business
Svchost.exeCalc.exe
Attacker hacks website.
Steals .pdf files
Web Portal
Attacker
creates malware,
embed in .pdf
Read email, open attachment
Emails
to the target EMAIL
HTTP (web) session to
command & control server
Remote control,
Steal data,
Persist in company,
Rent as botnet
WEB
Copyright © - ECS & Splunk 2018
12. EXAMPLE OF ADVANCED THREAT ACTIVITIES
.pdf executes & unpacks malware
overwriting and running “allowed” programs
Threat
Intelligence
Auth - User Roles
Host
Activity/Security
Network
Activity/Security
Transaction Gain Access
to System
Create Additional
Environment
Conduct
Business
Svchost.exeCalc.exe
Attacker hacks website.
Steals .pdf files
Web Portal
Attacker
creates malware,
embed in .pdf
Read email, open attachment
Emails
to the target EMAIL
HTTP (web) session to
command & control server
Remote control,
Steal data,
Persist in company,
Rent as botnet
WEB
Intrusion
Detection
Credit card
transmitted
Endpoint
Security
Hacker tool
found
Windows
Authentication
Admin account
used
Copyright © - ECS & Splunk 2018
13. CONNECT THE “DATA-DOTS”
TO SEE THE WHOLE STORY
Persist, Repeat
Attacker, know relay/C2 sites, infected sites, IOC,
attack/campaign intent and attribution
Where they went to, who talked to whom, attack transmitted,
abnormal traffic, malware download
What process is running (malicious, abnormal, etc.)
Process owner, registry mods, attack/malware artifacts,
patching level, attack susceptibility
Access level, privileged users, likelihood of infection,
where they might be in kill chain
• Third-party Threat Intel
• Open source blacklist
• Internal threat intelligence
• Firewall
• IDS / IPS
• Vulnerability scanners
• Web Proxy
• NetFlow
• Network
• Endpoint (AV/IPS/FW)
• Malware detection
• PCLM
• DHCP
• OS logs
• Patching
• Active Directory
• LDAP
• CMDB
• Operating System
• Database
• VPN, AAA, SSO
Delivery, Exploit
Installation
Gain Trusted
Access
Upgrade (escalate)
Lateral Movement
Data Gathering Exfiltration Persist, Repeat
Threat
Intelligence
Auth - User Roles
Host
Activity/Security
Network
Activity/Security
Copyright © - ECS & Splunk 2018
14. CONNECT THE “DATA-DOTS”
TO SEE THE WHOLE STORY
phishing
Download
from
infected site
1
2
5
6
7
8
3
4
Threat Intelligence Data
Host or ETDR Data
Web or Firewall Data
Threat
Intelligence
Data
Identity Data
Threat
Intelligence
Auth - User Roles
Host
Activity/Security
Network
Activity/Security
Delivery Exploitation & Installation Command & Control Accomplish Mission
EMAIL WEB EMAIL WEB
Copyright © - ECS & Splunk 2018
15. Security Information & Event
Management (SIEM)
Software products and services combine
security information management (SIM)
and security event management (SEM).
They provide real-time analysis of security alerts
generated by network hardware and applications.
Source: Wikipedia & Gartner
Copyright © - ECS 2018
16. SIEM USE CASES
Security &
Compliance
Reporting
Real-time
Monitoring of
Known Threats
Detecting
Unknown
Threats
Fraud
Detection
Insider
Threat
Incident
Investigations
& Forensics
Copyright © - ECS & Splunk 2018
17. SIEM EVOLUTION
Term Initially
Coined in 2005
by Gartner
v1.0 Ticketing &
Workflow
Integrations
v1.5 Risk Based
Analysis &
“Intelligence”
v2.0 “Next-Gen SIEM”v3.0
Initial Rule Sets
& Event Queues
Environment Awareness
& Correlation
Searches
Risk Management
& Threat Data
Intelligence
Machine Learning
& Orchestration
Copyright © - ECS 2018
19. SIEM COMPONENT PARTS
RULES
Correlation Searches,
Thresholds & Grouping
CONTEXT
Organisational
Awareness & Impact
Assessment
FRAMEWORKS
Scalable Functionality
& User Empowerment
INTEGRATION
Data Compatibility,
Extensibility &
Workflow Management
Copyright © - ECS 2018
21. A
B
C
D
INTEGRATION
Maximize cross-silo visibility by on-boarding ALL data sources.
Automate repetitive tasks and setup orchestration for the rest.
PREPARATION
Understand your project’s input and output requirements.
Champion the project and identify project dependencies.
SUCCESS CRITERIA
Identify the problem(s) you’re trying to solve.
Document the risks/threats and the controls/mitigations.
EMBEDDING
Position SIEM project as part of transformative change.
Enable and engage SecOps to own and evolve platform.
SUCCESSFUL SIEM
Copyright © - ECS 2018
23. WHO AM I?
SAM FARMER
•Alumnus of Edinburgh Napier
•Security Operations Specialist at ECS
• Security Operations SME
• Security Monitoring (SOC)
• SIEM Implementation
• Threat Hunter
Copyright © - ECS 2018
29. STACKING
sourcetype="stream:http"
| bin span=1d _time
| stats count as curr_count by _time
| appendcols [search index=botsv1 sourcetype="stream:http" | stats count as
total_count]
| eval avg_count = round(total_count/30,0)
| stats list(avg_count) as "Average Count", list(total_count) as "Total Count",
values(curr_count) as curr_count
Copyright © - ECS 2018
30. STANDARD DEVIATION
| bin span=3m _time
| stats count as curr_count by _time
| streamstats window=1 current=false avg(curr_count) as prev_count
| eval growth=curr_count-prev_count
| stats avg(curr_count) as average stdev(curr_count) as std_dev latest(curr_count) as latest_vol
latest(_time) as lt count(eval(curr_count>150)) as qualifying count as tots
| eval conf_int=average+(3.69*(std_dev/sqrt(tots)))
| where ((latest_vol>150 AND qualifying=1 AND relative_time(now(), "-4m")<lt) OR
(latest_vol>conf_int AND qualifying>=8))
| rename average as "Average" std_dev as "Standard Deviation" latest_vol as "Latest Volume" lt as
"Latest Time" qualifying as Qualifying tots as Total conf_int as "Confidence Interval"
| convert ctime("Latest Time") timeformat="%H:%M:%S %d/%m/%y"
Copyright © - ECS 2018
31. SPLUNK USER GROUP - EDINBURGH
• When:
• TBA (Register for Invite)
• Where:
• Edinburgh Napier University, 10 Colinton
Road, Edinburgh, EH10 5DT
• Register:
https://usergroups.splunk.com/group/spl
unk-user-group-edinburgh.html
Copyright © - ECS 2018
Editor's Notes Short Bio:
Harry McLaren is a Senior Consultant at ECS and is responsible for service delivery, technical leadership and people development in the rapidly growing Splunk consulting practice and is responsible for growing our team of talented Splunk Consultants. ECS, a specialist in enterprise IT services, has an award-winning IT security capability which is focused on Cybersecurity Operations Centres and IT security consulting.
1min Few Security based use cases you have leverage big data platforms for, but how?
1mins SIEM evolution and the (often fallacy) that is ‘next-gen’ SIEM. “Next-gen” shouldn’t even be a term as your security operational capability to grow organically and the tools should be able to keep up.
How a platform which can grow as your security maturity and technical ability also grows (not limited to only “out-of-the-box features”).
2mins Building full featured SIEMs is hard.
Many try, many fail.
Big data platforms only provide access to (hopefully) easy to search data.
Most end up as very basic rule engines similar in function to a distributed IDS (NIDS or HIDS).
2mins Rules
Threshold Based
Anomaly/Behaviour Based
Boolean Based
Context
Asset & Identity Awareness
Risk Profiling/Analytics
Approved Types of Activity vs Not
Frameworks
Scalability (Volume, Complexity)
User Empowerment (without being a platform expert)
Expansion and development of custom use cases.
Integration
Data Source Compatibility (Schema vs Write one, read multiple ways).
Workflow Integration & Centralised Investigation
Orchestration
3mins
Example high-level architecture of a SIEM platform.
Lots of components working together.
Inputs, procedures and outputs are covered.
Five frameworks mentioned covered in more detail.
Not going to talk all the way through each one, purpose is to show the types of frameworks required and illustrate the contents of them.
2mins Understand the reasons for the project, use cases, motivations and what constraints might apply.
Prepare, prepare, prepare. Ensure you have scoped all required inputs, outputs and the level of dependencies between them.
Integrate everything! Not just the data sources, but workflow, automation and orchestration.
SIEM can be very powerful tools, however if the team which is going to own it/use it doesn’t know how, it’ll go to waste. SecOps teams should be a the forefront of exploring the data, hunting and defining their own use cases.
2mins Image: https://www.techiexpert.com/difference-data-science-machine-learning/ Image: ThreatConnect https://www.threatconnect.com/blog/threatconnect-announces-context-enriched-intelligence/ Image: https://sqrrl.com/cyber-threat-hunting-1-intro/ Registration: https://usergroups.splunk.com/group/splunk-user-group-edinburgh.html
LinkedIn Group: https://www.linkedin.com/groups/12013212
1min