This talk discusses practical approaches to OT incident response, that will leverage the people, processes, tools, and relationships you most likely already have.

1. A proactive approach
to OT incident
response
Chris Sistrunk, PE
Technical Leader, ICS/OT

2. Who am I?
Chris Sistrunk, PE
Technical Leader
Mandiant ICS/OT Security Consulting
chrissistrunk@google.com
Mandiant (part of Google Cloud)
• Technical Leader, ICS/OT
• 9.5 years ICS/OT Security Consulting
Entergy
• Senior Electrical Engineer
• T&D SCADA, Substation Automation, Distribution Design
• 11+ years
BMS
Embedded
Devices
IoT
IIoT
OT
ICS, SCADA,
& DCS

3. Objectives
Similarities of IR in IT and OT
Unique considerations for IR in OT
Proactive Steps

4. Incident Response Lifecycle
NIST SP 800-61 Revision 2: Computer Security Incident Handling Guide

5. Theory of 99
Most threat activity happens in Windows and Linux Systems
99
%
1%
99% of systems compromised will be IT systems
99% of malware will be IT malware
99% of forensics will be performed on IT systems
99% of detection opportunities will be on IT systems
99% of “intrusion dwell time” will be on IT systems

6. OT Attack Vectors and Impact
• Value of detecting OT attacks
in Intermediary Systems
• Often a significant overlap
across tactics, techniques, and
procedures (TTPs) used by
threat actors targeting both IT
and OT networks
“Funnel of Opportunity”

7. Theory of 99: Common TTPs

8. Unique Considerations for IR in OT
Environmental, Health &
Safety
EHS has their own incident
response and management plan
when an accident or injury happens
in the workplace. If a cyber
compromise triggers a response
from these teams the plans will likely
need some kind of integration
Operations & Engineering
Probable that operation or
engineering teams are the first ones
to notice when there is an anomaly.
These same teams have deep
understanding of the control
systems and the process, they will be
keeping parts of the team
throughout the process
Third Party Support
External support from vendors,
OEMs, and engineering contractors
could be vital to ensure the right
tools and resources are available to
investigate and remediate. There
may be service contracts that
preclude ‘hands on’ unauthorized
persons

9. DFIR Framework for OT Systems
Preparation Phase
• OT Device and Tool Identification
• OEM Collaboration
• Data Identification and Collection

10. Identify what you have
Asset management
Asset management helps ensure
that security and engineering
teams know what devices exist in
their environments. Adequate IR
plans and playbooks for OT depend
on having the correct tools for
investigation and the restoration
process may require access to proper
critical spares.
Network architecture
Evaluate OT Network Segmentation
to support securing OT systems by
splitting the network into smaller
subnetworks, isolating network
traffic, lessening the attack
surface, and obstructing lateral
movement. Segmentation may also
provide capability to isolate
compromise in IT before it spreads
to OT.
Vulnerability & patch
management
Vulnerability & Patch Management
within your environment help your
organization reduce its security risk.
With a reduced ability to 'patch
everything' in OT, knowing what is
vulnerable, patching where possible, and
mitigating inherent risk is essential to
minimize attack surface.

11. Spreadsheet of
DooooOOOOOOOooooom
No excuse for an out of date, incomplete, unreadable asset inventory
Plenty of free and paid ICS/OT tools out there
MACHINE READABLE!!!

12. Did you know?
• Most ICS protocols are insecure by design
• Lack authentication and encryption
• Don’t usually have CVEs assigned and not usually flagged in
vulnerability software / scans
• Exceptions
• Modbus CVE-2017-6034 & CVE-2017-6032
• KNX Protocol CVE-2023-4346

13. Develop Capabilities
Visibility
Collecting telemetry data from OT
environments requires different
strategies than in traditional IT.
Network Security Monitoring
provides visibility where endpoint
agents are not practical. Process
data and device resource data will
be useful in the event of a
compromise.
Threat Hunting
Enable your threat hunting teams
with actionable threat intelligence
and up to date vulnerability alerts.
Leverage NSM for anomaly
detection and undocumented
vulnerabilities alerts operators and
analysts of potential security issues
and enables defense from network
intrusions and subsequent disasters.
Response
When threats or operational
anomalies are detected, monitoring
tools reduce forensic efforts and
speed response time by providing
the contextual information IR teams
need to investigate and remediate
risks and minimize the potential
impacts of an attack or
operational issue.

14. Threat hunting and IR aren’t possible without
visibility

15. The right tools for the job
Software
Some of the software tools used for
DFIR in controls systems are often
the same tools engineering teams
use to configure and program the
devices and are often not owned by
the asset owner. Some tools may
only be accessible to factory
engineers and in rare cases need to
be purpose built for the task.
Unique Connections
Many devices use proprietary
communications protocols and likely
have ports that use non-standard
pinouts. Not all serial cables are
created equal and in some cases
even devices with similar model
numbers have differences based on
hardware revisions.
System Parameters
Data from industrial processors can
often be logged in historians and
can be extremely useful when
investigating a compromise. CPU
usage, memory usage, logic scan
times, and other parameters may
indicate when something in the code
changed.

18. Want to contribute to OT DFIR?
http://otdfir.com

19. Have a plan, test it, & improve it
Incident response program
Incident Response Programs
outline an organization’s
procedures, steps, roles, and
responsibilities in the event of an
incident and helps your organization
before, during, and after a confirmed
or suspected security incident.
Plans & playbooks
OT Cybersecurity incidents are a
business continuity problem.
Effective plans and playbooks help
technical responders follow critical
steps in the process and help
executives make effective
decisions. OT IR may require
support from groups not normally
involved in Enterprise IR (EH&S,
Operators, Engineering, etc.)
Tabletop exercises
Tabletop Exercises evaluate your
organization’s cyber crisis
processes, tools, and proficiency in
responding to incidents and
provide an opportunity to
continually improve upon the
effectiveness of the program, plan,
and playbooks.

20. IR Plan?

21. OT Use Cases & Playbooks
• Commodity Malware in OT
• Conficker, Ramnit, Mariposa, Wannacry
• OT Credential Compromise
• Ukraine 1 attack, PLC ladder logic change (Aurora)
• Destructive Attack
• KillDisk, overwriting firmware (Ukraine 2015)
• Wiper malware (NotPetya) or ransomware spreading to OT
• Indirect attack that impacts enterprise resource planning / critical apps that causes OT to
shut down
• ICS Protocol Attack
• Stuxnet, Industroyer (Ukraine 2016 & 2022), Triton
Remediation for each play:
Sever IT / OT, manual mode, restore backups, paper, reset passwords,
etc.

22. ICS4ICS
• FEMA Incident Command System
• Scalable to handle any incident,
common roles and language
• Local to national response
• Hurricane response, fires, water
main breaks, pandemics….and now
cybersecurity incidents for
Industrial Control Systems
• https://www.ics4ics.org
• Almost 1000 global members
signed up for ICS4ICS email list
• Over 20 ICS4ICS Credentialed Type
4 Incident Commanders

24. Training & awareness
Close the
Skills Gap
Cyber
Awareness
in OT
Lessons
from Safety
Culture
Evolving
Landscapes

25. Storytime

26. Recent IR Examples
• Infected Manufacturing Line HMIs
• Wannacry > Line was shut down (infection from TightVNC to Internet)
• Multiple commodity malware > Lines still operated (infected USB long ago)
• Both instances, leveraged existing OT network sensor
• Ransomware on Electric IT, worked w/ Plant OT SMEs and Vendor
• Infected ICS engineering laptop
• so old, the power supply died as we were doing the analysis
• Infected Contractor laptop

27. Near Miss!
27

28. Open Discussion / Lessons Learned

29. TL;DR

30. Best Conversation Starter EVER
30
IT and OT folks, get together and
talk about cybersecurity issues

31. Recommendations for OT IR
• Collaborate IT security teams, OT teams, and OEMs
• Identify employees with knowledge of the process and your OT systems
Collaborate
• Include OT security in IR Plan and/or engineering procedures
• Create and maintain inventories of OT devices, tools, and protocols
Plan
• Develop awareness training for OT security and incident response
• Perform an annual OT TTX, including collecting logs from OT
Practice

32. Be proactive
You can do…

33. OT IR Resources
• https://doi.org/10.6028/NIST.SP.800-61r2
• https://doi.org/10.6028/NIST.SP.800-82r3 Section 6.4 Respond (RS)
• https://www.cisa.gov/sites/default/files/2023-
01/national_cyber_incident_response_plan.pdf
• https://www.publicpower.org/system/files/documents/Public-Power-
Cyber-Incident-Response-Playbook.pdf
• https://www.cisa.gov/topics/partnerships-and-collaboration/joint-
cyber-defense-collaborative JCDC Energy & Water Plans coming soon

34. OT IR Resources
• https://www.mandiant.com/resources/blog/Mandiant-approach-to-
operational-technology-security
• https://www.mandiant.com/resources/blog/mandiant-dfir-
framework-ot
• https://www.ics4ics.org
• http://otdfir.com (Community DFIR for PLCs project)
• https://github.com/mandiant/rpdebug_qnx
• https://www.slideshare.net/chrissistrunk/black-hat-usa-2022-arsenal-
labs-vehicle-control-systems-red-vs-blue
• https://github.com/mandiant/ics_mem_collect

35. thank you
Chris Sistrunk, PE
chrissistrunk@google.co
m