3. Examining Initial Data
• Look at original aler
t
• You may notice more than the person who
reported it di
d
• Ask about other detection systems and review
what they recorde
d
• Network administrators may not think like
investigator
s
• Gather context for the detection event
4. Preliminary Evidence
• Determine what sources of preliminary evidence
may hel
p
• Decide which sources you will actually us
e
• Collect and review the evidenc
e
• Identify sources that are easy to analyze, and
that quickly provide initial answers
6. Independent Sources
• Firewall logs don't depend on registry keys, etc
.
• Multiple independent evidence sources lead to
more reliable conclusion
s
• It's dif
fi
cult for an attacker to remove or modify
evidence from all source
s
• Less likely that a routine process would
overwrite or discard all evidenc
e
• Cross-check information, like date and time
7. Review
• Attacker may be causing more damag
e
• Test a detection method on one system, or a
small date range of log entrie
s
• Make sure your detection method is fast and
effective
11. Data Loss Scenario
• Large online retaile
r
• You work in IT security departmen
t
• Customers are complaining about spam after
becoming a new customer
12. Finding Something Concrete
• Anecdotal customer complaints are
inconclusiv
e
• Option
s
• Work with customers and review their emai
l
• Reliability and privacy problem
s
• Create fake customer accounts with unique
email addresses
13. Fake Accounts
• Use 64-character random username
s
• Unlikely that spammers would guess the
username
s
• Monitor those accounts
14. Preliminary Evidence
• Assuming customer data is being lost someho
w
• Find where customer data is and how it is
manage
d
• One internal database on production serve
r
• One external database at a third-party marketing
fi
rm
18. Theories & Simple Tests
• Inside
r
• No easy way to tes
t
• Modi
fi
ed code on website to capture email
addresse
s
• Enter some fake accounts directly into
database, bypassing the web for
m
• Someone copying backup tape
s
• Add some fake accounts to the backups
19. Three Sets of Fake Accounts
1. Made through the web for
m
2. Entered directly into the database,
bypassing the web for
m
3. Added only to the backups
20. Two Weeks Later
• Spam comes to the
fi
rst set of fake account
s
• And to the accounts manually entered into the
databas
e
• Suggests the website is not part of the
proble
m
• No spam to the accounts on the backup tap
e
• Backups aren't the source of data loss
21. New Theories
• Direct access to the databas
e
• Malware on the database serve
r
• Accessing it over the network
22. Monitoring Queries
• Network-level packet capture
s
• Expensive, powerful system require
d
• If queries are encrypted, or malware is
obfuscating them, it may be hard to decode
the traf
fi
c
• Database-level query monitoring and loggin
g
• Most ef
fi
cient and reliable technique
23. Next Steps
• Create a few more fake account
s
• Talk to database and application administrator
s
• To
fi
nd out where data is store
d
• Scan through logs to see what is "normal
"
• No queries or stored procedures perform a bulk
export of email addresses on a daily basis
24. Two Weeks Later
• New accounts get spa
m
• Retrieve query logs from time accounts created
to spam tim
e
• For the
fi
eld "custemail
"
• A single query is foun
d
• SELECT custemail FROM custpro
fi
le WHERE
signupdate >= "2014-02-16"
25. Query Details
• Feb 17, 2014 at 11:42 am GM
T
• Originated from IP in graphics arts dept
.
• Query used an database administrator's
username from the IT dept
.
• Interview reveals that graphics arts dept. has no
direct interaction with customers, only outside
vendors
29. Results from Workstation
• Examine images, focusing on actions at the time
of the quer
y
• Malware found on workstatio
n
• Persistent, provides remote shell, remote
graphical interface, ability to launch and
terminate processe
s
• Connects to a foreign I
P
• Has been installed for two year
s
• Cannot determine how system was originally
compromised
31. Scoping Gone Wrong
• After complaints from customer
s
• Search every computer in the company for
unique strings in the customer data
,
• And
fi
les large enough to include all the
customer records
32. Problems
• No evidence that the stolen data is stored on
company server
s
• No evidence that the data is all being stolen at
once in a large
fi
l
e
• And even so, it would probably be
compresse
d
• Large amount of effort; low chance of success
33. Another Unwise Path
• Focus on insider
s
• Who had access and knowledge to steal
the customer dat
a
• Compile pro
fi
les of numerous employee
s
• Review personnel
fi
le
s
• Background checks, surveillance software
capturing keystrokes and screen image
s
• Video surveillance installed
34. Problems
• Leads to a "witch hunt
"
• Invades privacy of employee
s
• Large effort, small chance of success
35. Another Unwise Path
• Because the data resides on the database
serve
r
• Image and analyze RAM from the database
server to hunt for malwar
e
• Because the hard drives are massive and too
large to investigate easily
36. Problems
• Once again, they have jumped to a conclusio
n
• And ignored other possibilitie
s
• Large effort, low chance of success
38. Funds Transfer
• Bank called the CEO--they blocked an ACH
transfer of $183,642.7
3
• To an account that was never used befor
e
• Flagged by their fraud prevention syste
m
• Transfer from CFO's account, but he says he
never authorized it
40. Preliminary Evidence
• Firewal
l
• Two weeks of log
s
• Examine this
fi
rs
t
• Should you examine CFO's laptop computer
?
• Live response, RAM, hard driv
e
• But maybe other computers are involved
41. Firewall Logs
• Look near the time the unauthorized transfer
occurred (4:37 pm
)
• See who logged in prior to tha
t
• Two computers logged in via HTTPS, making a
number of connections between 4:10 pm and
4:48 p
m
• From two IPs -- one is CFO's, other not
immediately recognized
42. Two Immediate Tasks
• Gather complete forensic evidence from CFO's
compute
r
• Live response, RAM, and hard dis
k
• Because evidence is being lost as time
passe
s
• Track down the other IP address and decide
what action is appropriate
43. DHCP Logs
• Search for the time in questio
n
• Get MAC address from DHCP log
s
• It's the MAC of the CFO's laptop (wireless card
)
• So there's only one computer involved
50. Scoping Gone Wrong
• There are no recent antivirus or IDS alert
s
• So you believe the security issue must be at the
ban
k
• Tell the bank to
fi
nd the attacker and put them in
jail
51. Problems
• No attempt to validate the bank's original dat
a
• Company assumes that existing network
security measures would detect a problem, if
there was on
e
• Assumption that a third-party can help yo
u
• While you wait, data on company systems is
lost
52. Another Unwise Path
• CEO believes that security measures are in
place to prevent malware, so the CFO must have
initiated the transfe
r
• The CEO wants you to investigate the CFO and
avoid tipping him (or her) off
53. Problem
• No security measures are perfect, not even two-
factor authenticatio
n
• Also, that sort of investigation is outside your
expertise, and should be referred to an outside
contractor
56. Purpose of Live Collection
• Preserve volatile evidence that will further the
investigatio
n
• Also collect log
fi
les and
fi
le listing
s
• Get answers quickl
y
• Minimize changes to the syste
m
• Avoid disrupting business, causing crashes, or
destroying evidence
59. Altering the Evidence
• All live response changes the system
• Purists don't like i
t
• But the alternative is to lose all volatile data
and get only a disk imag
e
• You can minimize changes, but not eliminate
them
60. Selecting a Live Response
Tool
• Homegrown Microsoft DOS batch script (or
bash
)
• Perl-based scrip
t
• There are specialized live-response products,
free and commercial
62. Factors to Consider
• Is the tool generally accepted in the forensic
community
?
• Does the solution address the common
operating systems in your environment
?
• Tools that use OS commands should contain
known good copies of those commands, not
trust the local commands on the suspect
system
63. Factors to Consider
• Does the solution collect data that is important
to have, in your environment
?
• How long does a collection take
?
• Recommended: less than an hour per syste
m
• Is the system con
fi
gurable
?
• Is the output easily reviewed and understood?
64. What to Collect
• Current running state of the syste
m
• Network connection
s
• Running Processe
s
• What happened in the pas
t
• File listings, system log
s
• Usually a higher priority
65. The Deep End
• Some organizations always collect entire RAM
contents, or hard disk image
s
• Don't collect data that you can't effectively
use or understan
d
• Collect data you can really use to quickly
determine the impact of the incident
69. Complete RAM Capture
• Requires specialized tools to collect and
interpre
t
• Not part of Live Respons
e
• Sometimes needed on carefully chosen systems
70. Collection Best Practices
• Practice on a test system
fi
rs
t
• Learn how fast the process is, and how large
the output i
s
• Practice handling problem
s
• Broken USB port or NI
C
• Locked screen
71. Caution: Malware
• The system you are examining may be infected
with malwar
e
• Any media you connect may become infecte
d
• Any credentials you use may be compromised
80. Horror Stories:
IR Procedures
• Copy live response toolkit to affected systems,
save collected data back to that same system
including full RAM dump, several gigabytes in
siz
e
• Remotely log in with domain administrator
account, run netstat & Task Manage
r
• Pull out the plug, image the hard drive
81. Good Methods of
Live Response
• Network share on a dedicated
fi
le serve
r
• Not part of any domai
n
• Used throwaway credentials not used for any
other purpos
e
• Two folder
s
• Read-only containing the live response toolki
t
• Writeable for output from live response toolkit
83. Live Response Tips
• Air-gap for evidence serve
r
• Logging and auditing access to evidence serve
r
• Automate process for consistenc
y
• Live Response software must run as Local
Administrator/root
84. Media
• Some computers cannot connect external medi
a
• Hardware failure, con
fi
guration, etc
.
• Common options for running toolki
t
• CD-ROM, DVD, network shar
e
• Encrypted network streaming tool like
cryptcat or stunnel to send output to another
system
85. Unexpected OS
• Cannot run your normal live response toolki
t
• If you can't update or modify your toolkit to ru
n
• Perform manual live response
86. Unexpected OS
• Create a checklist of the automated steps in the
toolkit for a similar O
S
• Research command-line option
s
• Test them on a known clean system, if possibl
e
• Manually perform steps to collect evidence
87. Automation
• Decreases human erro
r
• Makes processes more consistent and faste
r
• Helps to prevent bad guys from gathering
intelligence about how you respond to incident
s
• Anything you do on the evidence system may
be sent to the bad guys
90. Three Main Options
• Use a prebuilt kit
• Like Mandiant Redline or Velocirapto
r
• Create your ow
n
• Use a hybrid of the two
91. Mandiant Redline
• Install the Redline MSI package on a trusted
workstatio
n
• Create the Redline collector on the trusted
syste
m
• The only step you take on a suspect system is
to run the stand-alone Redline collector batch
scrip
t
• Automatically saves data to the same location
you ran the script from
92. Do It Yourself
• Make your own live response toolki
t
• Decide what OS to suppor
t
• Windows has many versions, and big
differences between 32-bit and 64-bi
t
• Find tools that collect the information you want
93. Windows Built-in Tools
• Copy these
fi
les from a
clean Windows syste
m
• Also copy cmd.ex
e
• "Trusted binaries
"
• Don't trust
fi
les on the
evidence machine
94. Free Tools
• Use command-line
versions, not GUI
version
s
• Easier to scrip
t
• Less impac
t
• Rename every tool
so you can identify
it as something you
added to the syste
m
• Prepend "t_"
95. Other Data Items
• Prefetch informatio
n
• System restore point informatio
n
• Browser history, and mor
e
• Balance your needs with the impact the
collection has on the system
97. Scripting Tips
• Add logging and compute a checksum of
collected dat
a
• Be careful with
fi
le and directory name
s
• They may be long or include space
s
• Test your script extensively
• Built a test environment that resembles
your production system
s
• Watch for errors and unexpected results
98. Memory Collection
• Tools for a full memory dum
p
• AccessData FTK Imager Lit
e
• Mandiant Memoryz
e
• Monsools Windows Memory Toolki
t
• Belkasoft RAM Capturer
104. LINReS
• From Network Intelligence Indi
a
• Written for RedHat 3 and 4, not updated since
200
6
• Useful mainly as an example to guide you in
making a custom tool
105. Do It Yourself
• Really the only optio
n
• Make some script
s
• Mandiant uses Bourne shell scripts that make
scripts for various Unix/Linux version
s
• Requires constant maintenance
113. Memory Collection
• The memory device is handled differently in
every version of Unix, BSD, and Linu
x
• In earlier versions, you could just use dd to
collect RAM through the /dev/mem devic
e
• Direct access to memory is now blocked for
security reason
s
• Use LiME – Linux Memory Extractor (link Ch 7c)
114. Loadable Kernel Modules
• LiME is an LK
M
• Must be compiled for the exact kernel version
running on the target syste
m
• No ability to include checksums of the outpu
t
• You must do that yourself
115. Collection from BSD-Based
Kernels
• Use dc3dd or dc
fl
dd to capture contents of
/dev/me
m
• They are like dd but also include checksum
s
• In recent versions, there's no End Of File mark
in /dev/mem, so you must manually specify how
many bytes to capture
116. Collection from Apple OS X
• Memoryze for Mac (link Ch 7d
)
• Mac Memory Reader seems to be gone