Log and logging overview A brief on Incident response and forensics Logs in incident investigations Just what is log forensics? Conclusions and call to action!

1. Using Logs for Breach Investigations and Incident Response Dr. Anton Chuvakin @anton_chuvakin SecurityWarrior LLC www.securitywarriorconsulting.com BrightTalk Forensics Summit 2011

2. Logs for Breach Investigations A few thoughts to start us off … All attackers leave traces. Period!  It is just that you don’t always knowwhat and where And almost never knowwhy… Logs are the place to look, first

3. Outline Log and logging overview A brief on Incident response and forensics Logs in incident investigations Just what is log forensics? Conclusions and call to action!

5. Routers/switches

6. Intrusion detection

7. Servers, desktops, mainframes

8. Business applications

9. Databases

10. Anti-virus

11. VPNs

12. Audit logs

13. Transaction logs

14. Intrusion logs

15. Connection logs

16. System performance records

17. User activity logs

19. Logs at Various Stage of Incident Response Preparation: verify controls, collect normal usage data, baseline, etc Identification: detect an incident, confirm incident, etc <- really? Containment: scope the damage, learn what else is “lost”, what else the attacker visited/tried, etc Eradication: preserving logs for the future, etc Recovery: confirming the restoration, etc Follow-Up: logs for “peaceful” purposes (training, etc) as well as preventing the recurrence

20. Various Logs for Incident Response

21. Firewall Logs in Incident Response Proof of Connectivity Proof of NO Connectivity Scans Malware: Worms, Spyware Compromised Systems Misconfigured Systems Unauthorized Access and Access Attempts

22. Example: Firewall Logs in Place of Netflow Why Look at Firewall Logs During Incident Investigation? 1990-2001 – to see what external threats got blocked (in, failure) 2002-2011+ – to see what internal system got connected (out, success) Thus, firewall logs is “poor man’s” netflow…

23. Recommendations Log denied connections Log allowed connection If cannot log all allowed connections, then log outbound allowed connections Watch for firewall rule changes

24. NIDS Logs in Incident Response Attack, Intrusion and Compromise Detection Malware Detection: Worms, Viruses, Spyware, etc Network Abuses and Policy Violations Unauthorized Access and Access Attempts Recon Activity

25. Just A Thought… Q: How many serious incidents were discovered by an IDS/IPS? A: Answers I’ve heard.. 0, 0, 0, 0 with an average of 0 

26. Server Logs in Incident Response Confirmed Access by an Intruder Service Crashes and Restarts Reboots Password, Trust and Other Account Changes System Configuration Changes A World of Other Things 

27. Example: “Irrelevant, You Say” Using disk failures for IDS  What is really there? Is this OUR server? Well … “Detection by catastrophe” Is CNN you IDS?

28. Recommendations On a typical Unix system log: Syslog = usually defaults are sensible Select file access = via kernel logging Select process execution = via kernel logging On a typical Windows server: Authentication = failure/success Account changes = failure/success Privilege use = failure/success Others as needed (see MS Recommended Audit Policy)

29. Database Logs in Incident Response Database and Schema Modifications Data and Object Modifications User and Privileged User Access Database Backups (!) Failures, Crashes and Restarts LOOK AT LOGS! 

30. Proxy Logs in Incident Response Internet access patterns Policy violations Accidental/malicious information disclosure Malware: spyware, trojans, etc Outbound web-hacking

31. Example: Proxy Logs vs Uploads How? Search for POST requests AND specific document content-types (e.g. msword, powerpoint, etc.) What? Look for a uploads to unusual sites (especially with unresolved IPs), web mail, or for sensitive document names Especially, look for uploads to unusual ports

32. Antivirus Logs in Incident Response Virus Detection and Clean-up (or lack thereof!) Failed and Successful Antivirus Signature Updates Other Protection Failures and Issues Antivirus Software Crashes and Terminations

33. Log Forensics

34. So, What is “Log Forensics” Log analysis is trying to make sense of system and network logs “Computer forensics is application of the scientific method to digital media in order to establish factual information for judicial review.” So…. Log Forensics = trying to make sense of system and network logs + in order to establish factual information for judicial review

35. How Logs Help… Sometimes Logs help to figure out who, where, when, how, what, etc. but … Who as a person or a system? Is where spoofed? When? In what time zone? How? More like ‘how’d you think’… What happened or what got recorded?

36. Who? Just who is 10.1.1.2? Do you know him? Is jsmith.example.com a who? Is JSMITH at JSMITHXAMPLE? Is JSMITHauthenticated by an RSA token at JSMITHXAMPLE and also logged to another system as “jsmith” a who? Is JSMITHauthenticated by a fingerprint reader at JSMITHXAMPLE a who?

37. Solving "Who?” Q: How do you know who? did something in a forensically sound manner? A: Offline evidence: we (or camera) see that he did it

38. When? Got timestamp?  - challenges to log timing! Completely false timestamp in logs It’s always 5PM somewhere: time zone Are you in drift? NTP might be Syslog forwarder delays Systems with two timestamps! It got logged at 5:17AM. When did it happen? Yes, even, 24 vs. 12 time Lying NTP? Is it possible?

39. Solving “When?” Q: How do you know when something occurred in a forensically sound manner? A: NTP sync religiously, make note of time zones, know the logging systems and keep logs secure to prevent timestamp modification

40. Where? The attack came from X.1.1.2, which belongs to Guanjou Internet Alliance, Beijing China The stolen data then went to X.2.2.1 which belongs to PakNet ISP in Karachi, Pakistan Result: Romanian hackers attack!  or Result: it was a guy from the office on the 3rd floor? or …

41. Solving “Where?” Q: How do you know wheredid something happen in a forensically sound manner? A: Only offline evidence can help: we (or camera) see that he did it! For remote access: unlikely to be available

42. Conclusions Turn ON Logging!!! Make sure logs are there when you need them When going into the incident-induced panic think ‘its all logged somewhere – we just need to dig it out’ Review logs to know your environment! Logs for Forensics Logs can tell you things, but are they “good evidence”? Logs become evidence only if precautions are taken

43. Questions? Dr. Anton Chuvakin SecurityWarrior LLC Email:anton@chuvakin.org Site:http://www.chuvakin.org Blog:http://www.securitywarrior.org Twitter:@anton_chuvakin Consulting:http://www.securitywarriorconsulting.com

44. More Resources Blog: www.securitywarrior.org SANS SEC434 Log Management Class Podcast: look for “LogChat” on iTunes Slides: http://www.slideshare.net/anton_chuvakin Papers: www.info-secure.org and http://www.docstoc.com/profile/anton1chuvakin Consulting: http://www.securitywarriorconsulting.com/

45. More on Anton Consultant: http://www.securitywarriorconsulting.com Book author: “Security Warrior”, “PCI Compliance”, “Information Security Management Handbook”, “Know Your Enemy II”, “Hacker’s Challenge 3”, etc Conference speaker: SANS, FIRST, GFIRST, ISSA, CSI, RSA, Interop, many, many others worldwide Standard developer: CEE, CVSS, OVAL, etc Community role: SANS, Honeynet Project, WASC, CSI, ISSA, OSSTMM, InfraGard, ISSA, others Past roles: Researcher, Security Analyst, Strategist, Evangelist, Product Manager