The UNC School of Medicine suffered a security breach last summer that required notification of over 100,000 patients that their information had been exposed. This presentation will talk about the scope of damage that is caused by a breach of this magnitude and the many steps that are necessary for damage control and recovery.

1. After the BreachAfter the Breach
Dennis SchmidtDennis Schmidt
Director, Office of Information SystemsDirector, Office of Information Systems
HIPAA Security OfficerHIPAA Security Officer
UNC School of MedicineUNC School of Medicine

2. OMG, We have a breach!OMG, We have a breach!
 In late July, 2009, UNC Information Technology
employees discovered that a server which
contained sensitive information on 180,000
research subjects, including 114,000 Social
Security Numbers, had been the target of a
computer hack in 2007. The compromised
server was taken down and the data on the
server were removed.

3. Incident DiscoveryIncident Discovery
 OIS receives call from departmental serverOIS receives call from departmental server
admin reporting that a server would not rebootadmin reporting that a server would not reboot
after power failure.after power failure.
 OIS technician suspects virus and performs fullOIS technician suspects virus and performs full
virus scan on machine. Virus detected.virus scan on machine. Virus detected.
 Technician is told by department that server mayTechnician is told by department that server may
contain sensitive information.contain sensitive information.
 Server turned over to OIS Information SecurityServer turned over to OIS Information Security
for forensic analysis.for forensic analysis.

4. Forensic AnalysisForensic Analysis
-- A Long, Painful Process ---- A Long, Painful Process --
 Verification – Verify the incident occurred
 Interview the SysAdmins and other users involved
 Examine system and application logs (Snort, Tipping
Point, etc.)
 Check volatile information using forensic tools
 System Description
 Physical observation, forensic tools
 Interview SysAdmins and users, determine use
 Hardware and software system characteristics
 Hard disk geometry

5. Forensic Analysis (cont.)Forensic Analysis (cont.)
 Evidence Collection
 All available computer information (volatile and
non-volatile) is collected and transferred to external
media or forensic workstation to perform analysis
tasks.
 Data must be collected in order of volatility and data
integrity safeguarded by hash signature, MD5

6. Forensic Analysis (cont.)Forensic Analysis (cont.)
 Timeline Creation & Analysis – Use time-stamps
from internal and external sources to correlate
into timeline that traces back the system activity.
 Media Analysis – Thorough examination of the
media layers (physical, data, metadata, file system
and file name) searching for evidence.

7. Forensic Analysis (cont.)Forensic Analysis (cont.)
 Data Recovery – extracting unallocated data in
order to recover any deleted files. File
fragments could represent a critical piece of
information relevant to the case
 String Search – searching for specific strings or
keywords contained inside files to reveal useful
information relevant to the case.
 Reporting -- detailed report(s) of the forensic
process explaining the evidence found, together
with the techniques and methodology used.

8. Houston, We have a problem!Houston, We have a problem!
 Virus/worm/trojan infection for 2 yearsVirus/worm/trojan infection for 2 years
 26 files containing over 500,000 records26 files containing over 500,000 records
 180,000 unique research subjects180,000 unique research subjects
 114,000 Social Security Numbers114,000 Social Security Numbers

9. Qualys Scan ResultsQualys Scan Results

10. But, did they get anything?But, did they get anything?
 When did compromise occur? Is it still active?When did compromise occur? Is it still active?
 When were the sensitive files put on theWhen were the sensitive files put on the
machine? When were they last accessed?machine? When were they last accessed?
 Was it during the compromise window?Was it during the compromise window?
 Is there any corroborating evidence on theIs there any corroborating evidence on the
network of file downloads from the server?network of file downloads from the server?

11. The Antivirus DilemmaThe Antivirus Dilemma
 Full virus scan changes the last accessed time onFull virus scan changes the last accessed time on
everyevery file.file.
 It now becomes impossible to determine if theIt now becomes impossible to determine if the
malware actually accessed specific files.malware actually accessed specific files.
 e.g., If compromise occurred one week ago, and laste.g., If compromise occurred one week ago, and last
access of sensitive file was one month ago, you knowaccess of sensitive file was one month ago, you know
the data was not likely accessed by the malware.the data was not likely accessed by the malware.
 If virus scan was done yesterday, you no longerIf virus scan was done yesterday, you no longer
know when the file was last accessed.know when the file was last accessed.

12. No Smoking GunNo Smoking Gun
 There was no way to prove that data on theThere was no way to prove that data on the
server was accessed inappropriately.server was accessed inappropriately.
 And… there was no way to prove that data onAnd… there was no way to prove that data on
the server wasthe server was notnot accessed inappropriately.accessed inappropriately.
 The doors were unlocked and people were in theThe doors were unlocked and people were in the
house, but we couldn’t prove that they stolehouse, but we couldn’t prove that they stole
anything.anything.

13. Second OpinionSecond Opinion
 Magnitude of potential breach warrantedMagnitude of potential breach warranted
additional opinionsadditional opinions
 ITS Security conducted parallel investigation toITS Security conducted parallel investigation to
verify or refute initial findingsverify or refute initial findings
 Additional corroborating data searchedAdditional corroborating data searched
 Network traffic logs (only last 90 days)Network traffic logs (only last 90 days)

14. Notification is not an IT DecisionNotification is not an IT Decision
 University Counsel makes final recommendationUniversity Counsel makes final recommendation
based on inputs from:based on inputs from:
 IT Security (OIS & ITS)IT Security (OIS & ITS)
 University RelationsUniversity Relations
 UNC Health Care Communications/MarketingUNC Health Care Communications/Marketing
 UNC Health Care CounselUNC Health Care Counsel
 HIPAA Privacy and HIPAA Security OfficersHIPAA Privacy and HIPAA Security Officers

15. How do we notify 180,000 people?How do we notify 180,000 people?
 Is their address current? Do we have anIs their address current? Do we have an
address?address?
 Are they still alive?Are they still alive?
 Who writes the letters?Who writes the letters?
 Who addresses the envelopes? Licks theWho addresses the envelopes? Licks the
stamps?stamps?
 Who handles phone calls from concernedWho handles phone calls from concerned
recipients?recipients?

16. The Notification ProcessThe Notification Process
 UNC Hired Rust Consulting to assistUNC Hired Rust Consulting to assist
 Consultation servicesConsultation services
 Mailed notification lettersMailed notification letters
 Established and staffed Call CenterEstablished and staffed Call Center
 Responded to calls; referred problem calls to UNCResponded to calls; referred problem calls to UNC
 Received 4,144 callsReceived 4,144 calls
 450 calls referred to UNC450 calls referred to UNC

17. Technical ResponseTechnical Response
 Major concern: Uncontrolled serverMajor concern: Uncontrolled server
proliferationproliferation
 Determine scope of problemDetermine scope of problem
 Protect high risk machines firstProtect high risk machines first
 Develop long term strategy to mitigate riskDevelop long term strategy to mitigate risk

18. The Scope of the ProblemThe Scope of the Problem
 500+ machines with server OS’s on SOM500+ machines with server OS’s on SOM
networknetwork
 2200 machines running a service2200 machines running a service
 2068 File Server / File Services2068 File Server / File Services
 1989 Remote Access / Remote Management1989 Remote Access / Remote Management
 762 Web Servers762 Web Servers
 194 Database Servers194 Database Servers

19. Manual Data CollectionManual Data Collection
 Mandatory self reporting of serversMandatory self reporting of servers
 433 servers reported433 servers reported
 98 server admins98 server admins
 47 different OS flavors and versions47 different OS flavors and versions
 Qualys scans on all servers reporting sensitiveQualys scans on all servers reporting sensitive
information (200 machines)information (200 machines)

20. Long Range StrategyLong Range Strategy
 IT Simplification and Security RFP (Dell)IT Simplification and Security RFP (Dell)
 Develop Plan for streamlining IT resources in SOMDevelop Plan for streamlining IT resources in SOM
 Develop strategic virtualization architectureDevelop strategic virtualization architecture
 Develop enterprise storage architectureDevelop enterprise storage architecture
 Develop security umbrella to cover centralizedDevelop security umbrella to cover centralized
operationoperation
 Goal: Provide robust centralGoal: Provide robust central servicesservices that willthat will
get end users out ofget end users out of serverserver businessbusiness

21. Recovery from the breachRecovery from the breach
 Moved data to centrally managed serversMoved data to centrally managed servers
 Database encrypted behind hardware firewallDatabase encrypted behind hardware firewall
 All working files encrypted with PGP Net ShareAll working files encrypted with PGP Net Share
 All machines, including desktops, scanned with QualysAll machines, including desktops, scanned with Qualys
 Well defined procedures documented, approved by IRBWell defined procedures documented, approved by IRB
 Two person rule for manual movement of data filesTwo person rule for manual movement of data files
 Update software to automate processesUpdate software to automate processes

22. How much did it cost?How much did it cost?
 Average breach reportedly costs $204 per nameAverage breach reportedly costs $204 per name
 $204 X 180,000 = $36.7 Million!$204 X 180,000 = $36.7 Million!
 Other references state that a major breach costsOther references state that a major breach costs
an organization aan organization a minimumminimum of $1 Million.of $1 Million.
 Postage alone cost $75,000.Postage alone cost $75,000.
 Rust Consulting cost $260,000Rust Consulting cost $260,000
 Thousands of person hours spent on the projectThousands of person hours spent on the project
 OIS Security, ITS Security, OUC, P&A, HIPAAOIS Security, ITS Security, OUC, P&A, HIPAA
Privacy, senior leadership, etc. etc. etc.Privacy, senior leadership, etc. etc. etc.

23. Lessons LearnedLessons Learned
 Implementation of IT Governance is criticalImplementation of IT Governance is critical
 Decentralized server environment is high riskDecentralized server environment is high risk
 New procedures for virus investigationsNew procedures for virus investigations
involving sensitive datainvolving sensitive data
 Disconnect from networkDisconnect from network
 Do not shut downDo not shut down
 Do not perform virus scanDo not perform virus scan
 Notify IT SecurityNotify IT Security

24. Questions?Questions?