Life after the hack

Life after the hack
OSInetFrédéric G. MARAND (fgm)

2/45DrupalDevDays Milan: Life After the Hack - (c) 2016 Frédéric G. MARAND for www.osinet.fr
Topics
• 1 Intro : setting the stage
• 2 Snapshotting
• 3 Maintaining presence
• 4 Crisis communication
• 5 Rebuild, don’t repair
• 6 Using forensics tools
• 7 Back online

1.1 Some fact checking first
• In this room …
• Who has been hacked already ?
• Who feels ready to face a hacked server ?
• Who actually has a contingency plan ?
• Who read node 2365547 ?

1.2 Can you say that again ?
I.A.N.A.L.
So be sure to get one !

1.3 Whence do I speak ?
• Drupal org member since 2005 (fgm)
• Drupal consultant, not a site building agency
• Worked on fixing broken (in) sites since 2008
• Auditing
• Fixing technical flaws
• Addressing intrusions / exploits
• Mostly Media and Government sites (.fr)
• Provisional member of the Security Team

1.4 Setting the stage
• 10:00 The daily scrum has just begun.
• 10:01 Phones rings : someone noticed your site
has been defaced and is warning you
• 10:02 Twitter and Reddit start buzzing
• 10:05 Phones ring all over the place, with
journalists and the various C-level execs on
the other end, your mailbox is filling with
warnings
• What is your next step ?

1.5 Get ready
• Pad 1 : discovery log
• all your work steps
• all your findings / observations
• with timestamps and numbers
• Pad 2 : remedies ideas
• cross-refer pad 1 numbers
• all your ideas for fixing the breach
• all your ideas for further hardening

2.1 Forensic copy : why ?
• First temptation : restore and resume
• But you’re still vulnerable
• So you need to diagnose
• Analyzing means modifying
• So preserve the « crime scene »
• Snapshot everything

2.2 Snapshots : pull the plug
• Prevents interference
• Shutdown
handlers, SIGPWR
• Self-destructing
code on network
loss
• Easy on VMs
But…
• Bare remote servers
• Further data loss
• Journaled FS
• Databases
• Service interruption

2.3 Snapshots : what ?
Not just the main DB
• Reverse Proxy logs
• Web fronts
• DB servers
• File servers
And also…
• External logs (SaaS)
• External transactions
• IDS/firewall logs
The site may just be an attack vector

3.1 Maintaining presence 1
• Yes
• Don’t tip off
hackers
• Keep generating
short-term value
• No
• Increasing
damage
• Responsibility
• Legal
• Financial
• Moral
As though intrusion had not been detected

3.2 Attacker workflow
Evolved
• Break in
• Dig for gold
• Implant zombie
• Wait for implant
migration to archives
• Activate
• Profit
• Alt : Need for Speed
• Use exploit ASAP
• While it lasts
• Usually least loss
• Alt : hidden steal
• Valuable content
• Identity data
• Close the door

• Limited static site
• Best with prior
work
• Minimal subset
• Possibly taken
from RP cache
• Very little load : can
run on RP heads
• Working limited site
• Alternate infra
• Alternate tech
• Updates ?
• Content created
during this step
Safe fallback mode

When all else fails
• Social networks
• Always there
• Also authoritative for audience
• Still needs some preparation :
• Accounts access
• Include them in long-term communication

4.1 Communicating : from tech
• Stakeholders
• Chain up to CxO level in most cases
• Prepare next steps, do not overreach
• Fear of reprisal ? Gag orders, SLAPP…
• Protection
• France : whistleblower protection (Sapin 2)
• Italy : Dec. 385 01/09/93 sect 52bis (banks)
• US : Anti-SLAPP
• Many other countries have similar rules

4.2 : Communication : C-level
• Legal counsel (first)
• Crisis Management specialists
• Law enforcement
• EU countries typically have specialized
units for « cybercrime »
• Other sites
• On same server
• On same network
• Online business partners

4.3 Communication : privacy
• In many cases personal data leaks
• will happen, or...
• unprovable they did not happen
• Operational constraints
• Commerce : PCI/DSS (12 steps etc)
• Health : (US) HIPAA Subtitle D E2.80.93
• Public image damage control
• A french example

5.1 Rebuild : keep, rollback or ?
• Restore and restart same ?
• Still just as vulnerable
• Keep and fix ?
• lots of time and effort reviewing
• never completely trusted : not just Drupal
• Throw away ?
• Event sites, past lines of biz, post-M&A...
• Can a static version suffice ?
• From RP snapshots : recent content

5.2 Rebuild : restore
• Needs backups from before the hack
• Do you know when it happened ?
• Remember attacker workflow « wait »
• GFS, continuous incremental, 15 min ?
• How much can you lose ?
• FLOSS solutions : Amanda, Bacula, custom
• Unprepared emergency ?
• Preproduction, CI builds...

5.3 Rebuild : sources + export
• Easy and reliable, but assumes :
• Code-driven development process
• Reliable data export system in place
• Flat content exports
• Content + assets repositories
• Still need to add the fixes
• Delay can be a problem on high-volume sites
• Bulk handling, Incremental loading

5.4 Rebuild : other cases
• Ad hoc « traditional » build process
• Longer, less reliable
• Too long to be a chance to fix the process
• From scratch
• Too long in most cases
• Do it as a complement after the fix
• Not NOW

6 Forensics : switching hats

6.1 Forensics : first, think !
• How did you become aware of hack ?
• What did it take to succeed ?
• Cast your net wide, think big
• « Unlikely » vs « impossible »
• Priority :
• Easiest attacks first
• OWASP 10
• GIYF : search your Pad 1 patterns

6.2 Forensics : keep in mind
• /anything/ may be erased after success
• But most of the time, not /everything/ will
• Anything you do leaves its own traces
• Work on copies of the snapshots
• You can restart from fresh copies anytime
• There maybe more than one exploit

6.3 Forensics : classics
• Code files :
• lax permissions
• filesystem traversal issues
• Remote payload execution by upload
• Nginx without extra hardening
• .htaccess won’t do much good
• In-DB PHP
• PHP module
• Eval-uated code

6.4 Forensics : non-Drupal
• Filesystem :
• <user>/www-data outside /sites
• www-data/www-data suspicious
• x bit on files below docroot
• timestamps
• outside sites/*/files = install
• exploits > install
• meld with fresh build from sources
• Also check outside docroot

6.5 Forensics : Drupal modules
• Code signing/diffing :
• Hacked!
• D7 : md5check, file_integrity
• Finding DB PHP
• QA (github)
• Misc
• security_review

6.6 Forensics : DB
• Quick wins :
• users.email!= users.init
• review roles, accounts with admin roles
• On corp. sites, users.email domains
• match users accounts with SSO data
• Diff DB snapshot with live
• Especially menu_router :
file_put_contents, assert
• Altova DatabaseSpy content compare

6.7 Forensics : sessions
• Sessions should be in persistent storage
• Remember when you pulled the plug
• Were your sessions in Memcache ?
• sessions.timestamp vs users_field_data :
created/changed/access/login
• for intranets : sessions.hostname

6.8 Forensics : logs
• You use off-site logs, right ?
• SaaS : Loggly, Logmatic, Logsene, Logz.io,
Papertrail, Scalyr….
• Remote ELK
• On site ?
• dblog {watchdog}
• syslog → follow the redirects
• mongodb_watchdog
• Application/WS logs

6.9 Forensics : sleuth tools
• Software
• Guidance
Software : Encase
• AccessData :
Ultimate
Forensics Toolkit
(FTK)
• Consider certified
consultants

7.1 Live again : restoring prod
• Recheck Pad 1 findings vs new build
• Usually, reset passwords. On D7 :
• update users
set pass = concat('ZZZ',
sha(concat(pass, md5(rand())))
);
• Prepare marketing/social copy

7.2 L8R : future-readiness

7.3 L8R : disaster prevention
• Developer education on security
• Security Team mailing list
• https://twitter.com/drupalsecurity
• https://www.drupal.org/security/rss.xml
• http://crackingdrupal.com/

7.4 L8R : disaster prevention
• Security process
• Analyse sec. releases to understand fixes
• Look for similar flaw in custom code
• Take part in contrib for more expertise
• Quality process
• Systematic peer code reviews
• Code-driver maintenance + dev process
• Automatic quality tools in CI
• Contrib updates scheduling

7.5 Continuous improvement
• You can’t improve what you don’t measure
• Get time metrics from Pad 1
• Build contigency plan from Pad 2
• Plan for periodic intrusion simulations

Life after the hack

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (13)

Similaire à Life after the hack

Similaire à Life after the hack (20)

Plus de OSInet

Plus de OSInet (14)

Dernier

Dernier (20)

Life after the hack