1. #BbWorld14
Security Automation at
Blackboard: What are we Doing?
Matthew Saltzman
7-16-2014
Matthew.saltzman@blackboard.com

2. #BbWorld14
Introduction
• Security at Blackboard has been becoming more and more
important in the past couple of years
• Number of notable improvements in the security of the Learn
platform:
– Whitelist replaces Blacklist in Content Editor
– Added security logging
– Improved password storage (SHA-512 with HMAC-SHA-512 Salt)
– Etc.
• Additionally, we’ve been able to target our resources better
– Assessments of Blackboard B2s through development process
– Embedding security into design process of new features/products
2

3. #BbWorld14
Introduction
• All these improvements require many, many more hours of
testing
– Without needing that many more man-hours of work
• To make this happen, we needed to automate as much of our
testing as possible
– Penetration Testing
– Static code analysis
– Reporting on results and problem areas
3

4. #BbWorld14
Infrastructure for Security Testing
Automation
4

5. #BbWorld14
Tools
• Jenkins
– Continuous Build Automation tool
• Chef/Ansible
– Configuration management platform
– Chef is better at continuous state management, in other words, pushing
changes to the server when updates occur
– Ansible is more useful for one-off configuration changes
• URL Database
– This is a database that has a listing of most URLs in the Learn
application, which we use to dictate what to scan
5

6. #BbWorld14
Jenkins
• Build management tool used for continuous delivery
– http://jenkins-ci.org/
• Errors in Jenkins signify a failed build that needs to be
corrected immediately
– New software won’t be released with build failures
• Can also be used to run scripts/tools each time the code is
updated
6

7. #BbWorld14
Jenkins Appearance
7

8. #BbWorld14
Chef
• Centralized configuration management system
– Centralized, in that a single server contains the information about chef
nodes (servers)/configuration scripts (cookbooks), and manages the
state of the servers
• Very useful for managing state that will change often, since the
state is managed from the configuration management system
• Not as useful for one-off changes, since there is a lot of
overhead
8

9. #BbWorld14
Chef Workflow
• Taken directly from the Chef website, http://www.getchef.com/chef
9

10. #BbWorld14
Ansible
• Core product is essentially a remotely executing shell script
• De-centralized, the invoking server pushes information to the
remote host which then runs
• Does not manage state automatically
• However, very useful for configuration changes that maintain
for long periods of time, or for remotely executing scripts
10

11. #BbWorld14
Ansible Script Example
• From http://www.ansible.com/how-ansible-works
11

12. #BbWorld14
URL Database
• Listing of Learn URLs to target for security testing
– Obtained through all of our QA testing/integration testing, manual
browsing of the application, and performance testing
• Logs all requests made to the system
• Log with request data gets piped through a parser, and put into
a relational database to power all penetration testing tools
– Blackboard Learn is difficult to navigate via spider, due to searches and
content creation/editing
– Was even more difficult in earlier versions of learn that relied on
framesets and iframes
• Attempts to correlate URL to Bb source files (work in progress)
12

13. #BbWorld14
Penetration Testing Automation
13

14. #BbWorld14
Tools
• Penetration testing is very much a tool-assisted process
• AppScan Standard
– Stand-Alone penetration testing tool
• All activity takes place internal to the tool itself
• Burp
– Security Proxy Tool
• Meaning a user will proxy traffic to the target site through it to trigger
scanning
14

15. #BbWorld14
AppScan Standard Automation – Overview
• AppScan Standard is our primary penetration testing tool
– Has a wide variety of different types of issues to look for
– Has many different methods for penetration testing
– Results are somewhat accurate, for high confidence rules
• Designed to be used manually once, and then repeat the same
tests over and over again
• Requires a significant time investment to go through and
validate results
15

16. #BbWorld14
AppScan Standard – UI For Manual Use
16

17. AppScan Standard Automation – Dependencies
#BbWorld14
• URL Database
– Uses the request information to create explore files, which determine
what to scan
• AppScan Standard applicatoin
• Automation Tool (Ansible in development)
– Used to run the AppScan Application
17

18. #BbWorld14
AppScan Standard Automation – Components
• Has 3 components
– Login Script (logs the tool back in if the logout page is tested or
reached)
– Explore File (list of pages, complete with headers, cookies, and
parameters)
– Policy Files (determines which violations to look for, and determines
the various methods of penetration testing that appscan can use)
• Has a command line interface to use automatically
18

19. #BbWorld14
AppScan Standard Automation – Process
1. Create a login script for each server being targeted
2. Create a process to convert data from the URL database into
an Explore File
– We are currently using a Stored Procedure
3. Create a policy file determining which rules to run and which
to ignore
4. Create shell script to run AppScan and create XML output file
for consumption later
– It isn’t enough to automatically run scans, the results need to be
reviewed as well
– Without processing the results before reviewing, this can be a long
and arduous process
19

20. #BbWorld14
AppScan Standard Automation – Demo
20

21. #BbWorld14
Burp Automation – Overview
• Accurate XSS testing tool
– Blackboard has had a significant number of XSS issues in the product
• Proxy system, so requires a tool, such as a browser, to navigate
through Burp to a target site
• Does not easily re-scan existing sites
– Is designed for a Security professional to use to spot-check the system
21

22. #BbWorld14
Burp – UI For Manual Use
22

23. #BbWorld14
Burp Automation – Dependencies
• URL Database
– Need to set up a number of requests to send through Burp
automatically
– Uses a very different format than AppScan Standard
• Automation Tool (Ansible coming)
– Used to run Burp
23

24. #BbWorld14
Burp Automation – Components
• Has three main components:
– Feed of RAW request data
– Java based Burp Plugin
– Burp State file
• The saved state of Burp, which contains a macro for logging into Learn, as
well as the types of issues and scanning techniques to be used
• Has the ability to be run from the command line in java’s
headless mode
24

25. #BbWorld14
Burp Automation – Process
• Combine the Burp Plugin Jar file and the Burp Jar file into a
single executable Jar file
• Obtain a set of pages in RAW form to be scanned
• Create burp state file
– Run the standard Burp UI
– Add a macro for logging into the server being targeted
• Run Burp command, passing in required parameters
25

26. #BbWorld14
Static Analysis
26

27. #BbWorld14
Tools
• Sonar
– Tool for running custom PMD rules
– Custom plugin for B2 permissions issues
• In development now
• Based on the rule of least privilege
• AppScan Source
– IBM Security Scanning tool
– Gradle plugin to run this
– Store output for use later
• Sonatype Component Lifecycle Management (CLM)
– 3rd party library vulnerability scanning tool
27

28. #BbWorld14
Sonar – Overview
• Sonar is an Open Source tool for static analysis
– Can be used for code quality, security, unit test analysis, etc.
• For the Security Team, this is where we run PMD rules, and
custom security plugins from
• Dependent on Jenkins
– Each B2 is scanned by Sonar as part of the build process
28

29. #BbWorld14
PMD Rules – Overview
• Runs in Sonar (http://www.sonarqube.org/)
• A number of Security Rules were created to inform us of
security API misuse
– Use of the Blacklist XSS filter instead of SafeHTML
– Use of insecure escaping algorithms
– Missing calls to validate nonce tokens on form submission
• Constantly in flux as the APIs in the product change
• Example:
29

30. #BbWorld14
PMD Rules – Process
• Each Rule contains one Java file representing the rule, and one
unit test
• The Sonar PMD plugin needs to be updated to include the rule
• Each rule needs to be assigned a certain severity
– Blocker and Critical issues prevent the build from being released
– Major, Minor, and Info are used for information purposes
• Won’t block the build
• Issues in these categories are added to the backlog if isn’t fixed
30

31. #BbWorld14
Sonar B2 Permissions Plugin – Overview
• B2s should never request more “permissions” than they use
– Aside from a few custom Learn specific permissions, the list is here:
http://docs.oracle.com/javase/7/docs/technotes/guides/security/per
missions.html
– Limiting permissions prevents exploits by limiting the impact area
Severity of security issues can most often be contained by
limited by the proper application of Java permissions
31

32. #BbWorld14
Sonar B2 Permissions Plugin - Components
• XML parsing class to determine which permissions are
requested
• PMD rules to determine which permissions are used
• Class to find out if any permissions were requested but not
used
• Sonar module to display excessive permissions (In
Development)
32

33. #BbWorld14
Sonar B2 Permissions Plugin – Process
1. Configure Sonar-Runner to point to the sonar instance being
used
2. Make sure your sonar instance is configured and turned on
3. Add the plugin to the sonar instance if this hasn’t already
been done
4. Navigate to the B2 being tested
5. Run the sonar runner
33

34. #BbWorld14
Sonar B2 Permissions Plugin - Demo
34

35. #BbWorld14
AppScan Source – Overview
• Security Source Code Scanning tool
– Inspects class files and source files for vulnerabilities in the code itself
– Source of the largest subset of vulnerabilities found by our tools
• List of findings is enormous, actual vulnerabilities are less than
5% of these
• Designed for automation, unlike dynamic penetration testing
35

36. #BbWorld14
AppScan Source – Manual Use UI
36

37. #BbWorld14
AppScan Source – Dependencies
• Jenkins
– AppScan is used to scan each individual B2 as an independent
application
– Can scan each B2 on each check-in by integrating with Jenkins
37

38. #BbWorld14
AppScan Source – Process
1. Compile the project to be scanned into class files
2. Run the AppScan Source script to scan a project
– Creates a new application record in AppScan Source
– Adds each project with classes and dependencies to that application
– Scans all projects inside the application
38

39. #BbWorld14
AppScan Source – Demo
39

40. #BbWorld14
Sonatype CLM – Overview
• 3rd party library scanning tool
– Examples: Struts/Spring
– Checks for vulnerabilities listed in vulnerability database like
NIST/MITRE only
• Jenkins plugin comes standard
• Removes burden of finding library vulnerabilities from
Engineers/Security Team
40

41. #BbWorld14
Sonatype CLM Report – Issue Appearance
41

42. #BbWorld14
Sonatype CLM – Dependencies
• Jenkins
– Each B2 package needs to be scanned for new libraries containing
vulnerabilities or new vulnerabilities in existing libraries
– Running as part of the build process helps ensure that newly found
issues get fixed quickly
42

43. #BbWorld14
Security Operations and Reporting
Console
43

44. #BbWorld14
Operations Console – Overview
• Issue repository for various scanning tools
– AppScan Standard results
– Burp Results
– AppScan Source results
• Internally developed technology
– https://github.com/blackboard/security-operations-console
• Allows for finding review
• Contains reports on those findings
44

45. #BbWorld14
Dynamic Analysis Findings Display
45

46. Operations Console – Reviewing a Dynamic Finding
#BbWorld14
1. Navigate to the URL presented
2. Enter a known vulnerable string into the entity
3. If issue is valid, select the valid radio button
– Enter bug ticket number in below text box
4. If issue is not valid, select the false positive radio button
– Enter a reason that the issue is not valid below, so when the issue is
reviewed, it’s understood why False Positive was selected
46

47. #BbWorld14
Static Analysis Findings Display
47

48. #BbWorld14
Operations Console – Reviewing Specific B2s (Static
Analysis)
• Static analysis is directly tied to individual projects
– So we can filter down to individual B2s as well
• Can also specify the severity and issue quality
– Severity of issues is determined by AppScan source, rather than our
team
– Quality is how confident AppScan Source is with its’ finding
• Can also select issues found during a specific time frame
– Allows us to review findings from particular builds
48

49. #BbWorld14
Static Analysis Issue Display
49

50. #BbWorld14
Operations Console - Reporting
• 3 Reports
– Dynamic Analysis CWE Reporting
• Valid vs False Positives
– Static Analysis Vulnerability Type Report
• Valid vs False Positives
– Issues found By Date
• New Issues vs Reviewed for both Static Issues and Dynamic Issues
50

51. #BbWorld14
Operations Console – CWE Report
51

52. Operations Console – Vulnerability Type Report
#BbWorld14
52

53. #BbWorld14
Operations Console – Issues By Date Report
53

54. #BbWorld14
Self Service Penetration Testing Tool
54

55. #BbWorld14
Self Service XSS Tool – Overview
• Tool that allows developers to test their own code on their
system without a security testing tool installed
• Based on Burp Proxy
• Internally developed
– Currently in progress
• Does not log data about findings
– Meant to be used prior to checking new code in, not as something for
the security team to evaluate issues on
55

56. #BbWorld14
Self Service XSS Tool – Current UI Appearance
56

57. #BbWorld14
Self Service XSS Tool – Components
• Burp Plugin
– Passes all URIs sent in into the proxy to the active scanner
– Sends results back to the self service tool’s web server to display to the
users
• Web Application
– Application that functions as a proxy for whichever system the
developer wants to target
• Which is how we can target users’ development builds
– Also reports issue data via web sockets
57

58. #BbWorld14
Self Service XSS Tool – Process
1. Enter the full URL of the system being targeted, and your
username and password (tells the system who you are)
2. Browse to the area of the application being tested
3. View the results for the area of the application being tested
4. Correct the issues found
5. Navigate back to that area of the application and retest
58

59. #BbWorld14
Self Service XSS Tool - Demonstration
59

60. #BbWorld14
Questions?
60

61. #BbWorld14
URL Database – Schema Diagram
61

62. #BbWorld14
AppScan Standard Explore File Format
<?xml version="1.0" encoding="utf-8"?>
<requests>
<request method="GET" host="appsec-atarg02.pd.local" path="/webapps/assessment/do/authoring/viewAssessmentManager" scheme="https"
SessionRequestType="Login" ordinal="156" port="8443">
<raw encoding="none">
GET /webapps/assessment/do/authoring/viewAssessmentManager HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
Connection: keep-alive
Cookie: session_id=F40578BAC62379AB617E0CF3705E3AA7; JSESSIONID=8F181C8BEE9867B127538607E6914F15.rootHost: fitnesse-patch02.pd.local
Referer: http://fitnesse-patch02.pd.local/webapps/blackboard/landingPage.jsp?navItem=cp_test_survey_pool%26course_id=_2_1%26sortItems=false
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2
</raw>
<cookie name="session_id" value="F40578BAC62379AB617E0CF3705E3AA7" path="/" domain="appsec-atarg02.pd.local" secure="False"
expires="1/1/0001 12:00:00 AM"/>
<cookie name="JSESSIONID" value="8F181C8BEE9867B127538607E6914F15.root" path="/" domain="appsec-atarg02.pd.local" secure="False"
expires="1/1/0001 12:00:00 AM"/>
<parameter name="assessmentType" captureIndex="0" value="Test" type="QUERY" linkParamType="simpleLink" separator="&amp;" operator="="
reportName="assessmentType"/>
<parameter name="course_id" captureIndex="0" value="_2_1" type="QUERY" link ParamType="simpleLink" separator="&amp;" operator="="
reportName="course_id"/>
<sessionCookies>
<cookie name="session_id" value="F40578BAC62379AB617E0CF3705E3AA7" path="/" domain="appsec-atarg02.pd.local" secure="False"
expires="1/1/0001 12:00:00 AM"/>
<cookie name="JSESSIONID" value="8F181C8BEE9867B127538607E6914F15.root" path="/" domain="appsec-atarg02.pd.local" secure="False"
expires="1/1/0001 12:00:00 AM"/>
</sessionCookies>
</request>
</requests>
62

63. #BbWorld14
AppScan Standard Automation Command
• appscancmd /test_only /starting_url http://${target_machine}
/dest_scan
Z:Security_AutomationScan_Files${file_name}.scan
/scan_template
Z:Security_Automationtemplateparameter.scant
/manual_explore_file
Z:Security_AutomationExplore_Data_Files${file_name}.exd
/policy_file
Z:Security_AutomationPolicy_Files${policy_file}.policy
/report_type rc_ase
63

64. #BbWorld14
Burp Login Macro
64

65. #BbWorld14
Burp Command
/usr/local/jdk7/bin/java -Dorg.owasp.esapi.resources=<project-directory>/
lib -Xms2048m -Xmx2048m -
XX:+ExplicitGCInvokesConcurrent -XX:+UseConcMarkSweepGC -
XX:+UseParNewGC -XX:+UseTLAB -Djava.awt.headless=true -
classpath <jar directory>/BurpExtender.jar:<project-directory>/
lib/esapi-2.0.1.jar:<project-directory>/
lib/burpsuite_pro_v1.4.12.jar:<project-directory>/
lib/ojdbc6.jar burp.StartBurp <target-machine>
<project-directory>/login_files/<target-machine>.burp.login
<location of scan files>/<output file name>
<xml_output_directory>/<output file name> <blackboard
version>
65

66. AppScan Source Gradle Plugin Execution Command
/usr/local/jenkins/tools/hudson.plugins.gradle.GradleInstallatio
n/Gradle_Default_Bb-custom_/gradle-1.12/bin/gradle --
stacktrace --refresh-dependencies -
PappscanBinDir=/usr/local/AppScanSrc/bin -
Pmongodb=appsec-db.pd.local AppScanSource -b
assignment/build.gradle
#BbWorld14
66

67. #BbWorld14
Operations Console Code Coverage Report
67

68. #BbWorld14
Dynamic Analysis Issue Display Key
Key Description
ID Unique identifier for the issue
URL The url in Learn that is vulnerable
Url Parameters The set of GET parameters
Body (POST) Parameters The set of POST parameters
Entity Type The vulnerable entity (Could be
Parameter, Cookie, Header, etc.)
Entity Name If applicable, the name of the parameter
Reasoning The reason that the entity is flagged as
vulnerable
CWE Common Weakness Enumeration
http://cwe.mitre.org/
Entity Change What, if anything, was altered in the
entity to show the vulnerability
68