TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
Creating Practical Security Test-Cases for Web Applications
1. Creating Practical Security
Test-Cases for Web
Applications
Rafal M. Los
HP ASC Sr. Security Solutions Expert
7 May 2009
1
2. Agenda
Understanding the QA/Security Relationship
Negative Testing 360°
Building Negative Tests
Implementation and Execution
Looking Ahead
7 May 2009
2
3. Agenda
Understanding the QA/Security Relationship
Negative Testing 360°
Building Negative Tests
Implementation and Execution
Looking Ahead
7 May 2009
3
4. Background
Why do QA teams care about
security?
• Traditionally security is left to the security team
• Security issues must be addressed throughout
SDL
• QA teams add missing element
QA teams are crucial to security
• Understand application test-cases
• Understand application workflows
• Security is a natural extension of quality
7 May 2009
4
5. QA – Security Relationship
Similarities – core principles
• Testing web application logic
• Functional testing on live code
• Specific data-sets used
Differences – outlying goals
• Stress-test vs. break test
• Positive vs. negative data sets
• Reinforcing positive vs. uncovering
negative
7 May 2009
5
6. The “Hacker” Mindset
Why would anyone want to break an application?
Fun
Malice
Profit
Attack users
Attack systems
Mentality difference
QA asks – How does it perform?
Hacker asks – How can I break it?
7 May 2009
6
7. Whose Problem is Security?
Many components to the security “problem”
Policy
Development frameworks/standards
Audit
Metrics
Security is a pillar of overall quality
Does it function?
Does it perform?
Is it secure?
7 May 2009
7
8. Agenda
Understanding the QA/Security Relationship
Negative Testing 360°
Building Negative Tests
Implementation and Execution
Looking Ahead
7 May 2009
8
9. Negative Testing Overview
What is negative testing?
Testing for unintended features
Testing using unintended data sets
Testing for unintended logic flow
“Negative testing involves understanding the
application, and finding ways to manipulate the
code to perform in ways as to create unintended
exposures”
7 May 2009
9
10. Negative Testing Overview
Selection bias Confirmation bias
Testing to confirm desired results
Testing using known desired data and flows
Testing which completely misses the point…
“…confirmation bias is a tendency to search for or
interpret new information in a way that confirms one's
preconceptions and to avoid information and
interpretations which contradict prior beliefs”
7 May 2009
10
11. Negative Testing Mindset
Traditional QA: proving the positive
• Prove certain activity functions as defined by
business case
• Requirements are easily defined in application
flow and function
Negative testing: finding the negative
• Find negative (unintended) functions/results
• No way to clearly define “bad stuff” as a
requirement to test against
7 May 2009
11
12. Negative Testing - Data
Types of negative data depends on purpose
• Exploit a client
• Client-side script or technology
• Corrupt or crash a system
• Database control characters
• Non-native character sets, system characters
• System commands
• Retrieve data from the system
• Database queries, control language
• System commands
7 May 2009
12
13. Negative Testing - Flow
Goal is to manipulate application logic
Identify “breakable” application logic
• Create a race condition
• Break application control-flow
• Force an out of process action
• Inject a rogue process
Test-cases based off of proper application logic flows
Requires in-depth knowledge of application flow
7 May 2009
13
14. Negative Testing - Tools
Tools are an integral part of
negative testing
• Manual tools
• Flow diagrams
• Data sets
• Logic charts
• Automated tools
• Black-box scanners
• Manu-matic framework tools
7 May 2009
14
15. Negative Testing - Tools
Automated tools *cannot* perform all testing
• Workflow-base vulnerabilities
• Complex attacks
• Multi-stage
• Business logic
Human beings must…
• Analyze the application logic and data
• Guide tools
• Interpret results
7 May 2009
15
16. Agenda
Understanding the QA/Security Relationship
Negative Testing 360°
Building Negative Tests
Implementation and Execution
Looking Ahead
7 May 2009
16
17. Building the Test
Phase 1 Phase 2 Phase 3 Phase 4
Mapping the Tools-Based Manual-Based Analysis &
Application Testing Testing Correlation
•Business logic •Tools-generated •Workflow defect •Analyze results
data-set testing tests of automated &
•Application flow
manual results
•Automated •Business logic
•Application
•Correlate P2/P3
crawler-based tests
surface
testing results
•Complex
•Application
•Known defects attacks
entry-points
7 May 2009
17
18. Building Data-Negative Tests
All possible inputs
Data-
•Letters
Negative Test
unknown Situational
•Numbers
Data
Refinement
(unknown
•Special characters
impact)
•Control characters •Database SQLi
•Client-side XSS
•XMLdb X-Path.i
Case-
Allowed
specific
(positive) •Cross-site scripting
malicious
characters •SQL Injection
•Overflows
7 May 2009
18
19. Building Data-Negative Tests
Manual human testing
Must build test data sets manually
Sniper approach (can be precise)
Often very slow, methodical
Identifies false-positives
Tools-based testing
Builds test data sets automatically
Shotgun approach (not precise)
Ability to be extremely fast
Trouble with false-positives
7 May 2009
19
20. Negative Data Sets
Facts about negative data
Negative data sets are best generated by tools if
the tester is not a security expert
Many pre-built negative data sets already exist
Sla.ckers.org – XSS cheat-sheet
Tools can point click test
Black-box testing tools save time & effort
Humans must analyze results
Must mix positive/negative data for completeness
Workflows often require good data to proceed
Automated negative-data testing fails without good data
7 May 2009
20
21. Flow Analysis Testing
Can a process step be bypassed?
Submit quote
for someone
Step 1 Step 2 Step 3 Step 4
else?
Verify Identity Request quote Receive quote Submit for
purchase
Can a process step be injected?
Step 1 Step 2 Step 3 Step 4
Verify Identity Request quote Receive quote Submit for
purchase
Injected!
Modify quote
7 May 2009
21
22. Flow Analysis Testing
Manual human testing
Can analytically identify specific weak points
Distinguishes between success/failure readily
Often very slow, methodical
Ability to tailor testing to situation/process
Tools-based testing
Attacks every point, cannot distinguish
Difficulty distinguishing success/failure
Ability to be extremely fast
Cannot think therefore has limited abilities
7 May 2009
22
23. Flow Analysis Testing
Facts about flow analysis testing
Tester must understand application flow
Proper application flow to turn into negative
“Random manipulation” rarely works
Focus on application control-points
Key points in application logic
Don’t leave your testing to tools-only
Most tools can’t identify control points, dive deep into flows
Human analyst has an obvious advantage (critical thinking)
7 May 2009
23
24. Agenda
Understanding the QA/Security Relationship
Negative Testing 360°
Building Negative Tests
Implementation and Execution
Looking Ahead
7 May 2009
24
25. Negative Testing Process
Analyze
Requirements
New Functionality
Execute Phase 1
Discovered?
Execute Phase 4 Execute Phase 2
Execute Phase 3
7 May 2009
25
26. Testing Negative Data
Identify all visible inputs (data “source”)
1.
Input positive data
i.
Analyze behavior
Input negative data
ii.
Analyze behavior
Identify all hidden fields (data “source”)
2.
Input positive data
i.
Analyze behavior
Input negative data
ii.
Analyze behavior
7 May 2009
26
27. Testing Negative Flow
… as we’ve learned this will be manual work
Map out all control-flows
Identify a potentially weak logic element
Walk the positive-control flow path
Ensure proper positive path is understood
Map possible negative-control flow paths
Execute negative-control flow paths
Analyst difference between positive/negative attempts
Repeat if necessary to adjust/adapt until satisfied
Attempt at least 3-5 loop-repetitions
7 May 2009
27
28. Identify Weaknesses
How do you identify a weakness/defect
Undesired application reaction
Crash?
Skip control step?
Disclosure of unintended data
Debug information
Disclosure of internal data
Disclosure of controlled data
7 May 2009
28
29. Agenda
Understanding the QA/Security Relationship
Negative Testing 360°
Building Negative Tests
Implementation and Execution
Looking Ahead
7 May 2009
29
30. Looking Ahead
Addressing “deep” defects
Workflow-based security defects
Traditionally cannot be scanned for (with automated
tools)
Analysis of Defects
When is a critical defect… not?
QA expertise contextualized defects
7 May 2009
30
31. Questions?
• Security Strategist
• Application Security Specialist
• Following the White Rabbit:
http://www.communities.hp.com/securitysoftware/blogs/rafal
• Digital Security SoapBox: http://preachsecurity.blogspot.com/
• Email: Rafal@hp.com
• Direct: (404) 606-6056
7 May 2009
31