Have you ever worked on a project where you felt testing was thorough and complete—all of the features were covered and all of the tests passed—yet in the first week in production the software had serious issues and problems? Join Dawn Haynes to learn how to inject robustness testing into your projects to uncover those issues before release. Robustness—an important and often overlooked area of testing—is the degree to which a system operates correctly in the presence of exceptional inputs or stressful environmental conditions. By expanding basic tests and incorporating specific robustness attacks, Dawn shows you how to catch defects that commonly show up first in production. She offers strategies for making robustness testing a project-level concern so those defects get the priority they deserve and are fixed before release. Join Dawn to learn about robustness tests you can add to your suite and execute in just a few minutes—even if your test team is over-tasked and under-resourced.

1. TH
AM Tutorial
10/1/2013 8:30:00 AM
"How to Break Software:
Robustness Edition"
Presented by:
Dawn Haynes
PerfTestPlus, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Dawn Haynes
PerfTestPlus, Inc.
Dawn Haynes is COO, principal trainer, and consultant for PerfTestPlus, Inc., and a former
director of the Association for Software Testing. Dawn’s unique blend of experience, humor, and
effectiveness at providing tools and techniques that help students at all levels generate new
approaches to common and complex software testing problems has resulted in her international
recognition as an elite trainer of testers. She provides consulting services and is a frequent
speaker at testing conferences, local groups, and intimate gatherings of testers.

3. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
1

4. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
2

5. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
3

6. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
4

7. © 2013 PerfTestPlus, Inc.
Random
Strikes
Exploratory
Testing
Bug
Safari
Fault
Injection
Error
Guessing
© 2013 PerfTestPlus, Inc.
5

8. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
6

9. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
7

10. Description
Miles
© 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
8

11. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
9

12. © 2013 PerfTestPlus, Inc.
Attack
• Directed and focused attempt to evaluate the
quality, especially reliability, of a test object by
attempting to force specific failures to occur.
Ref: How to Break Software - Whittaker
© 2013 PerfTestPlus, Inc.
10

13. Input
Output
Data
Computation
Force all error messages to occur
Force software to use default values
Explore data types & character sets
Overflow input buffers
Find interacting inputs
Repeat inputs/sequence
Force different outputs to be generated for same input
Force invalid outputs to be generated
Force output properties to change
Force the screen to be refreshed
Apply inputs using a variety of initial conditions
Force a data structure to store too many or too few values
Try to modify alternate data constraints
Experiment with invalid operand and operator combinations
Exploit recursion
Force computation results to be too large or too small
Find features that share data or interact poorly
Ref: How to Break Software - Whittaker
© 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
11

14. © 2013 PerfTestPlus, Inc.
Fill the file system to capacity
Force media to be busy or unavailable
Damage the media
File
Assign invalid file name
Vary access permissions
Vary/corrupt file contents
Kernel
Restrict or constrain resources like memory, CPU, threading, etc.
Other
Software
Make unavailable (uninstall, move or rename files, kill process)
Restrict or block access (disable methods, close ports, etc.)
Concurrency check (execute actions in other interacting software)
Ref: How to Break Software - Whittaker
© 2013 PerfTestPlus, Inc.
12

15. © 2013 PerfTestPlus, Inc.
Error
Guessing
• A test design technique where the experience of the tester is
used to anticipate what defects might be present in the
component or system under test as a result of errors made,
and to design tests specifically to expose them.
© 2013 PerfTestPlus, Inc.
13

16. © 2013 PerfTestPlus, Inc.
Business rules;
logic
• Subvert rules
• Bend/break logic
Workflows;
state models
Concurrency
Algorithms
• Skip steps
• Illegal U-turn
• Create zombies
• Interacting models
• Doing the same thing
• Doing different things
• Bust sorting logic
• Trick search operations
• Subvert parsing schemes
© 2013 PerfTestPlus, Inc.
14

17. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
15

18. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
16

19. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
17

20. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
18

21. © 2013 PerfTestPlus, Inc.
Robustness
Testing
Reliability
Testing
• Testing to determine the robustness of the
software product.
• The process of testing to determine the
reliability of a software product.
• An attribute of a component or system specified
or implied by requirements documentation (for
example reliability, usability or design
constraints). [After IEEE 1008]
Source: ISTQB Glossary, 2007
Feature
© 2013 PerfTestPlus, Inc.
19

22. Validation
Fault
Tolerance
• Confirmation by examination and through provision
of objective evidence that the requirements for a
specific intended use or application have been
fulfilled. [ISO 9000]
• The capability of the software product to maintain
a specified level of performance in cases of
software faults (defects) or of infringement of its
specified interface. [ISO 9126] See also reliability,
robustness.
Source: ISTQB Glossary, 2007
© 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
20

23. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
21

24. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
22

25. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
23

26. © 2013 PerfTestPlus, Inc.
© 2013 PerfTestPlus, Inc.
24

27. Source: ISTQB Glossary, 2007
© 2013 PerfTestPlus, Inc.
Robustness
The ability of software to
deliver value during normal
use without stressing out
users (or failing)
The ability of software to
deliver value consistently –
throughout the longevity,
intermittence, or frequency of
interactions
Reliability
© 2013 PerfTestPlus, Inc.
25