Much of the automating we do to support testing involves detecting change. Once our tests pass, they fail when the system changes and the automated execution alerts us to the change. There are other ways that automating can help us.

1. Automating to Augment Testing
Breakpoint 2020
Alan Richardson
— @EvilTester
— EvilTester.com
— github.com/eviltester
— compendiumdev.co.uk
— digitalonlinetactics.com
@EvilTester 1

2. What is Traditional Test Automation?
— Automated Execution of Paths through a System
— Automated Assertion on Conditions during the
execution
— Reporting on the Automated Execution
@EvilTester 2

3. Traditional Test Automation is a subset of
Automating
— Tools
— Build Process
— Release Process
— Model Based Automating
— Tactical scripting and activities
We automate processes. Doing stuff.
@EvilTester 3

4. What Can we Automate?
— Interaction
— Observation
— Interrogation
— Condition Checking
— Assertions
— Reporting
What else?
@EvilTester 4

5. Think Differently
Go beyond "Test Automation".
— What processes do I do?
— Would it help to automate some of that?
— What am I not doing?
— Could I do that if I had tooling or an automaton
to help?
— How can I reuse what I've got?
@EvilTester 5

6. e.g. What if...
— What if I could have an environment spin up
automatically?
— What if the browser could just 'be in the right
place' so that I could start exploring?
— What if I could test while a bunch of other 'people'
were using the same data at the same time?
@EvilTester 6

7. e.g. What if...
— What if something could just 'tell' me that this
page had some 404 request in the back ground as
I test?
— What if we could just have something running in
the background to exhaustively cover all the data
combinations - just in case?
What if...?
@EvilTester 7

8. Examples
— Fuzzers to create data
— Release scripts to create Test Envs
— Performance Tests as background load
— Automated Execution as release checking tools
— Monitoring to identify gaps in coverage - live
errors, weak signals
— etc.
@EvilTester 8

9. Unstable Application Context
— Registration process: 6 or so pages, 30+ fields.
— Outsourced Development, not viewed as high risk
— No Automating/Testing from development team
— Every release had something different not
working
— Testing would take days to hit upon a combination
of issue revealing data
— "Throw it over the wall" process
@EvilTester 9

10. Common "Test
Automation" Knowledge
"Never Automate an application
that has lots of bugs and is
constantly changing. Wait until the
application is stable."
@EvilTester 10

11. Knock on Effect
You spend a lot of people time
identifying the issues in an
unstable application, make their
life frustrating and make little
progress.
@EvilTester 11

12. Why do we automate?
— to save time
— to save money
— to increase coverage
— to allow testers to do something more valuable
instead
Could we do that?
@EvilTester 12

13. What could we do?
— Identify the common issues found
— Model the paths that they trigger along
— Identify data partitions used to find them
— Exhaustively traverse and report issues
@EvilTester 13

14. How could we do that
— Model Based Testing, not Model Based Tooling
— Model 'Valid' Equivalence Partition Data
— Paths, Inputs
— Makes 'Oracle' easy
— "Must go to next step",
— "Must report success on submission"
@EvilTester 14

15. The Example Application
testpages.herokuapp.com/
styled/validation/input-
validation.html
— this only has 5 fields so:
— we can automate it in about
an hour
— automation takes < 10 mins
to run
@EvilTester 15

16. Run The Test
@EvilTester 16

17. @EvilTester 17

18. Final Error Report
==================
@EvilTester 18

19. Backend Validation Issue
-----
Test: 15
-----
Using for age: 28
Using for firstname:
PRMN QBYUOUMKPYJHEIGJ BQJXYVIH
Using for Country: Panama
Using for lastname: A AAIRPTUVDSTYZHUC:
Using for notes: JHKIWFOPR:
Submit Form
@EvilTester 19

20. Frontend Validation Issue
-----
Test: 108
-----
Using for Country: Eswatini (fmr. "Swaziland")
Using for notes: HIUPDRY:
Using for age: 42
Using for firstname: JVTDFAYFCNZIFRSXNSOGI
Using for lastname: CCKJRUUJLRAMLL
Submit Form
@EvilTester 20

21. Sample Code
@Test
public void aTest(){
for(int x=0; x< 200; x++) {
report="";
reportThis(String.format(Test: %d%n", x));
try {
visitForm();
fillFormCorrectly();
submitForm();
checkValidInput();
}catch(Exception e){
errorReport = errorReport + report;
}
}
// print report to system out
@EvilTester 21

22. Fill The Form
private void fillFormCorrectly() {
String[] theFields = {"firstname", "lastname",
"age", "country", "notes"};
List<String> fieldNames = new ArrayList<>();
fieldNames.addAll(Arrays.asList(theFields));
while(fieldNames.size()>0){
String nextFieldToFill = fieldNames.get(
random.nextInt(fieldNames.size()));
fillFieldWithValidRandomValue(nextFieldToFill);
fieldNames.remove(nextFieldToFill);
}
}
@EvilTester 22

23. Fill a Field
private void fillFieldWithValidRandomValue(
final String nextFieldToFill) {
switch (nextFieldToFill){
case "firstname":
// 5 to 89
stringValue = getRandomLengthStringValue(5, 89, "firstname");
driver.findElement(
formFields.get(nextFieldToFill)).sendKeys(stringValue);
break;
@EvilTester 23

24. Fill Another Field
...
case "country":
// valid country from list
int countryIndex = countryPicker.getNextRandomCountryIndex();
Select select = new Select(
driver.findElement(formFields.get(nextFieldToFill)));
select.selectByIndex(countryIndex);
reportThis(String.format("Using for Country:%n%s%n",
select.getFirstSelectedOption().getText()));
break;
...
@EvilTester 24

25. Equivalence Partitions - it doesn't matter:
— what order we fill in the fields
— what string values we use so long as they are the
right length
— what age provided within range
— which country so long as in the list
— perform exhaustive coverage of country list
Randomise. Execute path enough times to achieve
coverage.
@EvilTester 25

26. Oracle
— can submit form
— no client side validation errors
— no validation errors on submitted page
— no server side validation errors
@EvilTester 26

27. Random Data
— when using random data
— report on data used when flagging an error
— strategically
— allow seeding to repeat tests
code github.com/eviltester/modelbased
@EvilTester 27

28. Think Differently About Automating
— Our "Test Strategy" was Agile, with a lot of
automating
— This project "didn't need automation"
— Automating was used as a tactic to support
testing
— Automate first. Test if stable.
— Automated across Equivalence Classes Models -
everything was valid
@EvilTester 28

29. Decisions
— I decided to 'risk' a few hours of my time, to see if
we could save days/weeks of testing effort.
— Tactical. Throw Away.
— Did not spend a lot of time on the code.
@EvilTester 29

30. Test Automation Does Not Mean Replacement
— Test Automation does not mean 'replacement'
— It means automating as part of a Test Approach or
Strategy
— Sometimes that means tooling
@EvilTester 30

31. Observation
— When testing I can't observe everything
— I can't watch the GUI and the Network Traffic, and
the back end server, and the memory usage and
the...
— Tools can help me observe, as I'm testing
— e.g. I can see status codes in network panel as I
test
@EvilTester 31

32. Interrogation
— I find it hard to 'Observe' the contents of
messages
— I have to open and Interrogate them later
— Could tooling 'observe' message content for me?
@EvilTester 32

33. Case Study
— Data Leakage Security Testing
— Using a site, wanted to identify any data leakage
possibilities
— Observed, not Interrogate
@EvilTester 33

34. CDP Test Support
— given a set of values to look for
— start a browser
— tell me if we find any of those values in requests
as I test
email_1=george@mailinator.com
email_2=bob@mailinator.com
@EvilTester 34

35. @EvilTester 35

36. CDP Test Support
— Tactical Automating
— Met a need
— Uses Chrome Debug Protocol
github.com/eviltester/cdptestsupport
@EvilTester 36

37. Think Differently About Automating
— What is the minimum you can do to increase the
capabilities of testers?
— Augment, rather than replace.
— Prefer and off the shelf tool prior to writing code.
@EvilTester 37

38. What does it take to Automate?
— Specialist Tools?
— Libraries?
— Expensive environments?
@EvilTester 38

39. What if we just used the
browser dev console?
To create data
@EvilTester 39

40. @EvilTester 40

41. Simple Code
for(x=1; x<=100; x++){
setTimeout(function (name){
document.querySelector(
'input.new-todo').value=name;
document.querySelector(
'input.new-todo').
dispatchEvent(new Event(
'change',{'bubbles':true}));
}, x*100, "todo "+x)
}
@EvilTester 41

42. Automate the Un-automatable
— Some applications are very hard to automate
externally
— e.g. Canvas Based - no HTML elements
— Automating might require visual automating or AI
— How do we Automate the Unautomatable?
e.g. phoboslab.org/xtype/
@EvilTester 42

43. @EvilTester 43

44. A Bit More Code Required
@EvilTester 44

45. Think Differently About Automating
— What are the untapped capabilities of the tools
you already use?
— Can you script your tooling?
— What can you do with a little imagination?
@EvilTester 45

46. Can you re-use your Automation code?
— Do you have abstractions you can re-use, or are
they locked to a framework?
— Is everything driven by Gherkin tools or can your
write custom execution?
— Can you quickly create adhoc Tests that do
something specific and unexpected?
@EvilTester 46

47. Could you use your abstractions...
— To write small, concurrent, autonomous bots?
@EvilTester 47

48. Given a multi-user app
— And Single User Tests Written
— And Abstraction layers in place
— When I want to know if it can handle multiple
users
— Then do I write completely new tests in a Stress/
Performance tool?
Or can I repurpose the abstractions to create multi-
threaded model based bots.
@EvilTester 48

49. Abstractions As Dependency
— Package Abstraction Layers for re-use
— Helps keep them 'clean'
— Strategic, but opens up more tactical use cases
<dependency>
<groupId>uk.co.compendiumdev.restmud.testing</groupId>
<artifactId>restmud-api-testing</artifactId>
<version>${restmud-api.version}</version>
</dependency>
@EvilTester 49

50. What is a bot?
— A class that implements as a thread
public class ThreadedRestMudTestBot implements Runnable{
@EvilTester 50

51. And does stuff... randomly
public void run() {
botState(STARTED);
running=true;
try {
while(!shouldStop){
myBot.executeARandomActionStrategy();
// Let the thread sleep for a while.
myBot.executeARandomWaitingStrategy();
botState(LAST_EXECUTED);
}
}catch (InterruptedException e) {
botState(INTERRUPTED);
}
botState(EXITED);
}
@EvilTester 51

52. Stuff it does are implemented as "Strategies"
public class RandomDoorOpenerStrategy
implements RestMudBotStrategy {
Which are executed.
@Override
public RestMudResponseProcessor execute() {
...
}
Which use the existing abstractions.
@EvilTester 52

53. Patterns
— Command Pattern
— Strategy Pattern
— Screenplay Pattern
etc.
@EvilTester 53

54. A Simple re-use example
— Rather than re-use the 'tests'
— Re-use the abstractions
— Tests are often not abstracted enough to make
them re-usable
— too many preconditions
— too many post conditions to tidy up
Create Abstractions that are libraries, rather than
frameworks.
@EvilTester 54

55. Basic Model Based Testing
— @Test code traditionally models path through
system
— Abstractions help introduce equivalence
randomness
— Strategy and Command Patterns help model at a
high level
— treat paths as data (randomly choose path)
@EvilTester 55

56. Test Automation Rules
— Don't automate unstable applications.
— Don't "just knock up code", Test Code is
Production Code
— Test Automation Takes a long time
— Tests should be short. Long running automation is
flaky.
@EvilTester 56

57. More Test Automation Rules
— Don't automate systems you haven't tested.
— Test Automation does not find bugs.
— Automate for the long term, not short term.
— Test Automation ROI - is it worth the cost?
@EvilTester 57

58. What if "Rules" were "Heuristics"
We have to know when they apply and when they
don't.
— We are not "breaking rules".
— We are making contextual decisions.
@EvilTester 58

59. Why do we automate?
— to save time
— to save money
— to increase coverage
— to allow testers to do something more valuable
instead
— etc.
Can you achieve any of these benefits quickly?
@EvilTester 59

60. Automating is Driven by
ideas
and supported by tooling.
@EvilTester 60

61. Don't let tools. Dictate
your Automating.
@EvilTester 61

62. Think Differently About
Automating
Automate to Augment Testing
@EvilTester 62

63. Alan Richardson
— @EvilTester
— EvilTester.com
— github.com/eviltester
— compendiumdev.co.uk
— digitalonlinetactics.com
@EvilTester 63