Доклад Пола Геррарда на конференции SQA Days-20. 24-26 ноября 2016. Минск www.sqadays.com

1. A New Model for Testing
@paul_gerrard
Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com

2. Helping clients transform their testing through
INNOVATION, COACHING and LEADERSHIP
Our CLIENTS
– Want to be agile rather than follow Agile dogma
– Have a pragmatic approach and are focused on delivery
– Want a solution that fits, not a badly fitting suit.

3. Agenda
• Innovations that Drive Change
• 3 Patterns of Software Development?
• A New Model for Testing
• Some Consequences
• Some Research
• A New Test Process?
• Close

4. What's happening?
• Stampede to mobile computing - Digital
• Big Data
• Internet of (Every)Thing, Pervasive Computing
• Continuous Delivery, DevOps
• (Test) Analytics, Data-Driven Development
• Shift-left, embedded testers, no test team
• Did you really think Agile was the last word?

5. Three Innovations That
Are Driving Change
Being agile
Shift Left
Analytics
(Digital/IoT/IoE and others too, of course)

6. #1
Agile doesn’t work
but being agile might
Agile is no longer "innovative"
Challenging Agile and being agile
might be
Is continuous delivery for you?
Have you
been Agiled?

7. #2
Shift Left
Developers in test?
Testers in development?
Automation, analytics, collaboration, experimentation

8. #3
Test Analytics
Shift-Right?
Analytics from production
Analytics from test
An opportunity for testing

9. Three Software
Development Patterns?
A Model

10. There are three patterns, not just
two
• Many ‘new’ approaches are labelled 'Agile' but
– They aren’t Agile, they just work
– Aimed at factory workers, not knowledge workers
• Continuous-delivery approach is appropriate
for rapid-delivery into online/mobile systems
• CD advocates often have an Agile background,
so you might think CD is Agile
• Not so.
Intelligent Definition and Assurance Slide 10

11. Three development patterns
Intelligent Definition and Assurance Slide 11
Structured
Agile Continuous
Autonomous

12. Key pattern characteristics
Characteristic Summary
Structure Structure of the project team
Pace/cadence What drives the rate of decisions making?
Leadership How is the team managed/directed?
Definition In what format is requirements knowledge captured?
Testing How is testing (mostly) performed?
Auto. Test When are automated tests created? Who leads?
Measurement What/how is project measurement performed?
Governance What form does governance take?
Intelligent Definition and Assurance Slide 12
All models are WRONG, some are USEFUL
Needless to say, I’ll be generalising quite a lot 

13. Profiles of the three patterns
Characteristic Structured Agile Continuous
Structure Managed team Autonomous Production Cell
Pace/cadence Business decision Team decision Feedback
Leadership Project Managed Guided Research Line Managed
Definition Fixed spec Dynamic spec Live Specs
Testing Scripted Exploratory Automated
Auto. Test Retrospective Developer led Pervasive
Measurement Pervasive Avoided Analytics
Governance Bureaucratic Trust-based Insight-Driven
Intelligent Definition and Assurance Slide 13

14. There aren't three
patterns
There are many

15. From high process to DIY process
• In the past, high process was seen to be the
technical solution to the software problem
• But technical solutions cannot solve human
problems
• True agility means a team can adapt or select
its process on-the-fly
• We need to strip down our thinking to the
essential aspects…

16. The old ways won't
work in the future
We need a New Model of Testing
(free from logistics)

17. Forget Logistics
(for the time being)
Document or not?
Automated or manual?
Agile v waterfall?
This business or that business?
This technology v that technology?

18. ALL Testing is
Exploratory
We explore sources of knowledge ...
... to build test models ...
... that inform our testing.

19. Examples of models

20. Models are innate, essential, human

21. Judgement, exploring and testing
Testing
(the system)
Our model(s) are adequate
Our model(s) are not adequate
Exploring
(sources)
Judgement
Creates test
models
Uses test
models
We explore sources of knowledge to build test models that inform our testing
BTW – Do Developers explore the same way? I think so.

22. Exploration process
ExplorationDefinitions
specs/stories
People
(& you)
Sources
Require-
ments
Test
Models
Enquiring
Challenging
Sources:
People, documents,
experience, system under test
Modelling
Test Models:
Can be documented
or mental models
Predicting
System
under test

23. Testing process
Testing
System
Under Test
Refining
Informing Applying
Interpreting
Test
Models
Revise the
System
More exploring Reporting

24. New ModelTesting
My BBC talk: http://www.bbc.co.uk/academy/technology/article/art20150522113029398
29 page paper: http://dev.sp.qa/download/newModel

25. Some Consequences
There are others

26. Relation to
TDD and BDD?
Is TDD really testing
BDD is modelling using stories

27. Test automation from a
different perspective
Automation efforts fail too often
Automation uses different test models

28. Capabilities
Enquiring, Modelling, Predicting, Challenging
Informing,Applying, Interpreting, Refining
Reporting and Logging

29. • Analysis, enquiry and elicitation
• Modelling
• Creation of custom models, using
heuristics, guesses, brainstorming,
ideation, creative thinking
• Custom test design techniques
• Comparison of models, value, advantages,
disadvantages, compromises
• Identification, validation and use of
oracles
• Predicate logic and proof
• Hypothesis and inference
• Socratic method
• Rapid Review and Inspection techniques
• Test case design
• Test models and the meaning of coverage
• Testing as controlled experiment
• Observation, Note taking, recording
A very different skillset
• Basic data analysis and statistics
• Decision-making with incomplete data
• Computer forensics
• Fault tree analysis
• Failure diagnosis
• Bug advocacy, triage processes and
negotiation
• Meaningful software and test metrics
• Visual presentation of data
• Reporting and presentation skills
• Understanding stakeholders
• Test analytics
• Risk management, risk-based testing and
decision-making
• Critical Thinking
• Interpersonal skills
• Dealing with uncertainty/fallibility

30. Testing Career Development
(speculative)
Foundations
Technical
Management
Strategic
Test Strategy Project
Intelligence
Test Assurance
Exploration Forensics Interpretation
Scripting/
Programming
Test Automation Technical (Excel,
SQL, OS utils etc)
Stakeholder
management
Analytics &
visualisation
Managing
uncertainty
Critical Thinking
ISTQB, TMMi etc...
Supplier
Management
Test Process
Management
Methodology

31. Will Robots Replace
Testers?
Some research
There is a paper at:
https://tkbase.com/resources/viewResource/14

32. A recent study*…
• Over the next two decades, 47% of jobs in the
US may be under threat
• It ranks 702 occupations in order of their
probability of computerisation
– Telemarketers: 99% likely
– Recreational therapists: 0.28% likely
– Computer programmers: 48% likely
• Something significant is going on out there
• If programmers have a 50/50 chance of being
replaced by robots, we should think seriously on
how the same might happen to testers.
* “The future of employment: how susceptible are jobs to computerisation?”
http://www.oxfordmartin.ox.ac.uk/downloads/academic/The_Future_of_Employment.pdf
Intelligent Definition and Assurance Slide 32

33. Some systems-related occupations
Intelligent Definition and Assurance Slide 33
Occupation Rank (out of
702)
Probability of
Computeris-
ation
Computer and Information Research Scientists 69 1.5%
Network and Computer Systems Administrators 109 3.0%
Computer and Information Systems Managers 118 3.5%
Information Security Analysts, Web Developers,
and Computer Network Architects
208 21%
Computer Occupations, All Other 212 22%
Computer Programmers 293 48%
Computer Support Specialists 359 65%
Computer Operators 428 78%
Inspectors, Testers, Sorters, Samplers and
Weighers
670 98%

34. Test Automation =
Mechanical Tools
What we REALLY need are
THINKING TOOLS

35. The term Test Automation misleads
• It misleads as a label because the whole of
testing cannot be automated
• The label is bad, but the scope of Test
Automation is what I call ‘Applying’ in the
New Model
Intelligent Definition and Assurance Slide 35

36. Testers need ThinkingTools
• There are ten testing activities in the New Model
– Test automation tools only support one:‘Applying’
• The remaining nine activities (information
gathering, analysis, modelling, challenging, test
design and so on) are not well supported
• All require some level of thinking and skills
• Checking is possible when a system and its
purpose are well understood and trusted
• Test automation tools are simple in principle…
… compared to the rest of the test process.
Intelligent Definition and Assurance Slide 36

37. Four quadrant model of intelligent
test tools
Ability to Investigate
AbilitytoCaptureKnowledge
• Text editors, Screen Shots
Models, visualisations, relationships, transformations
• Note Takers
• Mind Maps
• UML/Case Tools
Control,imagination,discernment,foresight
• Pencil and paper, sketching tools
Intelligent Definition and Assurance Slide 37

38. Intelligent Definition and Assurance Slide 38
TERMINATOR
TESTER
Not Yet!

39. Cervaya™
Tool Architecture
A nine month bot journey
(But it’s been a twenty year testing
journey so far)
https://huddle.eurostarsoftwaretesting.com/resource/test-
management/new-model-testing-new-test-process-tool/

40. The vision thing
• I want a bot partner/pair that supports
exploratory testing
• You know my view (model)
of testing already
• Can we use the explore v test paradigm in a bot
that allows you to:
– Explore, take notes and model
– Record ideas, risks, tests, outcomes and bugs
– Generate reports and documentation as a
consequence, rather than requiring you write them
• Codename: Cervaya (cervaya.com is a holding page for now)
Intelligent Definition and Assurance Slide 40

41. Schematic
Intelligent Definition and Assurance Slide 41
Schema Server
Schema
Manager
(web site)
Schema
Repository
(web service)
Cervaya Bot Client
Robot Engine
Cervaya Server
Actions
(web services)
Application
Web Site
Speech
Recognition
Interface
Command
Line
Interface
Administer Schemas
through the Web
Download
Schema Perform Robot
Actions through
services
Application Reporting,
Monitoring, Control,
Management
Many App Servers – one
(or many) per Schema
robotschemas
cervaya

42. The target user is…
• (Not really a freestyle exploratory tester
working in a start-up or small product company)
• Target users are testers working in:
– Regulated, high-integrity, safety critical
– High documentation, high accountability
– Environments where testers are constrained by their
processes
– Testers struggling to apply ‘trad’ methods in an Agile,
Digital, DevOps environment
• Why can’t we dictate a detailed test plan for a
bot to document, analyse, visualise?
Intelligent Definition and Assurance Slide 42

43. The location hierarchy
• Explorers create maps (or should do!)
• The system model/map is a hierarchy of locations
as follows:
• Application
– Places
• Features
– Forms
» Fields
• All other content is located with respect to some
level and location in the system model.
Intelligent Definition and Assurance Slide 43

44. A New Test Process

45. Exploration support
• Frustration:
– testers are unimaginative, working by-rote
– constant pressure to cut costs
• Productivity of exploratory testing is proven
• Testers want to explore, but the need for control and
documentation constrains them
• Testers needs tools that can capture plans and tester
activity in real-time
• The next generation will be led by tools that support
the exploration of sources of knowledge
• These tools might use a “Surveying” metaphor.
Intelligent Definition and Assurance Slide 45

46. Staged Process
System Testing Re-Test
Big, up-front Test
Planning
Regression
Test
Development
Automation
From Staged to Continuous
Scoping exploration sessions
Interactive Testing Sessions
Clarifications
Continuous Integration
Changes
Refinements
Continuous Dev, Test and Delivery
Bot Supported
Bot Supported

47. Speech recognition, text to speech
• Right now, I’m using:
– Google Speech Recognition (a slow web service)
– pyttsx a PythonText to Speech module (depends on
pywin32-extensions for Windows)
• I’m looking at Amazon Alexa as an alternative
– https://www.youtube.com/watch?v=hPXS7rC1PWo
– Alexa is the web service, Echo Dot is the latest device
that captures and synthesises voice
– It promises to be a lot faster (and it works better).
Intelligent Definition and Assurance Slide 48

48. Real-Time test management
• Once you have a shared system model:
– The activity of the team can be visualised in real-time
– Tester activity: visible – testers liaise when necessary
– Test paths through the system map can be traced
• Developers can also see progress and requests
for more information in context:
• Cervaya has an integration with Slack:
– Model changes, additions, queries, test failures can be
posted to Slack/ChatOps service.
Intelligent Definition and Assurance Slide 49

49. From Staged to ContinuousTesting
• Waterfall projects create, execute tests in stages
• Shifted-left testers can:
– Model features as they are created by the developers
– Create accurate tests a low level as the functionality
emerges over time
– End to end tests use low-level tests as building blocks
• Tests and test run histories are manage against
versions (and platforms/environments soon)
• The aim: to export tests as soon as practical to
be automated to provide continuous feedback.
Intelligent Definition and Assurance Slide 50

50. Testing Styles,Approaches
“Structured/Waterfall/
Staged”Testing
• Systematic
• Transparent, Documented
• Reviewable
• Auditable
• Repeatable, measurable
• Automatable
• Inflexible, not responsive
• Obsolescent/inaccurate
documentation
• Prone to biases, inattention
• Outdated process
• Expensive, Inefficient
• Unimaginative, boring
“Exploratory”Testing
• Agile (with a small ‘a’)
• Improvised, imaginative
• Flexible and responsive to change
• Faster, cheaper
• More effective
• Personally enjoyable
• Not repeatable
• Not easily automated
• Little or no documentation
• Hard to manage
• Hard to scale
• Opaque
• Not auditable, measurable.
Intelligent Definition and Assurance Slide 51
We need to change
the testing metaphor

51. Surveying and testing – the new
process?
• The “tester as surveyor” affects the relationship
of testing to development
• A new style of testing process emerges:
– No more documentation created in a knowledge
vacuum
– Iterative, incremental knowledge capture closely
aligned with the emergent delivery of functionality
• Could this be an Agile test process at last?
• It fits the increasingly popular Continuous
Delivery, DevOps development approaches.
Intelligent Definition and Assurance Slide 52

52. Close

53. New Model Testing
changes what testers &
developers need to
learn
Roles of developers/testers become blurred
Perhaps they are not different disciplines after all?
Why have a tester certification?

54. Do Testers Need to
Learn How to Code?
Some, perhaps many will have to

55. Testers don’t own
testing anymore
Be aTestMaster
"Assurance through the lifecycle"

56. The phase after
development is
REWORK, not Testing
Testing must align with development,
not compete with it or rescue it

57. A New Model for Testing
@paul_gerrard
Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com