Quality engineering & testing in DevOps IT delivery with TMAP

Quality Engineering
& Testing
in DevOps IT delivery
21 October 2021
Rik Marselis
TMAP: the body of knowledge for
quality engineering in IT delivery

3
Quality Engineering & Testing in DevOps with the TMAP body of knowledge © 2021 Sogeti. All rights reserved.
Challenges of today’s high-performance IT delivery
The business demands:
• Deliver business value
• Deliver quality at speed
The team challenges are:
• Quality engineering is everyone's responsibility
• QA & testing is integrated in process and people
The focus is:
• Organize high-performing cross-functional teams (you build it, you run it!)
• Automate everything (as long as it is useful)

4
The starting point of organizations is to deliver business value

5
Supply information
based on
measuring indicators
Our “VOICE model” for delivering business value
Measuring indicators
=
(mainly) testing
Figure 3.1 from the book Quality for DevOps teams

6
Examples of indicators in the VOICE model
TMAP defines four groups of indicators, per group you find a few examples below (more in book)
You must find indicators that show whether you are moving towards the pursued business value.
Business value related indicators
- Customer satisfaction
- Number of service calls to helpdesk
IT delivery related indicators
- Business features done
- Percentage of availability (up-time)
Team related indicators
- Availability of necessary skills
- Satisfaction and happiness of team members
Problem related indicators
- Number of anomalies registered compared to expected
- Mean time to fix operational failures
And keep in mind:
a few well-measured and properly
followed-up indicators are much
better than a long list of unpractical
indicators.

7
Waterfall V-model Scrum DevOps Demand-Supply SAFe®

8
DevOps – highlights
The six DevOps principles:
1. Customer-centric action
2. Create with the end in mind
3. End-to-end responsibility
4. Cross-functional autonomous teams
5. Continuous improvement
6. Automate everything you can
DevOps is a cross-functional systems engineering culture that aims at unifying
systems development (Dev) and systems operations (Ops) with the ability to create and
deliver fast, cheap, flexible and with adequate quality, whereby the team as a whole is
responsible for the quality. Other areas of expertise, such as business analysis and quality
assurance (including testing) are usually integrated in the team. A DevOps culture has an
Agile mindset that can be supported/implemented by e.g. the Scrum framework.
The six fundamental DevOps activities

TEAM:
Together
Everyone
Achieves
More
Example of a cross-functional
high-performance IT delivery team
The starting point for DevOps:
A cross-functional team
Aligned with the DevOps culture of IT delivery – where roles are important and functions
are not – keep in mind that if in TMAP you see a term such as developer or tester, we
refer to the role of a team member at a specific moment in time, not to a function.

Quality Engineering is about team members and their stakeholders
taking joint responsibility to continuously deliver IT systems
with the right quality at the right moment
to the businesspeople and their customers.
It is a principle of software engineering concerned with
applying quality measures to assure built-in quality.

11
Quality Engineering & Testing support delivering business value
Quality Engineering
Testing
Focus:
“Detective”
quality measures
Focus:
“Preventive”
quality measures
Build quality in,
right first time
Provide information
about quality & risks

12
Definition of testing within quality engineering & business delivery
Testing consists of verification, validation and exploration activities
that provide information about the quality and the related risks,
to establish the level of confidence
that a test object will be able to deliver the pursued business value.

13
Approaches for test design
Focus on exploration
Focus on
verification & validation

14
Poll for the audience: which subjects would you like us to discuss?
Photo of the book launch event on 17 March 2020
From left to right: Rik Marselis, Dennis Geurts, Berend van Veenendaal and Wouter Ruigrok
This photo was edited to include Dennis who couldn’t be present at the actual launch
Go to: Menti.com

15
Select from the following subjects
QA & Testing topics and Test Design
Continuous everything: Monitoring & Control and Reporting & Alerting
CI/CD pipeline and Test automation
Unit testing and mutation testing
Closing

17
QA & testing topics
Figures 11.1 and 11.2 from the book Quality for DevOps teams

18
The QA & testing topics related to the DevOps activities
Organizing
topics
Performing
topics
DevOps
activities

19
Coverage-based testing with test design techniques
A test design technique is a standardized method of deriving test cases from a specific test basis
that will achieve a certain coverage.
Applying a test design technique results in test situations, logical test cases & physical test cases.
Our investigation has shown that there are about 25 to 30 test design techniques in literature.
All test design techniques can be categorized in the four coverage groups mentioned below.
Keep in mind that you don’t have to
know all existing test design
techniques.
Knowing a few techniques for each
coverage group makes that with just
about 8 techniques you can handle
any testing challenge.

20
16 test design techniques defined in TMAP
https://tmap.net/page/test-design and https://tmap.net/building-blocks/coverage
Figures 45.2, 45.3, 45.4 and 45.5 from the book Quality for DevOps teams

Monitoring & Control
Reporting & Alerting

22
Continuous everything to achieve built-in quality

23
Continuous improvement is about Product, Process and People
Product
People
Process

24
Monitoring, control, reporting & alerting

26
Automate everything (as long as it is useful)
An automated pipeline enables: elimination of errors due to manual task execution, provisioning
of standardized development feedback loops and enabling of fast product iterations.

27
Capabilities & tools
When talking about a CI/CD pipeline people often show all
the tools in the pipeline.
But tools are exchangeable and often change.
To set up a good pipeline it’s very important to define the
capabilities your organization needs from the pipeline.
Over time the capabilities needed will evolve, at the start
the pipeline will only cover a few capabilities, when maturity
grows the capabilities that are automated also grow.
Examples of capabilities are (more examples are described in section 6.3 of the book):
• Continuous Integration  code review
• Continuous Delivery  configuration management
• Continuous Deployment  deployment of the build application (container, package)
• Continuous Monitoring  user telemetry to measure user sentiment and behavior
• Continuous Quality & Testing  automated anomaly detection
Examples of tools in a pipeline

28
Organize your automated test varieties:
Testing pyramid and testing quadrants

Unit testing & Mutation testing

30
Unit testing
Keep in mind that the statement “we have reached
100% code coverage” in itself doesn’t give useful
information.
The type of code coverage achieved is what matters
Code coverage demonstrates the percentage of program code that is
covered by tests. Different test design techniques guarantee less or
more coverage.
The spheres of testing

31
Mutation testing tests the test
Mutation testing is a type of testing where certain statements in the source code are changed
(mutated) to check if test cases will identify the fault that was introduced this way.
This is a manner to verify the quality of the test set (instead of the object under test).
Mutation testing usually is supported by tools.
Mutation testing focuses on conditional statements.
Suppose we have the following code and test cases:
IF A > 10 PRINT “YES” ENDIF
Testcase1: A := 10, expected result no print
Testcase2: A := 11, expected result “YES”
What mutation in this code would not make any of these tests fail?
IF A <> 10 PRINT “YES” ENDIF  Testcase3: A := 9, expected result no print
IF A = 11 PRINT “YES” ENDIF  Testcase4: A := 12, expected result “YES”

33
We proudly present: Quality for DevOps teams
and the renewed body of knowledge www.tmap.net

34
TMAP training courses and certification scheme of
in partnership with
TMAP: Quality
for cross-
functional teams
TMAP: High-
performance
quality
engineering
TMAP:
Organizing built-
in quality
at scale
Performing
people
3-day training course
exam of 30 questions
3-day training courses
exams of 40 questions
All people
involved in
cross-
functional
teams
Organizing
people
Exams provided by
Info about TMAP certification at:
www.TMAPcert.com

35
TMAP:
body of knowledge for
quality engineering in
all IT delivery models
www.ICT-books.com

37
Stay up-to-date with the latest TMAP developments
Follow TMAP on LinkedIn:
www.linkedin.com/company/2020-tmap-quality-engineering-for-high-performance-it-delivery/
(use the link or just search for TMAP and the
world of quality engineering on LinkedIn)

38
About Sogeti
Learn more about us at
www.sogeti.com This message contains information that may be privileged or
confidential and is the property of the Capgemini Group.
Copyright© 2020 Sogeti. All rights reserved.
Sogeti is a leading provider of technology and engineering services. Sogeti delivers
solutions that enable digital transformation and offers cutting-edge expertise in Cloud,
Cybersecurity, Digital Manufacturing, Digital Assurance & Testing, and emerging
technologies. Sogeti combines agility and speed of implementation with strong technology
supplier partnerships, world class methodologies and its global delivery model,
Rightshore®. Sogeti brings together more than 25,000 professionals in 15 countries,
based in over 100 locations in Europe, USA and India. Sogeti is a wholly-owned subsidiary
of Capgemini SE, listed on the Paris Stock Exchange.
TMAP website:
www.tmap.net
Sogeti academy:
academy.sogeti.nl
TMAP:
body of knowledge for
quality engineering
Feel free to contact me on:
Rik.Marselis@Sogeti.com

39
TMAP: body of knowledge for quality engineering

Quality engineering & testing in DevOps IT delivery with TMAP

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Quality engineering & testing in DevOps IT delivery with TMAP

Similaire à Quality engineering & testing in DevOps IT delivery with TMAP (20)

Plus de Rik Marselis

Plus de Rik Marselis (20)

Dernier

Dernier (20)

Quality engineering & testing in DevOps IT delivery with TMAP