Доклад Gualtiero Bazzana на конференции SPM Conf-5, 6 ноября 2015 г., Минск www.spmconf.ru

1. The Business Value
of SW Quality
Minks, November 2015
G. Bazzana – ISTQB® President

2. ISTQB® 2015 2
• Quantifying the impacts on sw engineering practices on Sw
Project productivity
– Approach
– Quantitative Data
• The importance of testing
– Good practices
– Certification of sw testing competencies
– ISTQB®
Contents

3. ISTQB® 2015 3
Drivers
COST
QUALITY
RISK
AGILITY
• Quicker time
to market
• Consumption
based pricing
• Increasing
dependency of
business from SW
• Multiple platforms
• SW Intensive
systems
• Knowledge
retention
• Competition
• Tightening
margins

4. ISTQB® 2015 4
APPLICABLE MARKETS
IT
Engineering
Telco

5. ISTQB® 2015 5
• We need to provide evidence of quantitative benefits
• In God we trust, all others must bring data
Identifying «Best Practices»
Metrics
Benchmarks
ROI Analysis

6. ISTQB® 2015 6
BASIC DEFINITIONS OF SOFTWARE QUALITY
• Functional Software Quality
Software that combines low defect rates and high levels
Of user satisfaction. The software should also meet all
user requirements and adhere to international standards.
• Structural Software Quality
Software that exhibits a robust architecture and can operate
In a multi-tier environment without failures or degraded
performance. Software has low cyclomatic complexity
levels.
• Aesthetic Software Quality
Software with elegant and easy to use commands and
Interfaces, attractive screens, and well formatted outputs.

7. ISTQB® 2015 7
ECONOMIC DEFINITIONS OF
SOFTWARE QUALITY
• “Technical debt”
The assertion (by Ward Cunningham in 1992) that
quick and careless development with poor quality leads
to many years of expensive maintenance and enhancements.
• Cost of Quality (COQ)
The overall costs of prevention, appraisal, internal failures,
and external failures. For software these mean defect prevention,
pre-test defect removal, testing, and post-release defect repairs.
(Consequential damages are usually not counted.)
• Total Cost of Ownership (TCO)
The sum of development + enhancement + maintenance +
support from day 1 until application is retired.
(Recalculation at 5 year intervals is recommended.)

8. ISTQB® 2015 8
World Quality Report Findings

9. ISTQB® 2015 9
• Budgets dedicated to QA&Testing are growing at warp speed, with a growth
pace much higher than anticipated, reaching in average 35% of the IT
budgets
• Test automation is increasing, jumping in one year from 28% to 45% of test
cases
• Digital transformation is putting more and more emphasis on customer value
and end-user experience as targets for testing
• Security is the most pressing concern
• Agile and DevOps adoption has become widespread and call for extreme
level of speed and integration in testing practices; a lack of professionals
with Agile Testing expertise is among the top-three challenges
• Mobile testing is maturing, being adopted by 92% of the organizations and
consuming 35% of the QA&Testing Budgets for new projects
• Testing Centers of Excellence are widely adopted and are turning into more
agile and hybridized forms
• Test environments are being enhanced with virtualization and cloud solutions
HIGHLIGHTS FROM THE
2015 WORLD SW QUALITY REPORT

10. ISTQB® 2015 10
• Software engineering and software project management are complex
activities. Both software development and software management have dozens
of methodologies and scores of tools available that are beneficial. In addition,
there are quite a few methods and practices that have been shown to be
harmful.
• In order to evaluate the effectiveness or harm of various methods and
practices Capers Jones has developed a scoring method.
• Data collected from 1984 through 2015
– About 725 companies (150 clients in Fortune 500)
– About 40 government/military groups
– About 25,000 total projects
– New data = monthly benchmark studies
– Data collected from 27 countries
– Observations during more than 17 lawsuits
• The analysis is based on the author’s book Software Engineering Best
Practices published by McGraw Hill in 2010. Some new data is taken from
The Economics of Software Quality published by Addison Wesley in 2012.
QUANTITATIVE BENCHMARKS
(Based on work of Capers Jones)

11. ISTQB® 2015 11
Certified reusable components 80%
Experienced development teams 65%
Effective methodologies for specific project types 40%
High-level programming languages 30%
Use of inspections for complex systems 27%
Experienced managers 25%
Moderate unpaid overtime by teams 20%
Impact of Critical Factors on
Software Productivity . POSITIVE (1)

12. ISTQB® 2015 12
Low requirements creep 20%
Logical, planned architecture for large systems 20%
Model-based development 20%
Due diligence on COTS acquisitions 20%
Use of static analysis before testing 18%
High CMMI levels 15%
Low cyclomatic complexity (< 10) 15%
Effective project status tracking 15%
Effective defect prevention 15%
Experienced test teams 12%
Impact of Critical Factors on
Software Productivity . POSITIVE (2)

13. ISTQB® 2015 13
Experienced clients 10%
SCRUM < 1000 function points 10%
24-hour continuous development 7%
Effective parametric estimating tools 7%
Testing by certified test personnel 7%
Annual staff training > 5 days 7%
Formal mathematical test case design 6%
Co-located teams 5%
Impact of Critical Factors on
Software Productivity . POSITIVE (3)

14. ISTQB® 2015 14
High requirements creep: poor change control -60%
Inexperienced managers -50%
Truncating testing to "meet schedule" -45%
Concealing problems in status reports -45%
Inexperienced clients -40%
Chaotic, unplanned architecture for large systems -37%
Inexperienced development teams -35%
Inaccurate manual estimates -33%
Concurrent maintenance and development tasks -30%
False claims by outsource vendors -30%
Impact of Critical Factors on
Software Productivity – NEGATIVE (1)

15. ISTQB® 2015 15
Adding personnel to late projects -25%
Low-level programming languages -25%
Excessive unpaid overtime by team -23%
Manual estimates > 1000 function points -23%
Poor status tracking -20%
Unverified, buggy COTS acquisitions -20%
High cyclomatic complexity (> 25) -18%
Ineffective methodologies -15%
Inexperienced test teams -15%
Distributed teams: poor communications -15%
Impact of Critical Factors on
Software Productivity – NEGATIVE (2)

16. ISTQB® 2015 16
Waterfall > 5000 function points -12%
Low CMMI levels -10%
Agile > 5000 function points - 9%
Informal test case design - 8%
No annual staff training - 7%
Testing by developers only - 6%
Impact of Critical Factors on
Software Productivity – NEGATIVE (3)

17. ISTQB® 2015 17
CONCLUSIONS ON SOFTWARE QUALITY
• No single quality method is adequate by itself.
• Inspections + static analysis + formal testing > 99% efficient.
• Defect prevention + pre-test removal + formal test best overall
• Higher CMMI levels, TSP, RUP, Agile, XP are effective
• Quality excellence has ROI > $15 for each $1 spent
• High quality benefits schedules, productivity, users!
• Poor quality leads to cost and schedule overruns!

18. ISTQB® 2015 18
USEFUL AND HARMFUL QUALITY METRICS
• Useful quality metrics
– Defect potentials using function points
– Defect detection efficiency (DDE)
– Defect removal efficiency (DRE)
– Delivered defects per function point
– Defect removal cost per function point
– Cost of quality (COQ)
– Cyclomatic complexity
– Test coverage

19. ISTQB® 2015 19
US AVERAGE FOR SW QUALITY AS OF 2015
Defect Removal Delivered
Defect Origins Potential Efficiency Defects
Requirements 1.11 88.00% 0.13
Architecture 0.25 94.00% 0.02
Design 1.20 93.00% 0.08
Coding 1.30 97.00% 0.03
Documents 0.50 93.40% 0.04
Bad Fixes 0.35 85.00% 0.07
TOTAL 4.71 92.57% 0.35
(Data expressed in terms of defects per function point)
(Function points show all defect sources - not just coding defects)
(Code defects = 35% of total defects)

20. ISTQB® 2015 20
RANGES OF DEFECT REMOVAL EFFICIENCY
Worst Median Best
Defect Potentials 1,100 1,000 900
Defect Prevention 40% 50% 60%
Pre-test inspections 70% 85% 90%
Pre-test static analysis 40% 65% 75%
Unit tests 20% 25% 35%
New Function tests 25% 35% 45%
Regression tests 15% 25% 35%
System test 35% 45% 55%
Acceptance/Beta tests 20% 30% 40%
DELIVERED DEFECTS 53 18 2
REMOVAL EFFICIENCY 95% 98% 99.7%

21. ISTQB® 2015 21
ROI FROM STRUCTURED TESTING
Requirement Analysis Development Test Production
5 5 % 20 % 40 % 30 % < 5 %
4 3 % 12 % 30 % 50 % 5 %
3 0 % 2 % 20 % 70 % 8 %
2 0 % 0 % 3 % 80 % 17 %
1 0 % 0 % 2 % 50 % 48%
FAULT DISTRIBUTION
1 5 20 50 100
TESTMATURITY

22. ISTQB® 2015 22
ROI FROM STRUCTURED TESTING
 With no structured test process, 50% of the overall issues are
found during the test phase.
 With a structured test it is reasonable to expect to find 80% of
the issues during the test phase.
 Bug fixing cost if the problem is found during the test phase: 2
md
 Bug fixing cost if the problem is found after release: 4 md
 A medium sized application generates overall 500 issues
(both test and production).
 Suppose to reserve a testing team for an effort of 180 md,
then…

23. ISTQB® 2015 23
ROI FROM STRUCTURED TESTING
RESULTS:
 Issues in production (after release) are reduced by 60%
 The overall effort (test + bug fixing) is reduced by 8%
Without
Structured Test
With
Structured Test
Number of faults found during test 250 400
Effort to fix during test 500 800
Number of faults found in production 250 100
Effort to fix faults found in production 1000 400
Effort for structured testing 0 180
Overall effort 1500 1380

24. ISTQB® 2015 24
• In the following some factors related to testing are presented, with their impact
in terms of reduction or increase in project work hours wrt to projects
consisting of traditional application development methods such as “waterfall”
development performed by organizations that do not have apply systematically
sound software engineering practices
INCREASING PRODUCTIVITY OF SW PROJECTS
GOOD TESTING PRACTICES CAN HELP
Factor Impact on productivity
Experienced test teams 12% Improvement
Testing by certified test personnel 7% Improvement
Testing by developers only 6% Worsening
Informal test case design 8% Worsening
Inexperienced test teams 15% Worsening
Truncating testing to "meet schedule" 45% Worsening
(Capers Jones, Scoring and evaluating software methods,
practices, and results, version 15.0, September 6, 2014)

25. ISTQB® 2015 25
• ISTQB®: International Software Testing Qualifications Board
(www.istqb.org):
• - Non-profit association
– Founded in 2002
– Has its own constitution, rules and regulations
– Composed of volunteer international Testing Experts
– Responsible for the “ISTQB® Certified Tester” scheme worldwide
• ISTQB® is the world’s leading organization for Software Testing Certification
WHAT IS ISTQB®?
Advancing
the software testing profession

26. ISTQB® 2015 26
ISTQB® - Current Product Architecture

27. ISTQB® 2015 27
“To continually improve and advance
the software testing profession by:
Defining and maintaining a Body of Knowledge
which allows testers to be certified based on best practices,
connecting the international software testing community,
and encouraging research.”
ISTQB® VISION

28. ISTQB® 2015 28
ISTQB® VIDEOS
• ISTQB® Videos give you insights into the ISTQB® Certified Tester scheme
http://www.istqb.org/introduction-to-istqb.html

29. ISTQB® 2015 29
WORLD-WIDE COVERAGE
 Countries covered by Member Boards (49 Member Boards covering 72 countries, representing
over 90% of the world-wide GDP) and Global Exam Providers
 Countries covered by Global Exam Providers
The list of Member Boards and Global Exam Providers is available on the ISTQB® Web Site

30. ISTQB® 2015 30
More than
560.000
exams world wide!
Figures as of 2015Q2
Executive Summary
In 2015Q2, 18993 exams
and 14036 certificates
Trend YOY is OK
Wrt FL+AL+EL  + 8,5%
Adding-up Agile 
+ 14,5%
Close to
410.000
certificates world wide!

31. ISTQB® 2015 31
Agile uptake
VERY Good:
- Almost 2000 exams
- 1600 certificates
- By 33 Boards
- Forecast for 2015:
over 4000 exams
Figures as of 2015Q2
Executive Summary (2)
Certificates issued in 108 countries

32. ISTQB® 2015 32
ISTQB® ECO-SYSTEM

33. ISTQB® 2015 33
• International recognition of acquired competencies
and skills
• Authorized to use the “Certified Tester” logo
(specifying the level of certification)
• Whole career path support, from Foundation to
Expert level
• Higher appeal in the labor market
BENEFITS FOR PROFESSIONALS

34. ISTQB® 2015 34
BENEFITS FOR PROFESSIONALS - SURVEY
• Would you recommend the ISTQB® Foundation Level (CTFL) certification
to your colleagues?

35. ISTQB® 2014 35
BENEFITS FOR COMPANIES
• ISTQB® certification can provide a competitive
advantage for companies, promising a higher level of
reliability of the applications being developed due to
efficient and cost effective testing practices derived
from the ISTQB® competencies
• Consulting companies with certified staff can offer
higher-level services to customers, increasing
revenues and brand value
• ISTQB® has defined a “Partner Program” for
companies that engage a large number of certified
testers

36. ISTQB® 2014 36
BENEFITS FOR ATPs
(Accredited Training Providers)
• Educational institutions and consulting companies
may become an ISTQB® Accredited Trainer Provider
(ATP) according to processes and rules defined at the
international level
• Accredited Training Providers ensure a high standing
of training is delivered through having:
• certified trainers
• the content, quality and syllabus coverage of
training materials checked by ISTQB® Boards
• advance notice of changes to the ISTQB®
Glossary and syllabi
• Accredited Training Providers are entitled to use
relevant logos and are listed in the ISTQB® Website

37. ISTQB® 2015 37
• The exam is based on multiple-choice questions for Foundation and Advanced Levels
• For Expert Level, an essay is also required
• Exams can be attempted also without having attended a training course (e.g. through
self-study)
EXAMS

38. ISTQB® 2015 38
• Questions are classified according to the cognitive level, the K-level
(also known as level of knowledge):
• K1 = Remember (recognize a term or concept)
• K2 = Understand (able to explain a concept)
• K3 = Apply (select correct application of concept or technique)
• K4 = Analyze (can distinguish between facts and inferences for deeper
understanding)
• K5 = Evaluate (can make judgments based on criteria and standards)
• K6 = Create (can put elements together to form a coherent or functional whole)
• The number of questions for each topic is related to the length of the
topic in the syllabus
• For more details, see the FAQ section in the ISTQB® Website
EXAM QUESTIONS – K LEVELS

39. ISTQB® 2014 39
Provides recognition to
Companies
that are investing in the
ISTQB® scheme
ISTQB® PARTNER PROGRAM

40. ISTQB® 2015 40
ISTQB® Planned evolution
 ISTQB® is evaluating an evolution of its product portfolio
architecture and contents in order to:
 Maintain the mission and vision of ISTQB® and keep the high quality
of deliverables that has marked the success of the scheme world-
wide
 Make the scheme more modular
 Make it easier for professionals to obtain the certification they are
interested in
 Provide an overall framework in which all future potential modules
may fit in a way which is coherent and understandable to our
stakeholders
 Maintain the validity of certifications already obtained

41. ISTQB® 2015 41
The customer perspective
 Some people like to follow a breadth-first generalist approach (going
from Foundation to the “classic” Advanced)
 Whereas others prefer to follow a depth-first approach (going from
Foundation to Specialist)
 We have so many different professionals interested in our programme
that we cannot think we can propose a “one size fits all” approach …
 Indeed, it is exactly the goal of a modular approach to allow people to
design their own training/ certification path and allow them to broaden
their knowledge base

42. ISTQB® 2015 42
Evolution Axes
 The review of the product architecture and product
portfolio embraces three aspects
1. The overall framework in which the various certification
modules have to fit
2. The modules that populate the framework: fitting the current
modules (the existing ones and those under development) and
identifying/ positioning the potential new ones
3. The preconditions/ pre-requisites/ entry criteria

43. ISTQB® 2015 43
Overall Framework/ architecture
Levels and categories
 The future ISTQB® Portfolio will be a Matrix
characterized by:
 Levels (FL, AL, EL): they identify progressively increasing
LOs
 Categories: they identify differing target groups of
certification modules:
 Core
 they cover a topic in a broad, horizontal way,
 valid for any technology/ methodology/ application
domain
 Allowing for a common understanding
 Agile
 So important in new developments
 Specialist
 they cover a specific topic/ domain in a vertical way,

44. Let’s Make a Difference…
Let’s support our
Customers…
THANK YOU!