Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Software Testing Foundations Part 7 - Basics of Test Management
1. Software Testing Foundations #7
Basics of Test Management
Nikita Knysh
nknysh@gmail.com
http://www.facebook.com/groups/istqb/
2. Agenda
• Test independence, Testing Roles
• Prioritization of Tests, Test Exit Criteria
• Cost of Defects, Cost of Testing, Test Estimation
• Test Strategy and Planning
• Approaches to Test Design and Test
• Risk Management
• Test Activity Management
• Incident Management
• More on Test Management
3. Options for Independent Testing
• Developer tests code by another developer (component,
integration test)
• Testers in the development team (component,
integration test of small projects)
• Testing teams within the project team (integration test of
large projects, system test)
• Independent testers for specific testing tasks
(performance, security, etc.) (integration test of large
projects, system test)
• Separate organization is responsible for testing
(integration test of large projects, system test)
4. Testing Roles
• Test manager (test leader)
• Test designer (test analyst) – analyze requirements,
write test specs, prepare & acquire test data
• Test automator
• Test administrator – maintain test environment
• Tester – review & execute tests, log & evaluate
results
5. Prioritizing Test Cases
Criteria for prioritizing:
• Usage frequency (probability of failure)
• Risk of failure (failure probability and severity)
• Visibility of failure to end user
• Priority of (functional) requirements
• Importance of (non-functional) quality characteristics
• Importance of system components
• Complexity of individual components
• Project risk (try to find failures that are hard to fix first)
Defect-prone test objects: where many faults have been
found before, more are present.
6. Test Exit Criteria
• Test coverage (requirement covered, test cases run
successfully)
• Product quality (number and breakdown of faults
found)
• Residual risk: not executed tests, not fixed bugs,
incomplete coverage
• Economic constraints
7. Cost of Defects
• Direct cost
▫ Arises to the customer due to failures during operation
• Indirect costs
▫ Arises to vednor in form of penalties, loss of
reputation, sales, etc.
• Defect correction cost. With every test level, the
correction costs for a fault double with respect to
the previous level.
8. Testing Cost Factors
• Maturity of development process
• Quality & testability of software
• Test infrastructure (env, tools, standards/procedures)
• Qualification of employees
• Requirements to quality
• Test strategy (the only one TM can influence &
control in short term)
9. Test Effort Estimation Approaches
• By test task owner or expert
• Based on experience & typical values
If no data: test tasks (on all levels) take 50% of overall
project resources.
10. Test Strategy
Defines the project's testing objectives and means to
achieve them.
The goal is to choose a test approach that optimizes
the relation between costs of testing and costs of
defects.
11. Approaches to Test Planning
• Preventive approach (like use of V-model) is way
better that reactive approach where testers are
involved (too) late.
• Analytical approach to test planning uses data and
(mathematical) analysis; heuristic approach relies on
experience of experts & rules of thumb (used when
data not available or modeling is too complicated).
12. Types of Approaches to Test Design & Test
• Model-based
• Risk-based
• Process- or standard-compliant
• Reuse-oriented
• Checklist-based (methodical)
• Expert-oriented
13. Risk Management
One of the best criteria when selecting and prioritizing
test objectives is risk.
Risk = probability (or frequency) * damage.
• Project risks threaten project’s capability to deliver
product.
• Product risks result from problems with delivered
product.
Risk management: assess, prioritize and mitigate (or
fight) risks.
14. Test Activity Management
• Test cycle planning (development status, test results,
resources). Result: (regression) test plan for
upcoming lifecycle.
• Test cycle monitoring. Metric types: fault- and
failure-based, test case-based, test object-based,
cost-based. Result: test status report (test progress
within the whole project) in the end of each cycle.
• Test (cycle) control – if testing is delayed test
manager takes actions like add resources, extend
test duration or adjust test plan.
15. Incident Management
• Test log analysis or documentation review -> registering
an incident if needed.
• Incident DB should have reporting and analysis tools.
• Incident report structure should be defined.
Incident classification: severity – impact to system, priority
– order of correction
Change control board accepts or rejects incident reports
and change requests in larger projects. Usually consists of
product management, project management, test
management, customer.
16. Configuration Management
A test process cannot be properly executed without
reliable configuration management!
Requirements to configuration management:
• Version management and configuration
identification in particular version
• Incident status and change status accounting.
IEEE 828 Standard for Software Configuration
Management Plans.
17. Sources of Standards
• Company standards
• Best practices
• Quality management standards
• Standards for particular industrial sectors
• Software test standards.
