Contenu connexe Similaire à Psqt east risk testing (20) Psqt east risk testing1. Risk Driven Software Testing
All
The Software
Processes
Prioritize
Document
& Analyze
Scope Tests Define Strategies
Gather Design
Requirements
Develop Tests Schedule
Manage Resources
Test and Report Allocate Resources
Fix Implement
Risk Driven Software Testing
1 PSQT East Tutorial v.1.0 ©TQ2003
3. Materials in this Tutorial
Review materials in student notebook
• job aids at front of notebook
• agendas
• slides
• exercise materials
• references
Establish parking lot on a flip chart page for topics to
handle later
Risk Driven Software Testing
3 PSQT East Tutorial v.1.0 ©TQ2003
4. Logistics
Starting and ending times
Breaks
Exercise Teams
Timekeeper
Pagers and Location of support facilities
Phones -- Off Number for
messages:
Emergency
procedures:
Attendance at all sessions is expected
Risk Driven Software Testing
4 PSQT East Tutorial v.1.0 ©TQ2003
5. Exercise: Identifying Risky Projects
Using the exercise booklet, and working in teams of
three, complete the exercise named
“Identifying Risky Projects”
Expected time: 20 minutes
Risk Driven Software Testing
5 PSQT East Tutorial v.1.0 ©TQ2003
6. Debriefing: Identifying Risky Projects
What steps happened in your mind for you to identify
risk?
What types of projects you find risky?
How does this affect the way you test?
Risk Driven Software Testing
6 PSQT East Tutorial v.1.0 ©TQ2003
7. Course Objectives
Identify testing project risks and refine the plan to include
mitigation and contingency activities
• State testing goals and objectives
• Identify risks with regard to product and project characteristics
• State testing activities with acceptance criteria
• Select a testing lifecycle to match the products risks and the project's
schedule
Risk Driven Software Testing
7 PSQT East Tutorial v.1.0 ©TQ2003
8. Common Testing Problems
Problems we are used to see happen:
• we ship before we have finished testing
• our customers find all our defects
• we don’t have the testers when we need them
• testing is ad-hoc and results are irreproducible
• we don’t test for the critical success factors
• we don’t test versus requirements from customer
• there are no acceptance criteria for any phase and deciding
when the product is ready becomes a point of contention
• ...
Good risk management processes will help!
Risk Driven Software Testing
8 PSQT East Tutorial v.1.0 ©TQ2003
9. Risk Management Prevents Problems
Problems that risk management can prevent
• We anticipated lateness but accepted sending out an
unfinished product
• We have too many defects to fix too late in the cycle
• We have too many tests to run in a short period of time
• We have tested for the simple defects and the customer gets to
test for the “killer” bugs
• …
We know something can happen but we hope it does not
happen.
Risk Driven Software Testing
9 PSQT East Tutorial v.1.0 ©TQ2003
10. Start by Reviewing Project Parameters
Project vision
• Are objectives clear and understood?
• Are objectives of customer and provider known?
Project scope, milestones and deliverables
• Are features agreed to?
• Are commitments agreed to and documented?
Roles and responsibilities
• Are roles and responsibilities clear to all involved?
• Are all roles staffed?
• Is sponsorship and accountability clear?
Use knowledge of specific project roles to identify risks
Risk Driven Software Testing
10 PSQT East Tutorial v.1.0 ©TQ2003
11. Consider Lessons Learned
For projects like this in the past
• What risks were identified and handled well?
• Are there lessons from those to be applied in this project?
• What risks were identified, but became problems later?
• What should be done differently in this project?
Are there other experiences of the organization that
indicate lessons to be applied here?
Is there information from the industry as a whole that
can be applied?
Risk factor tables include some such lessons
Risk Driven Software Testing
11 PSQT East Tutorial v.1.0 ©TQ2003
12. Sources and Consequences of Risk
Categories of Sources Project Consequences
Mission and goals Cost overruns
Decision drivers testing concerns
Schedule slips
Organization management Inadequate functionality
Customer / end user Canceled projects
Budget / cost Sudden personnel changes
Schedule Customer dissatisfaction
Project characteristics Loss of company image
Development process Demoralized staff
Development environment Poor product performance
Personnel and relationships Legal proceedings
Operational environment
New technology
Risk Driven Software Testing
12 PSQT East Tutorial v.1.0 ©TQ2003
13. Critical Success Factors
A product’s critical success factors (CSFs) are related
to:
• business needs for its development
• coverage of all its intended user constituencies
• fitness for use in the intended context
• lack of dissatisfiers
• competitive advantage
– price?
– delighters?
Risk Driven Software Testing
13 PSQT East Tutorial v.1.0 ©TQ2003
14. CSF: Business Needs (Why?)
Typical business reasons for new systems or changes:
• cost reduction
– reducing time in the field reduces costs
• increased efficiency of work or resource
– a better interface makes one person capable of doing the
job of two
• developing new markets
– using a “smart card” allows very small business to get into
the credit card market
• improved resource management
– electronic data management (EDM) systems allow insurance
claims to be processed in hundreds of different locations,
depending on work load
What Is The Customer Going To Get Out Of The Use Of The
Product?
Risk Driven Software Testing
14 PSQT East Tutorial v.1.0 ©TQ2003
15. CSF: User Constituencies (Who?)
Answer the questions:
• given the business needs, what goals will the product help
achieve?
• who will have to use the product (different user constituencies)
to achieve those goals?
At least three levels of need:
• need to increase the bottom line
– typical user constituency: upper management
• need to gather supervisory data
– typical user constituency: middle management
• need for an efficient interface
– typical user constituency: end user
• other?
Risk Driven Software Testing
15 PSQT East Tutorial v.1.0 ©TQ2003
16. CSF: Fitness for Use (How?)
“Fitness for use”
• the product may meet all stated requirements but not support
solving a real need or problem for a given user constituency
• testing features does not cover the workflows
Test in the context of the job
Test the product as if you would have to perform the job
yourself
Use your expertise in testing to extend the possibilities
of use cases
Risk Driven Software Testing
16 PSQT East Tutorial v.1.0 ©TQ2003
17. Exercise: Critical Success Factors
Using the exercise booklet, and working in teams of
three, do the exercise named Product Critical Success
Factors in the exercise booklet.
Expected time: 15 minutes
Risk Driven Software Testing
17 PSQT East Tutorial v.1.0 ©TQ2003
18. Debriefing: Critical Success Factors
Can testing activities or techniques bring quality to the
product?
Has focusing on the Critical Success Factors helped you
identify potential problems?
Risk Driven Software Testing
18 PSQT East Tutorial v.1.0 ©TQ2003
19. Characteristics of Good Risk Statements
State specific conditions, easy to detect
State specific consequences, which can be measured
Are good enough to proceed to planning
• good risk statements nearly write the action plan
Address risks, not project tasks in plan that are not yet
done
Address potential problems, not current problems
• Risks are possible future events
• Handle current problems another way
Write several risk statements as a class
Risk Driven Software Testing
19 PSQT East Tutorial v.1.0 ©TQ2003
20. Exercise: Product Risk
Using the exercise booklet, and working in teams of
three, do the exercise named Product Risk in the exercise
booklet.
Expected time: 20 minutes
Risk Driven Software Testing
20 PSQT East Tutorial v.1.0 ©TQ2003
21. Debriefing: Product Risks
Are different testing activities or techniques applicable
to different risks?
How does testing help reduce project risks?
Risk Driven Software Testing
21 PSQT East Tutorial v.1.0 ©TQ2003
22. High-Level Testing Goals
Effectiveness:
• Defect detection
– percentage of total found in some time frame
– severity found after testing
• Other?
Efficiency:
• Resource usage
– time
– people
– machines
• Reporting
– time spent reproducing the defect by the developers
• Other?
Risk Driven Software Testing
22 PSQT East Tutorial v.1.0 ©TQ2003
23. Setting Goals
Budgeting Resources:
• You have $200
• You want to:
– Go to the ball game
– Treat your significant other to his/her favorite meal
– Buy a new set of tires
– Pay off your car insurance
– Join a health club
– Raise your daughter’s allowance
– And so many more things!
How do you prioritize these?
Risk Driven Software Testing
23 PSQT East Tutorial v.1.0 ©TQ2003
24. Exercise: High LevelTesting Goals
Using the exercise booklet, and working in teams of
three, do Activity 3 of the exercise named High Level
Testing Goals in the exercise booklet.
Expected time: 20 minutes
Risk Driven Software Testing
24 PSQT East Tutorial v.1.0 ©TQ2003
25. Debriefing: High Level Testing Goals
What is the benefit of stating testing goals?
What is the impact of testing goals on the project’s
plan?
What is the impact of the project’s plan on the selection
of testing goals?
Risk Driven Software Testing
25 PSQT East Tutorial v.1.0 ©TQ2003
26. Acceptance Criteria
For each test selected, define:
• Environment tests would have had to run on
• Regression suites covered by the tests
• Functionality covered by the tests
• Performance testing goals reached
• Volume Testing goals achieved
• Reliability Testing (when applicable)
• Usability Testing (when applicable)
How can we tell that the work product is safe for
releasing it to the user?
Risk Driven Software Testing
26 PSQT East Tutorial v.1.0 ©TQ2003
27. System Acceptance Test
System Testing Phase Report
• Reports on
– tests performed
• performance
• volume
• stress
• regression
• functional
• others? ...
– tests results
• remaining deficiencies
– test coverage
• percentage of a given unit
Risk Driven Software Testing
27 PSQT East Tutorial v.1.0 ©TQ2003
28. System Acceptance Criteria
Probably only main deployment environment
• but NOT the development environment only
Usually all regression suites for the entire tests
• usually automated
Most, if not all, functionality
• focus on functionality not tested or changed after unit code
Performance Testing, Volume Testing
• goals MUST be met, no excuses
• deployment environment used in tests
Reliability Testing
• if applicable, statistically controlled
Usability Testing
• if critical, or if it leaves development organization
Risk Driven Software Testing
28 PSQT East Tutorial v.1.0 ©TQ2003
29. User Acceptance Test
Purpose is to detect remaining defects
• focus might change
Different things to different people
• another level of system acceptance
• emphatically focused on usability
• performed by users exclusively
• performed by user proxies, exclusively
• performed by the IV&V of the organization
• other? ...
Which is yours?
Risk Driven Software Testing
29 PSQT East Tutorial v.1.0 ©TQ2003
30. User Acceptance Criteria
Some general rules
• Cover all deployment environments
• Only do regression if regression suites are different
– or significant fixes and / or changes have happened after
system acceptance
• Functionality focus should be on usual rather than exceptional
• Performance Testing should focus on throughput
• Volume Testing might be skipped
– unless significant fixes and / or changes have happened
after system acceptance
• Reliability Testing
– only when thoroughly planned from day one
• Usability Testing
– probably the most emphatic effort goes here
Risk Driven Software Testing
30 PSQT East Tutorial v.1.0 ©TQ2003
31. Exercise: User Acceptance Criteria
Using the exercise booklet, and working in teams of
three, do the exercise named User Acceptance Criteria in
the exercise booklet.
Expected time: 30 minutes
Risk Driven Software Testing
31 PSQT East Tutorial v.1.0 ©TQ2003
32. Debriefing: User Acceptance Criteria
How do the business goals influence the testing
acceptance criteria?
Which are some of the unavoidable testing acceptance
criteria?
Are there any good reasons to change some testing
acceptance criteria after the test has been executed?
Risk Driven Software Testing
32 PSQT East Tutorial v.1.0 ©TQ2003
33. Risks from the Project
Risks from the project come from:
Project Plan Schedule Constraints
• testing might be cut short if development is late
• testing might have all its tasks in the critical path
Model being followed by the project
• Simple Waterfall,
• Parallel Waterfall,
• Evolutionary,
• Prototyping,
• Spiral
Design Architecture
Resources
• missing critical skills
• budgetary shortcomings
Risk Driven Software Testing
33 PSQT East Tutorial v.1.0 ©TQ2003
34. Design Architecture
Figure out what the Architecture is going to be
• Ask the designer
• Request information on the design elements early
If not traditional, plan accordingly
• Use scenarios profusely
• Try having testers join teams early
• Use testers insight of design to develop test cases
• Push for updated documentation
• Push for consistency reviews across documents
– Requirements traceability
– Formal reviews
Risk Driven Software Testing
34 PSQT East Tutorial v.1.0 ©TQ2003
35. Testing Deliverables
Planning Assets
• Test Plan
Testing Assets
• Testing procedures
• Testing suites
• Testing templates
Reporting Assets
• Individual tests results
• Test statistics
• Acceptance report
Risk Driven Software Testing
35 PSQT East Tutorial v.1.0 ©TQ2003
36. Exercise: Project’s Risks
Using the exercise booklet, and working in teams of
three, do the exercise named Project’s Risks in the
exercise booklet.
Expected time: 30 minutes
Risk Driven Software Testing
36 PSQT East Tutorial v.1.0 ©TQ2003
37. Debriefing: Project Risk
How does the project schedule influence the testing
project’s schedule?
What other areas of a project should be explored for
risk?
Risk Driven Software Testing
37 PSQT East Tutorial v.1.0 ©TQ2003
38. Strategy Problems
Problems that faulty test strategies can cause
• we spend too much time on just one phase
• we place all our effort at the end of the project
• we ignore regressions
• we test in the wrong configuration
• we receive work products that are unfit to test
• we get into endless arguments about product fitness to ship
• ...
Good testing strategies help!
Risk Driven Software Testing
38 PSQT East Tutorial v.1.0 ©TQ2003
39. Selecting a Strategy
Decide on
• how much testing is required
• of what type
• when will it happen
• who will do it
• how will it happen, e.g.:
– Will integration be top-down or bottom-up?
– Will clear box be manual or automatic?
Risk Driven Software Testing
39 PSQT East Tutorial v.1.0 ©TQ2003
40. Defining a Strategy
Review and rework the business goals
Review project variables and rework the Testing Phases
• Consider cost, time to market, personnel, product life
expectancy, users, constraints
Emphasize the tasks that tie with the goals
• try to probe weak areas of the project
Points to ponder:
• regression (when, how much, which)
• coverage (how much, what type, when)
• deadlines (slipping deadlines, schedule compression)
• integration (how, when, by whom)
• assignment of responsibility (developers, testers, users)
Risk Driven Software Testing
40 PSQT East Tutorial v.1.0 ©TQ2003
41. Test Strategies (1)
Analyze business case
• where is the payoff?
– think in terms of customer satisfaction
– identify particular functionality or killer faults
Probe the quality of the product with regard to it
Risk Driven Software Testing
41 PSQT East Tutorial v.1.0 ©TQ2003
42. Test Strategies (2)
Analyze users
• by frequency of use
– sporadic, heavy, etc.
• by organizational level
– general managers, middle managers, end users, etc.
• by their knowledge of the software
– experts, newcomers, etc.
Build a profile of system usage
• sketch scenarios
• assign probabilities for each scenario
– e.g.., one out of eight times the plan will be run unchanged
Use this data to design the test cases to optimize testing
coverage of most frequent paths
Risk Driven Software Testing
42 PSQT East Tutorial v.1.0 ©TQ2003
43. Test Strategies (3)
Analyze time to market
• are you going to be pressed for time?
– => focus on existing functionality
– => test system rather than component
– => test typical rather than exceptional
Decide which tests can be run within the time
constraints
• If some of the fundamental tests cannot be run, move tests
forward
Risk Driven Software Testing
43 PSQT East Tutorial v.1.0 ©TQ2003
44. Test Strategies (4)
Analyze cost
• how many staff/hours can you pay?
Figure out if the tests you have so far selected fit within
the budget
If so, if you have room for more…
Analyze constraints
– performance
– precision
– volume
• find critical quantities
• analyze or set stress testing
…then include tests for them
Risk Driven Software Testing
44 PSQT East Tutorial v.1.0 ©TQ2003
45. Test Strategies (5)
Analyze personnel
• who can you count on
– reinforce testing of the products of the project’s weaker
links
• which of your documents are going to be weak for lack of
experts
– requirements or specs <=> functional tests
– high-level design <=> integration
– modules <=> module testing
Analyze product life expectancy
• find the payoff of documented procedures, test case suites,
results, etc..
– don’t overspend in a product that has a short life
expectancy
– don’t under spend in a product that will be around long
Risk Driven Software Testing
45 PSQT East Tutorial v.1.0 ©TQ2003
46. Example Strategy (1)
Business goal
• Keep and expand customer base
Internal translation to project
• Make sure that the user finds the entire current version’s
functionality works as usual
Testing strategy
• Test old before new, in every phase
Note:
Underlying assumption is
that you will not have time to do it all
Risk Driven Software Testing
46 PSQT East Tutorial v.1.0 ©TQ2003
47. Example Strategy (2)
Business goal
Exceed customer’s expectations of product quality
Internal translation to project
• Fewer than 10% of total bugs caught by the user
Testing strategy
• Move functional test suites to developers before and during
unit code and testing
Note:
Underlying assumption is
that you will not have time to do it all
Risk Driven Software Testing
47 PSQT East Tutorial v.1.0 ©TQ2003
48. Limiting the Scope of the Testing Effort
Some things cannot be tested
• quality
• user-friendliness
• timeliness
•…
Some things you might not want to test
• regression on a new or relatively small enhancement
• performance
• stress
•…
Some things you might not have the ability to test
• reliability (e.g. MTTF)
• availability (e.g. (MTTF/(MTTF+MTTR)))
•…
Risk Driven Software Testing
48 PSQT East Tutorial v.1.0 ©TQ2003
49. Constraints on the Lifecycle
Review the phases against the Project’s constraints
• Can you accommodate the project plan schedule constraints?
• Is the model being followed by the project
– Simple Waterfall,
– Parallel Waterfall,
– Evolutionary,
– Prototyping,
– Spiral
allowing for the testing tasks you’ve set?
– e.g. might not have integration testing
• Can the tests selected be adjusted to the design architecture?
– three tiers might bring a whole set of problems
• Are the goals compatible with the project’s shortcomings?
Risk Driven Software Testing
49 PSQT East Tutorial v.1.0 ©TQ2003
50. Exercise: Testing Lifecycle
Using the exercise booklet, and working in teams of
three, do Activity 1 of the exercise named Testing
Lifecycle in the exercise booklet.
Expected time: 30 minutes
Risk Driven Software Testing
50 PSQT East Tutorial v.1.0 ©TQ2003
51. Debriefing: Testing Lifecycle
How do the business goals influence the selection of
testing lifecycle?
Which are the unavoidable testing deliverables?
Are there any good reasons to avoid some testing
phases?
Risk Driven Software Testing
51 PSQT East Tutorial v.1.0 ©TQ2003
52. Identifying Test Tasks
Review the V-Model
Pick and choose the adequate phases to meet the goals
You have twenty days to visit all of Europe
You may never be able to go back
How do you budget your time?
Testing is always under-budgeted and over-committed
Risk Driven Software Testing
52 PSQT East Tutorial v.1.0 ©TQ2003
53. Risk Action Planning
Deal with high exposure risks first
• Research: Do we know enough about this risk?
• Accept: Can we live with it and do nothing about it?
• Manage: Can we take action?
• Avoid: Should we cancel the project or change the approach?
Balance the threat of the risk against the effort to avert it
• How great is the threat?
• How much does it cost to avert?
Risk Driven Software Testing
53 PSQT East Tutorial v.1.0 ©TQ2003
54. Risk Contingency Plans
Devise contingency plans for
• High exposure risks, in case the mitigation strategy fails
• Any risk for which there is no possible mitigation action
Specify risk measures and trigger values
• Measures of time, resources to handle risk
• Measures of risk impact
• Trigger values that tell it is time to use contingency approach
now
Agree with customer and management at project start
how contingency plans will be funded and handled
Risk Driven Software Testing
54 PSQT East Tutorial v.1.0 ©TQ2003
55. Example Contingency Triggers
For risks leading to schedule slips
• Latest date to allow you to use alternative platforms
• Latest date to select another vendor
For risks requiring additional effort or time
• Latest date to have time to locate the resources
• Greatest amount of penalty or fine to incur
• Greatest amount of investment available for overrun
Limit for extra cost to the customer
Limit for learning time
Risk Driven Software Testing
55 PSQT East Tutorial v.1.0 ©TQ2003
56. Example of Action and Contingency
Risk Statement
• Since the project team is already behind schedule by two full
weeks, we might not have time to cover all the high yield tests
before the cutover date and the quality of the product will be
seriously imperiled.
Risk Action
• Provide one of our test engineers as a testing consultant to the
development team, so they can test and fix before they send
the product to the Testing Team.
Contingency Plan
• If by the first of April, we do not have an integration test
version of the product, drop the web testing suites and focus
on regression so our customer can use a significantly
improved current version for six months without a Web
interface.
Risk Driven Software Testing
56 PSQT East Tutorial v.1.0 ©TQ2003
57. Example Contingency for Lateness
Plan your testing activities in passes
• First Pass (mandatory):
– test all modules/components
– use most frequently used scenarios
– use few test cases
– use selected error cases
• Second Pass (supplementary):
– test only modules with fixes, do first pass on components
– cover most scenarios
– use test cases covering all data equivalence classes
– include test cases for “bad values”
• Third Pass (complementary):
– test only modules with fixes from second pass
– cover all test suite
– go for 100% clear case coverage
Risk Driven Software Testing
57 PSQT East Tutorial v.1.0 ©TQ2003
58. Exercise: Managing Risk
Using the exercise booklet, and working in teams of
three, do the exercise named Managing Risk in the
exercise booklet.
Expected time: 30 minutes
Risk Driven Software Testing
58 PSQT East Tutorial v.1.0 ©TQ2003
59. Debriefing: Managing Risk
this will be an exercise that for every risk identified in
the exercises make them think an action plan and/or a
contingency plan.
Risk Driven Software Testing
59 PSQT East Tutorial v.1.0 ©TQ2003
60. Summary
Key Activities in Defining the Testing Strategy
• identify test tasks and their goals
• rank test tasks by their goals
• define the limits of the testing effort
• detail testing tasks
• define testing tasks entry criteria
• define testing tasks acceptance criteria
Outputs to the process include
• individual test task goals
• phase acceptance criteria
• detailed testing project tasks
• all in the updated test plan
Risk Driven Software Testing
60 PSQT East Tutorial v.1.0 ©TQ2003
61. Next Steps
Review Priorities
and Expectations
Develop Action List
Set Completion
Dates
Review Results
Risk Driven Software Testing
61 PSQT East Tutorial v.1.0 ©TQ2003
62. Course Objectives
Identify testing project risks and refine the plan to include
mitigation and contingency activities
• State testing goals and objectives
• Identify risks with regard to product and project characteristics
• State testing activities with acceptance criteria
• Select a testing lifecycle to match the products risks and the project's
schedule
Risk Driven Software Testing
62 PSQT East Tutorial v.1.0 ©TQ2003
63. And now… let’s hear from you!
Thank you very much for your participation!
Please complete a course evaluation form
• very useful in improving the course
• most helpful if very honest
Please send us your comments as you work with this
material
• email your instructor - contact info in student guide
• send us comments on what works and what doesn’t so we can
improve the course
Risk Driven Software Testing
63 PSQT East Tutorial v.1.0 ©TQ2003
64. Related Courses from TeraQuest
Project Launch Workshop
Project Planning and Tracking
Testing Management Workshop
Quality Assurance Workshop
Configuration Management Workshop
Peer Reviews and Inspections
Risk Driven Software Testing
64 PSQT East Tutorial v.1.0 ©TQ2003
65. Information about TeraQuest
TeraQuest Metrics, Inc. Process assessment
P.O. Box 200490 Process improvement
Austin, Texas 78720-0490 SPI and KPA training
USA Risk management
Software measurement
1-512-219-9152 (phone)
1-512-219-0587 (fax)
TeraQuest Web site: http://www.teraquest.com
Email questions to: info@teraquest.com
Risk Driven Software Testing
65 PSQT East Tutorial v.1.0 ©TQ2003