2. 2
Keynote Outline
Let’s break the ice!
Why do you need good requirements?
How can you better structure and what for?
Some tips and tricks
A TRH! Oh great!...erh remind me…what for exactly?
Pilot, prioritize, communicate, follow-up
Yes, but I’m an Analyst, not a TM or a TA…
That’s all folks!
5. 5
Keynote Outline
Let’s break the ice!
Why do you need good requirements?
How can you better structure and what for?
Some tips and tricks
A TRH! Oh great!...erh remind me…what for exactly?
Pilot, prioritize, communicate, follow-up
Yes, but I’m an Analyst, not a TM or a TA…
That’s all folks!
6. 6
Functional Defect
Project : Louvois
Late payments, wrong salary calculation…
The French Ministry was unable to fix the software issue…
Families are appalled
346 Mio € Loss – no usage
French Servicemen hit by a payroll software issue
9. 9
User / Customer
Issue
Technical Issue
Functional Issue
Ow! Let’s go fishing!
Production
Maintenance
Poor Non-
Functional Testing
Poor Code
quality
Poor Technical
architecture
Bug - Defect
Failure - Issue
Insufficient
involvement
Poor business
processes
Communication
Needs
Market
Poor Requirements
Poor Functional
Architecture
Poor functional
Testing
Insufficient
involvement
Poor Business
Processes
Communication
Needs
Market Poor Non-
Functional Testing
Poor Code
quality
Poor Functional
Testing
Production
Maintenance
Poor Technical
Architecture
Poor Requirements
Poor Functional
Architecture
CMMI® for DEV
10. 10
Keynote Outline
Let’s break the ice!
Why do you need good requirements?
How can you better structure and what for?
Some tips and tricks
A TRH! Oh great!...erh remind me…what for exactly?
Pilot, prioritize, communicate, follow-up
Yes, but I’m an Analyst, not a TM or a TA…
That’s all folks!
12. 12
How Do We (testers) speak?
• Constraints:
– Time
– Costs
– Resources
• Requirements:
– Coverage
– Test depth
Coverage
Depth
Costs, Resources
Time
15. 15
Keynote Outline
Let’s break the ice!
Why do you need good requirements?
How can you better structure and what for?
Some tips and tricks
A TRH! Oh great!...erh remind me…what for exactly?
Pilot, prioritize, communicate, follow-up
Yes, but I’m an Analyst, not a TM or a TA…
That’s all folks!
16. 16
Use Patterns…
• …and make TESTABLE requirements:
– FUNCTIONAL
• As ROLE I need ACTION in order to OBJECTIVE + context qualifyers
• As user I need to access my personal file in order to complete it with the necessary info
– SYSTEM
• When TRIGGER the system NAME must ACTION/PERMISSION to
WHOM/WHAT in order to OBJECTIVE + qualifyers
• When book is available, BorrowBook must allow subscribers to borrow if subscription paid and
limit not reached
18. 18
Often (very often)…
• Absence of references
• Few measures and metrics (or none)
– Unknown coverage
– Unclear efficiency
– No capitalization
• Decisions = estimations
• Management cannot evaluate
properly
19. 19
What Would we Want to Know?
• What’s left to do?
• Am I on track?
• What can I improve?
• What is MY quality?
• Do defects get solved?
• What did I learn?
Not measuring = Not
knowing
20. 20
• No clear specification
• What is really needed?
– Basic or advanced functionality?
– Usability?
– Performance?
– Security?
• When do we have enough tests?
– How critical is the functionality?
– Deep or shallow test?
• How long to design, execute the tests?
• What should we test first?
• How can we draw a conclusion from the tests?
Difficulties Encountered
21. 21
Keynote Outline
Let’s break the ice!
Why do you need good requirements?
How can you better structure and what for?
Some tips and tricks
A TRH! Oh great!...erh remind me…what for exactly?
Pilot, prioritize, communicate, follow-up
Yes, but I’m an Analyst, not a TM or a TA…
That’s all folks!
22. 22
A TRH???
« Logic : a good tool that is sold to us with no user manual » - Pierre Véron
23. 23
Hierarchical tree view of application requirements
Business process thinking
How to reach business goals (notion of benefit)
Process-oriented
Tracing down from process to functions
Quality reference
Structuring with a TRH
test requirement 1
test requirement 1.1
test requirement 1.2
test requirement 1.2.1
test requirement 1.2.2
test requirement 1.2.3
test requirement 1.3
test requirement 1.4
test requirement 1.3.1
test requirement 1.3.2
24. 24
The Fundamental Test Process
Test planning and control
Test analysis and design
Test implementation
and execution
Evaluating exit criteria
and reporting
Test closure activities
TRH used throughout
these stages
26. 26
The Problem with Requirements
• Requirements form the basis for testing
• Requirements are not always testable
– Often imprecise, ambiguous
– Often incomplete
– Can be contradicting or totally wrong
– Sometimes, requirements simply don’t exist
• Challenges
– make imprecise, incomplete requirements testable
– Identify wrong, missing requirements
27. 27
How would you test a train ticket application?
A Small Example …
29. 29
Constructing the Hierarchy
HOW?
Let’s bring
the Money In
Sell tickets
Exchange
tickets
Refund
tickets
International
National Full price
Off peak
Reduced - Young
Reduced - Sr
Reduced Loyalty
HOW?
30. 30
Asking How, Creatively
• Use alternative formulations:
– What needs to happen?
– Are there any related problems?
– What are you likely to have done previously/next?
– What needs to happen when this fails?
– How does the user know this?
– How could things be screwed up?
• Use basic test specification techniques to stimulate responses.
– “Manage things” → CRUD
(create/read/update/delete)
– Equivalence partitioning
– Boundaries
– Negative cases
• Make sure sub-requirements cover parent 100%
31. 31
Establish Business Context
Let’s bring
the Money In
Sell tickets
Exchange
tickets
Refund
tickets
International
National Full price
Off peak
Reduced - Young
Reduced - Sr
Reduced Loyalty
WHY?
WHY?
WHY?
32. 32
Asking Why, Unaggressively
• The question “why?” by itself can be quite intimidating (or even
look foolish)
• Use alternative formulations:
– What is the purpose of that?
– Can you explain why you need it to do that?
– What’s the relation to your business?
– What business problem is this solving?
– What was the thinking behind that?
– What is the underlying reason for that?
– What would happen if that feature was not available?
– …
33. 33
How to Get There?
• Previous Knowledge & Experience
• Reading and retro-analysis
• Face-to-face meetings
• Workshops
NEVER ASSUME !!!
- Loyalty is not about calculating
& redeeming points
It is about COMMUNICATION
34. 34
Keynote Outline
Let’s break the ice!
Why do you need good requirements?
How can you better structure and what for?
Some tips and tricks
A TRH! Oh great!...erh remind me what for exactly?
Pilot, prioritize, communicate, follow-up
Yes, but I’m an Analyst, not a TM or a TA…
That’s all folks!
35. 35
A TRH What for?
• Prioritise
• Communicate
• Pilot
• Estimate
• Test strategy backbone
test requirement 1
test requirement 1.1
test requirement 1.2
test requirement 1.2.1
test requirement 1.2.2
test requirement 1.2.3
test requirement 1.3
test requirement 1.4
test requirement 1.3.1
test requirement 1.3.2
1
1
1
2
2
3
3
36. 36
Test Implementation and Execution
TRH
Test
design
spec
Test
design
spec
Test
procedure
spec
Test
case
spec
Test
procedure
spec
Test
Plan
bugs
Baseline
planning
Progress
report
37. 37
TRH and Test Design
TEST SPECIFICATION ID: XYZ
PURPOSE: Verify that search function
examines articles.
APPROACH REFINEMENTS: …
38. 38
TRH and Process
Test procedure
Scenario
Data
Validation
points
Physical Design
WHAT ?
Test procedure
Scenario
Data
Validation
points
Test case1
Test case2
Test case3
Test case5
Test case4
Test Requirements Hierarchy
WHY / WHAT?
test requirement 1
test requirement 1.1
test requirement 1.2
test requirement 1.2.1
test requirement 1.2.2
test requirement 1.2.3
test requirement 1.3
test requirement 1.4
test requirement 1.3.1
test requirement 1.3.2
Logical Design
WHAT?
39. 39
Who’s the Pilot?…Damn’ it’s the TRH!
• Nb testable requirements (leaves)
– Identified / Developed / written
– % Coverage(s)
– % Executed
Progress (t vs. t-1 vs. planned)
– Risk / Impact (before and after execution)
– Planning / Effort
40. 40
In Practice
• Planning - Execution
– 100 requirements to test
– 100 persons*days budget
– Writing / Execution = 60/40
• Quality
– 50% requirements coverage
– 50% passed product
– Grey zones (50 shades)?
41. 41
Software Tools…Who Cares?
• There are (too) many of them
– From Excel to Doors via TestLink & others
– Integrated in suites (HP, IBM) or not
– OpenSource or Commercial
• Most (at least the known ones) are good
• Tools are NOT the issue
42. 42
…Provided they Allow…
• Real-time share and follow
• Information and reporting
(collect and process)
• Decisions
• Anticipation
• Adapt the tool to YOUR organisation
46. 46
…but How to Use them Matters
• They must be robust…
– The « in-house » requirements tracker
• …fit to your needs and organisation
• …and should allow to share and pilot
– Customer / supplier « encode twice »
47. 47
Requirements
• Total Nb & Testable >> coverage
• Repartition / severity
• Repartition / phase / version
• Repartition / priority
• Nb incorrect or maintained
requirements
• Nb scenarios / Requirement
• Nb unused requirements
• Nb added requirements
• Nb changed requirements
48. 48
Test Scenarii – Test Designs
• Total & Testable (Nb)
• Répartition / severity
• Repartition / phase or version
• Repartition / priority
• Nb scenarii reviewed and/or incorrect
• Nb scenarii / Requirement
• Nb scenarii unused
• Nb scenarii added
Measurements details
Fine but complex vs.coarse but blurry estimations
Differs from phase to phase depending on goals
49. 49
Defects
• Total Nb
• Repartition / severity
• Repartition / priority
• Repartition / status
• Repartition / responsible
• Nb found / unit time
• Nb false positive
• Nb unsolved
• Nb found next phase
50. 50
Adjust your Strategy
You are HERE
Time
Scope
EXPECTED Deadline
EXTENDED Deadline
100%
0%
50%
1
3
2
66%
Productivity
51. 51
Different Mitigation Options
• Option 2
– Limit Scope (shrink till you drop)
• Quality drop
• Increased risk
– Let it slip (the whoosh sound of dealines)
• Late market presence
• Extra costs
• Option 3
– Increase resources (Stakhanov – Taylor)
• Extra costs
• 2 women + 4,5 mths = NO baby
52. 52
One Example
• Mitigating options 2 (slippage) and 3 (increase ressources)
– Use what you already know to predict and plan
• Risk oriented strategy
– Sanity check
– Flexible resources
– Planning by roles and capability
• Use existing resources
• Identify and manage activities
– Test new functionality
– Re-test (bug fix)
– Regression test (re-validation)
– …/…
• Multiply PC (automation)
– Execute based on risk / priority
• Most important first
• Most important = deeper and longer
BA
54. 54
Test Campaigns
1 2 3 4 5 6 7
Go / No Go
SP 2.1 SP 2.2 SP 2.3 ?
Production?
Production?
h1 h2 h3
Inherited from
SP1.x
h1 h2
Σhi
Σhi
d2 d3
Manual Tests (n) Manual tests (n+1) + automated (n)
• ! Performance
• ! Availability
• ! Stability
27/03 02/05 21/05
Inherited
from SP2.0
d1 0 ?
?
?
Correction > c1 Correction > c2
SC
21/03 18/04
8 ?
2/04
di = Nb estimated defcts / phase
ci = Nb fixed defects in the same phase
hi = Nb inherited defects
Σ’hi
Σ’hi
55. 55
And if you Have…TIME
• Today (status time)
– Consumed vs, attended resources
– Forecast based upon experience
• Mean time (write / fix / execute)
• Mean time (code / analyse / fix)
• Application up time
• Application performance
• MTBE, MTBF
Good stop criterion…
if enough maturity
58. 58
Do NOT Overperform
Date 27/08
Sprint 2
Release 3
# days 15/23
15
# Testers Effective Cum. % planned resources for SP2.3
Execution test designs 5,5
Check printed documents 3
Ad hoc tests 5 63%
Quotation (Product Management) 0 100%
Re-test SP1 + SP2 ad hoc defects 0 100%
Traduction 0 100%
Total resources coverage Release 3 13,5 62%
Availability - Lost time
Availability of the application that day 100,0% week 1/5 week 2/5 week 3/5 week 4/5 week 5/5
Average availability for SP2.3 85,7% 80,1% 78,6% 95,6% 85,7%
Test Weather
Report
58%
Test execution (2849 data sets)
# data sets foreseen that date 170 Found defects
# data sets tested that date 90
% data set coverage SP2.3 - against planned ds 53% OK NOK Not testable Intermediate
# data sets executed SP2.3 2148 1850 233 0 65
86,1% 10,8% 0,0% 3,0%
% data set coverage SP2.3 - against total ds 75,4% Docs waiting for check
# SP2 defects foreseen to be re-tested during SP2.3 445
# SP2 defects re-tested that date 22 OK NOK Not testable
# SP2 defects re-tested while executing ds SP2.3 211 184 26 1
87,2% 12,3% 0,5%
% re-test defects coverage 47%
Average datasets/p/d foreseen for SP2.3 19,9 *Average test design (15 ds/p/d) and check document (29 ds/p/d)
Average datasets/p/d effective that day 16,4
Average datasets/p/d effective in SP2.3 17,3
Re-test SP2 AD HOC defects
# SP2 ad hoc defects foreseen to be re-tested during SP2.3 106
# SP2 ad hoc defects foreseen that date 0 NOK + OK +
# SP2 ad hoc defects re-tested that date 0 OK NOK New defect New defect
# SP2 ad hoc defects re-tested in SP2.3 99 81 16 0 0
81,8% 16,2% 0,0% 0,0%
% SP2 ad hoc defects re-test coverage 93%
Average defects re-tested/p/d foreseen for SP2.3 10,0
Average defects re-tested/p/d effective that day -
Average datasets/p/d effective in SP2.3 -
% data sets executed SP2.3
Cumul # SP2 ad hoc defects re-tested in SP2.3
Cumul # SP2 defects re-tested while executing ds SP2.3
# Defects
# encoded defects that day 18 Urgent High Medium Low
# active defects in Defect Tracker for SP2.3 102 20 51 24 7 102
# active defects in Defect Tracker 411 29 257 110 15 411
Open IT exam Pend. TEST Sub-total Active Whole
IT Factory 65 0 21 86
Ready TLF Pend. TLF User Exam
Test Factory 2 297 26 325
Closed Deferred Archived
Not active defects 454 47 1507 2008
Test exam
0 0
Remarks:
Today :
Finalised retest of fiches linked to "marques-modèles" has been done
Application has been better regarding response time but a lot of instabilities (T000): see daily urgent defects sheet for details.
Program for tomorrow :
Same as today
2419
411
Fitness
Program
Data set foreseen / executed
0
500
1000
1500
2000
2500
3000
5/aug
7/aug
9/aug
13/aug
19/aug
21/aug
23/aug
27/aug
29/aug
2/sep
4/sep
6/sep
Executed Cumulated DS
DS foreseen this day (intial
planning)
120
140
0
500
1000
1500
2000
2500
3000
Availability of the application
91%92%
24%
100%
93%
64%
89%
82%
86%
100%100%100%
92%
71%
100%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
5/aug
7/aug
9/aug
11/aug
13/aug
15/aug
17/aug
19/aug
21/aug
23/aug
25/aug
27/aug
29/aug
31/aug
2/sep
4/sep
6/sep
Availability
Retest defects SP1 foreseen / executed
0
20
40
60
80
100
120
140
5/aug
7/aug
9/aug
13/aug
19/aug
21/aug
23/aug
27/aug
29/aug
2/sep
4/sep
6/sep
Re-tested this daySP1 + SP2 ad
hoc
Foreseen to be retested this day
(intial planning)
59. 59
Keynote Outline
Let’s break the ice!
Why do you need good requirements?
How can you better structure and what for?
Some tips and tricks
A TRH! Oh great!...erh remind me what for exactly?
Pilot, prioritize, communicate, follow-up
Yes, but I’m an Analyst, not a TM or a TA…
That’s all folks!
60. 60
Yes, Sure…However
• You are also accountable for Quality
• You can set-up and follow what everyone understands
– Added value
– Business means
• You bring the missing link to life
62. 62
Keynote Outline
Let’s break the ice!
Why do you need good requirements?
How can you better structure and what for?
Some tips and tricks
A TRH! Oh great!...erh remind me what for exactly?
Pilot, prioritize, communicate, follow-up
Yes, but I’m an Analyst, not a TM or a TA…
That’s all folks!