Test Managers: How You Can Really Make a Difference

TP
PM Tutorial
10/1/2013 1:00:00 PM

"Test Managers: How You Can
Really Make a Difference"
Presented by:
Julie Gardiner
The Test People

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Julie Gardiner
The Test People
In the IT industry for more than twenty years, Julie Gardiner has held positions as an analyst
programmer, Oracle DBA, and project manager. She has first-hand experience as a test
analyst, test team leader, test consultant, and test manager. Previously with UK-based Grove
Consultants for seven years, Julie now is head of agile, talent and transition management with
The Test People. A certified ScrumMaster and agile coach, Julie also specializes in risk-based
testing, agile testing, test management, and people issues.

Passionate about Testing, Excited about Technology
Passionate about Testing, Excited by Technology

Test Managers:
How you can really
make a difference
13th August 2013
Julie Gardiner
Email: julie.gardiner@thetestpeople.com
Twitter: @cheekytester

©
Julie
Gardiner


Contents

What makes us valuable on project
How to keep testing valuable on projects
Showing the value of testing

1


What does a Trusted Advisor look like?


What senior managers are looking for
§  commitment not just involvement
plus… a
–  knowledgeable enthusiasm
sense of
–  high quality teams
humour
§  respect for their decisions
J
–  an understanding of office politics
–  an appreciation of other pressures and issues
§  honesty and courage
–  eyes on the deadline but hands firmly on quality
–  a realist with a hint of optimism

2


What if senior managers seem uninterested?
§  lack of understanding
solu%on:
educa%on

–  what do we do with our time?
–  misconceptions regarding testing/development
§  boredom
solu%on:

enquire

–  information overload – too much detail
–  the wrong information presented (does not understand)

“The bitterness of poor quality is remembered long
after the sweetness of meeting the schedule is
forgotten”


Being a “Trusted Advisor” requires three core skills

Source:
The Trusted Advisor
David Maister,
Charles Green
Robert Galford

building
relationships

3


Trust must be earned & deserved
§  trust grows over time rather than just appears
§  beware that trust can be lost in an instant
§  trust is personal
§  it involves both rational and emotional factors
§  trust is a two-way relationship
§  trust is different for the test manager & senior
manager
§  trust will entail risk
§  if risks are not created, trust will flounder

earning
trust

to be trusted is never a right…
©
Julie
Gardiner
it is always a privilege and honour


Building relationships
§  deal with senior management as “human beings”
§  formula to building a strong relationship
§  BSR = AQ + LTA

building
relationships

§  be open and honest – not a sycophant
§  when you need help – ask
–  this can be a sign of strength rather than weakness

§  show appreciation

make it your goal to be someone managers turn to first
when issues arise and advice is required.

4


Giving advice effectively
§  be prepared
–  look beyond the “now” & understand the whole
situation

giving
advice

§  advice is almost never a purely logical process
–  don’t exaggerate or lie – ever! (this destroys trust)
–  give good news before bad news (don’t hide bad
news)
–  be fair with information
§  know your audience
–  the advice we give must be helpful
–  different people need different information


Developing the right mindset

having a right
attitude and
mindset is a
massive step
closer to
becoming a
“Trusted Advisor”
to Senior
Management

six key attitudes to consider
§  focus on the other person
§  being sincere
§  having confidence (in oneself)
§  adopting a blameless culture
§  taking responsibility
§  recognise strengths and weaknesses
–  in you and your team

5


Contents



Company Objectives?
Quality

Priority
QUALITY

TIME

Cost

Time

COST

©
Julie
Gardiner

6


Managing testing
Policy

overall approach to
testing

Plan

Measure

Strategy

Review

Goal


Test policies

•  represents testing philosophy for organisation
–  outlines the corporate goals for testing

•  guides to stated goals and objectives
•  communicates good practice for the organisation
–  approach, culture and standards

• 
• 
• 
• 

developed by the IT, R&D or Product Development
apply to both new development and maintenance
short, static, high-level document
some areas covered include
–  test process, levels of testing, success factors, measurements, test
process improvement

7


Test policy examples

•  testing is to be riskdriven
•  reviews will include
testers
•  auditable test
documentation records
will be kept

•  entry and exit criteria
will be specified and
enforced
•  the value of testing will
be monitored and
reported

What policy statements do you have in your organisation?


Policy, Strategy and Plans

Company Policy for Testing
Company/Project Strategy for Testing
Project (High Level) Test Plan

8


Monitoring
•  regular and timely supply of information
•  what is actually happening
•  to compare with the plan
•  right use of words, graphs and tables
•  interpretation & explanation is often required
•  enables effective and timely control
•  used to predict what is likely to happen

now to introduce some powerful
monitoring techniques to display on your
dashboard…


One: S-curves
How
to
use
S-‐curves

S-‐curves
for:

tests
run/a@empted

tests
passed/complete

bugs
found

bugs
ﬁxed

Pme

Plot
the
points
you
know,
and
planned
end
point

e.g.
no.
tests
run
so
far
&
planned
ﬁnal
no.
at
end
date

Plot
trend
line:
3rd
order
polynomial

Useful
approximaPon:
straight
line

Source:
Marnie
Hutcheson

9


Example using S-Curves

Source: Tim Trew, Philips


Two: Defect Measure (DM) Rate

DM = 10*H+5*M+L

DM Rate = DM/(hours of test effort)

10


Three: System Availability
Host System Availability
Wed
Tue
Mon
Sun

Days

Sat

Available

Fri

Down time

Thu
Wed
Tue
Mon

0

2

4

6

8

10

Hours available in a day

Based on 10 hour day we lose 2 days of down time!


Four: Regression Test Coverage
Test
Procedure

Solo
Personnel

Solo
Payroll

Personnel
Breadth

Payroll
Breadth

Personnel
Scenario
1

Payroll
Scenario
1

Key:

.0
Release
.1
Release
1

Release
1.0
Release
11.1
Release
1.2.2

not
run

defects
found

no
defects

11


Five: Radar Diagram
Rating
Code Turmoil(loc)
Defects found this week
Total Open Defects
Test Success Rate
Test Completion Rate
TOTAL RATING
Raw Data
Code Turmoil
Defects found this week
Total Open Defects
Test Success Rate
Test Completion Rate

10/23/2013
1127
23
24
95.50%
44.80%

Ranges
1 = 0-50, 2 = 51-100, 3 = 101-150, 4 = 151-200, 5 = 201-250, 6 > 250
1 = 0-2, 2 = 3-4, 3 = 5-6, 4 = 7-8, 5 = 9-10, 6 > 10
1 = 0-10, 2 = 11-20, 3 = 21-30, 4 = 31-40, 5 = 41-50, 6 > 50
1 = 100%-98%, 2 = 97%-95%, 3 = 94%-92%, 4 = 91%-89%, 5 = 88%-86%, 6 < 86%
1 = 100%-95%, 2 = 94%-90%, 3 = 89%-85%, 4 = 84%-80%, 5 = 79%-75%, 6 < 75%
5 to 10
11 to 15
>15

6
6
2
1
5
20

Test Completion
Rate

17-Feb-13
1409
16
32
98.70%
70.00%

24-Feb-13
1020
6
7
99.50%
71.80%

Code
Turmoil(loc)
6
5
4
3
2
1
0

Test Success
Rate

03-Mar-13
799
13
12
99.80%
78.90%

Defects found
this week

Total Open
Defects

Source: Mike Ennis, Managing the End Game of a Software Project, STAREast


Six: Outstanding Risks

start

Residual Risks

today

end date

residual risks
of releasing
TODAY

all risks
‘open’ at
the start

Progress through the test plan

Source: Risk Based E-Business Testing – Paul Gerrard & Neil Thompson

12


Objective

Objective

Objective

Objective

Objective

Benefit

Benefit

Benefit

Benefit

Benefit

Benefit

Benefit (or objective) based test reporting

Open

Risks

Closed

Open

Closed

Closed

Open

Open

Closed

Source:
Paul
Gerrard

Benefits available for release


Seven: Reliability/Confidence Curve
alternative 1

alternative 2

Time (minutes)

Reliability/Confidence Curve
100
80
60
40
20
0
0

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Release

1

1.1

Time to failure

R 1.0

R 1.1

R 1.2

R 1.3

13


Contents



Three key measures to consider

cost saving
per bug

Defect Detection
Percentage
(DDP)

Pre-release
Defect Fixed
Percentage
(DFP)

14

1000


100

10

“Testing is expensive”

1

Red

Des

Test

Use

•  compared to what?
•  what is the cost of NOT testing, or of bugs missed
that should have been found in test?
–  cost to fix bugs escalates the later the bug is found
–  poor quality software costs more to use
o  users take more time to understand what to do
o  users make more mistakes in using it
o  morale suffers
o  => lower productivity

•  what does a bug cost in your organisation?


How much does it cost (based on Boehm)

How much do defects cost?
18000
16000
14000
12000
10000
8000
6000
4000
2000
0

Folksam

ru
nn
ing
li v
e

sy
st
em
/a
cc
ep
ta
nc
e

in
iti a
l te
st

at
ion
sp
ec
i fic

re
qu
i re
m

en
ts

Boehm

Source: Kristina Bihlar, Folksam BR

15


How much do our defects cost
Defect in:
requirements
specification
initial test
system/acceptance test
live running

Manhour to fix

Cost
1
2.5
4
8
15

Cost per hour in SEK:

665
1663
2660
5320
9975

665

What if we found 100 defects during testing:
During system/acceptance test
what could have been found in initial test
what could have been found in analysis
But it could have cost:
Source: Kristina Bihlar, Folksam BR

We saved:
We saved:

532,000
266,000
465,500

997,500 …


How expensive for you?

•  do your own calculation
–  calculate cost to fix bugs missed
by testing
–  calculate cost to fix bugs found in
testing
–  calculate cost of testing
o people’s time, machines, tools
•  estimate if no data available
–  your figures will be the best your
company has!

16


Questions you may be asked
How good is the testing anyway?
Can you prove you are doing a good job?
Your testing can still be just as good in less time, can’t it?
(That deadline pressure really didn’t matter, did it?)
Is the testing any better for this release?
(Have we learned anything?)
(Have we really improved our testing?)
How many bugs have we missed?


Some questions for you
•  do you keep track of defects?

–  defects found in testing
o  different test stages (e.g. system test, user acceptance
test),
o  different releases (e.g. testing for an incremental
release)
–  defects found in live running
o  reported by users / customers
•  can you find these numbers from a previous project and your current
project?
•  do you have a reasonable number of defects found?

if so, you can use DDP to measure your test effectiveness

17


Defect Detection Percentage (DDP)
Defects
found
by
this
tesPng

Total
defects
including
those
found
aYerwards

•  "this" testing could be

–  a test stage, e.g. component, integration, acceptance,
regression, etc.
–  all testing for a function or subsystem
–  all testing for a system


Effectiveness at finding defects

Defects
Found

50
45
40
35
30
25
20
15
10
5
0
Release

100%
90%
80%
70%
60%
DDP
50%
40%
30%
20%
10%
0%

Time

Defects found
Defects found
Total defects
10
50
38
37
35
27
24
19
0
88
87
85
77
74
69
62
50
42 after testing: 12 found:
in testing:

DDP =

50
69
62
87
85
77
74
50
88

57
59
65
68
72
81
= 100 %

18


Options for measuring DDP

•  what defects?
–  simplest: all test defects / all defects so far
–  by severity level
•  how "deep" to go? (how much analysis)
–  deeper analysis gives more detailed information
o but is it really worth the extra effort that would
be needed?
•  start simple
–  simple information is much better than none
–  learn from what information you have


DDP versus DFP
Defects
found
by
tesPng

Defect
DetecPon
Percentage
=

all
defects
found

Testing
Defects

found

and
fixed

Pre-‐release

Defect
Fix
Percentage
=

Defects

found

and
not
fixed

Defects
found

aYer
release

defects
fixed
before
release

all
defects
found
before
release

19


Top 10 common responses from the survey
“what
do
senior
managers
actually

look
for
in
a
test
manager?”


Top 10 surprise responses from the survey

“what
do
senior
managers
actually

look
for
in
a
test
manager?”

20


Ways forward for you

•  set yourself an agenda
–  choose most important improvement areas
–  be practical
–  plan the assessment of the changes
•  plan actions and dates
–  assign responsibility
–  how to support the initiative
–  timetable for improvement milestones
•  the value of today is what you do next

Passionate about Testing, Excited by Technology

Questions

?
Phone: +44 (0)113 8151779
Email: julie.gardiner@thetestpeople.com
Web:Julie
Ghttp://www.thetestpeople.com
©

ardiner

21

Test Managers: How You Can Really Make a Difference

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (16)

Similaire à Test Managers: How You Can Really Make a Difference

Similaire à Test Managers: How You Can Really Make a Difference (20)

Plus de TechWell

Plus de TechWell (20)

Dernier

Dernier (20)

Test Managers: How You Can Really Make a Difference