Visit SoftTest Ireland www.softtest.ie and sign up for access to free Irish Software Testing events.

1. People Challenges
for Test Teams
Stephen K. Allott
SoftTest Ireland Copyright 2008 ElectroMind Ltd 1

2. SoftTest Ireland Copyright 2008 ElectroMind Ltd 2

3. SoftTest Ireland Copyright 2008 ElectroMind Ltd 3

4. SoftTest Ireland Copyright 2008 ElectroMind Ltd 4

5. Introductions
My background
Aims of this workshop
Explain some common
testing challenges based on
recent experience
Ask for your experiences
Agree a list of challenges
we all face as testers
Try to come up with some
quick wins & longer term
approaches to solutions
SoftTest Ireland Copyright 2008 ElectroMind Ltd 5

6. My recent clients
Investment bank
Airline web site
Travel company web site
Mobile phone operator
Retail web site
Travel company back office
Recruitment web site
Includes assignments done in association with TestHouse (London)
and Neueda Technologies (Antrim & Dublin)
SoftTest Ireland Copyright 2008 ElectroMind Ltd 6

7. Their technologies
Microsoft Dot (.) NET 1/2/3
Microsoft VSTS / TFS
Oracle
MS SQL Server
Windows XP
Citrix Metaframe
Quality Centre / QTP
SoftTest Ireland Copyright 2008 ElectroMind Ltd 7

8. Number of testers
Typically 12 to 14 in the test team
A couple of clients had only 1, some have none!
SoftTest Ireland Copyright 2008 ElectroMind Ltd 8

9. Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 13
Key Area Controlled Efficient Optimising
Test strategy A B C D
Life-cycle model A B
Moment of involvement A B C D
Estimating and planning A B
Test specification techniques A B
Static test techniques A B
Metrics A B C D
Test automation A B C
Test environment A B C
Office environment A
Commitment and motivation A B C
Test functions and training A B C
Scope of methodology A B C
Communication A B C
Reporting A B C D
Defect management A B C
Testware management A B C D
Test process management A B C
Evaluation A B
Low-level testing A B C
Composite TPI Test Maturity Matrix from several clients
SoftTest Ireland Copyright 2008 ElectroMind Ltd 9

10. Summary of TPI results
No documented test process
No formal test policy, strategy or plan
Many tests, no formal design
No metrics
No proper test environment
Some tested in “live”
No documented test basis
Not involved early enough
SoftTest Ireland Copyright 2008 ElectroMind Ltd 10

11. Some common observations
Constantly changing business priorities
Increasing (frantic) demand for new functionality
Weekly, sometimes daily release cycles
Attempt to use automation to solve all the problems
No numbers to explain testing’s added value
Small important teams yet with little influence
SoftTest Ireland Copyright 2008 ElectroMind Ltd 11

12. People challenges – behaviour
Motivation
A few are resigned to how it is, no hope?
Some expect spoon feeding
Some adopt a negative attitude
Many are willing to make improvements, if there was time
Turnover – very high offshore
Culture
Some managers get too involved in detail
Co-dependency – testers must push back
It hasn’t broken yet so . . .
SoftTest Ireland Copyright 2008 ElectroMind Ltd 12

13. People challenges – skills
Testing Skills IT Skills
Processes Programming
Standards Databases
Techniques Infrastructure
Tools Web
Domain Knowledge Soft Skills
Industry specific Communication
Application knowledge Negotiation
E-Business Managing technical teams
SoftTest Ireland Copyright 2008 ElectroMind Ltd 13

14. People challenges – perception
In general testers are not valued
Testers are often blamed
Project delays
Poor quality
Managing expectations with RBT
Lack of awareness of what the test team does
SoftTest Ireland Copyright 2008 ElectroMind Ltd 14

15. People challenges – managers
Demand for certified staff
Demand for skills & knowledge
Pressure to cut costs and use offshore resources
Developing good test leads
Role of testers in an agile project
Selling the value of testing to the organisation
SoftTest Ireland Copyright 2008 ElectroMind Ltd 15

16. Themes
Facilitated discussion bearing in mind the following themes:
1. Relentless business pressure for new functionality
2. Constant and last minute changes to requirements
3. Introduction of new development / testing methodologies e.g. agile
4. Very frequent releases, often weekly, sometimes daily!
5. Increasing application complexity over time
6. Knowledge gap (e.g. what lies beyond the browser?)
7. Steep learning curve (business as well as technical)
8. Use of contract and offshore resources (loss of in-house expertise)
9. Little or no documentation against which to test
10. Lack of attention to all testing requirements (e.g. middleware, back
end database, non-functional quality characteristics)
SoftTest Ireland Copyright 2008 ElectroMind Ltd 16

17. Discussion – how do we respond?
Business Pressures Technology Offshore
New functionality Complex Cheaper
Frequent change Rapid change Plentiful
Maturity Interfaces Expertise
Behaviour Skills Managers
Motivation Analysis Costs
Attitude IT / Technical Time to market
Culture Business Agile
Testing
Soft skills
SoftTest Ireland Copyright 2008 ElectroMind Ltd 17

18. Conclusions
In my opinion testers of the future must . . .
Measure / appreciate application complexity
Be able to accept and manage last minute changes
Be flexible but push back with estimates
Increase their technical knowledge
Learn more about testing, especially techniques
Apply the tried and tested techniques
Capture metrics to help prove the business case
Get involved in the projects much earlier
Recruit / train / certify / develop / coach = Retain
Share your views with the group
SoftTest Ireland Copyright 2008 ElectroMind Ltd 18

19. Final Thoughts / Questions?
SoftTest Ireland Copyright 2008 ElectroMind Ltd 19

20. People Challenges
for Test Teams
stephen.allott@electromind.com
+44(0) 773 476 1363
www.electromind.com
SoftTest Ireland Copyright 2008 ElectroMind Ltd 20

21. People Challenges
for Test Teams
Appendix – Bonus Slides
SoftTest Ireland Copyright 2008 ElectroMind Ltd 21

22. A typical project in trouble
CR Churn by Month - AveCR=47 AveChurn=150 AveChurnCR=3
300 279 300
263
250 254 250
229
213
200 200
Changes to CRs
CR count
150 150
116
124
100 100
65
58 55
50 40 50
27
14
0 0 0 0
June July August September October
Months
CRs Churn Bugs
SoftTest Ireland Copyright 2008 ElectroMind Ltd 22

23. Bug snapshot by priority
Sev 1 Sev 2 Sev 3 Sev 4 Total
Pri 1 57 81 14 152
Pri 2 19 52 12 1 84
Pri 3 6 14 10 30
Pri 4 2 1 3
None 68 288 114 10 480
Total 146 428 154 21 749
SoftTest Ireland Copyright 2008 ElectroMind Ltd 23

24. Bug snapshot by resolution
Sev 1 Sev 2 Sev 3 Sev 4 Total
By design 10 37 29 4 80
Duplicate 6 33 7 1 47
External 2 16 4 0 22
Fixed 86 254 95 12 447
Not repro 29 56 4 0 89
Postponed 12 17 6 4 39
Will not fix 1 15 9 0 25
Total 146 428 154 21 749
SoftTest Ireland Copyright 2008 ElectroMind Ltd 24

25. Impact of the improvements
CR Churn by Month - AveCR=47 AveChurn=150 AveChurnCR=3
300 279 300
263
250 254 250
229
213
Changes to CRs
200 200
CR count
150 150
116
124
100 86 100
65 66 90
58 55 61
50 40 50
27 30
14 17 18 15 15 10 10 5 5
13 12 9
0 0 0 5 1 0
r
ry
ne
ry
r
ly
ch
ril
r
er
t
be
be
be
s
Ju
Ap
ua
a
gu
ob
Ju
ar
m
nu
m
em
br
M
Au
ct
ve
ce
Ja
Fe
O
pt
No
De
Se
Months
CRs Churn Bugs
SoftTest Ireland Copyright 2008 ElectroMind Ltd 25

26. Corporate Sales and Sales and Marketing Systems
1. Consulting Practice Mgt.
Marketing Systems 2. Vendor Account Mgt.
Events Systems 3. Event Mgt.
1. Vendor 4. Leads and Referral
2. User Reg. Databases
3. User Shop 1. Profile
2. Common
3. Event OLTP
System Interfaces 4. Transaction OLTP
1. Event Orders 5. Contact History OLTP
2. Billing & Financials 6. Domain Reference
3. Credit Card (real-time)
4. Event Communication / FAX
5. Direct email service World-wide Marketing DB
6. Direct mail Postal interface 1. Profile OLAP
7. Leads & Referrals 2. Event OLAP
8. OLTP Staging Interface 3. Analytical Reporting
9. OLAP Staging Interface 4. Contract History OLAP
10. SAP 5. Domain
11. Licensing
12. Call Center
Source: Reality Test Inc.

27. Basic complexity calculation
Characteristic Value Factor Total
Windows 37 8 296
Objects 23 1 23
Tables 47 1 47
Apps Impacted 3 10 30
Factors are weighted from a testing perspective
SoftTest Ireland Copyright 2008 ElectroMind Ltd 27

28. Comparison across applications
Application UI Back Env Test Middle Total
End Code Tier
Sales Mgr. 1360 2250 152 0 0 3762
Call Center 960 824 90 129 0 2003
Web Sales 389 183 109 0 2 681
Middleware 0 146 23 27 232 428
component
SoftTest Ireland Copyright 2008 ElectroMind Ltd 28

29. Complexity comparison
4000
Complexity Value
3000
2000
1000
0
er
er
r
e
ss
TR
ta
es
te
as
ag
ag
el
la
en
al
C
B
D
G
an
an
S
C
SS
e
g
eb
M
M
dg
l
in
al
P
W
nt
es
ok
le
C
ou
w
al
Lo
no
S
cc
K
A
Application Name
SoftTest Ireland Copyright 2008 ElectroMind Ltd 29

30. Actual staffing comparison
20
Number of Testers
15 Estimated
Actual
10
5
0
er
er
r
e
ss
TR
ta
es
te
as
ag
ag
el
la
en
al
C
B
D
G
an
an
S
C
SS
e
eb
g
M
M
dg
l
in
al
P
W
nt
es
ok
C
le
ou
w
al
Lo
no
S
cc
K
A
Applications
SoftTest Ireland Copyright 2008 ElectroMind Ltd 30

31. Organisation evaluation
Organisation within the company
Process definition and execution
Teamwork
Ability to do the job
SoftTest Ireland Copyright 2008 ElectroMind Ltd 31

32. Organisation within the company
Section Item being evaluated Max Score
1.1 Reporting structure 50
1.2 Professionalism 170
1.3 Organisational churn 135
1.4a Staffing – permanence 30
1.4b Staffing – stability 35
1.4c Staffing – morale 50
1.4d Staffing – number of testers 40
1.4e Staffing – organisation type 40
1.5 Budget control 30
SoftTest Ireland Copyright 2008 ElectroMind Ltd 32

33. Process definition and execution
Section Item being evaluated Max Score
2.1 Software development life-cycle 60
2.2 Design reviews and documentation 70
2.3 Early participation in development 50
2.4 Ability to influence schedule deadlines 60
2.5 Development standards 60
2.6 Defined testing approach & procedures 190
2.7 Metrics reporting 60
2.8 Churn 90
2.9 Patch releases 40
2.10 Post-project reviews 60
SoftTest Ireland Copyright 2008 ElectroMind Ltd 33

34. Teamwork
Section Item being evaluated Max Score
3.1 Support from production support 60
3.2 Support from management 60
3.3 Correct level of quality 90
3.4 Resource allocation 80
SoftTest Ireland Copyright 2008 ElectroMind Ltd 34

35. Ability to do the job
Section Item being evaluated Max Score
4.1 Adequate equipment / lab 200
4.2 Automated testing 70
4.3 Programming ability 70
4.4 Non-testing activities 50
SoftTest Ireland Copyright 2008 ElectroMind Ltd 35

36. Total evaluation score
1300
Total Evaluation Score
1200
Evaluation
1100
1000
900
Q1/00 Q2/00 Q3/00 Q4/00 Q1/01 Q2/01 Q3/01
Date
SoftTest Ireland Copyright 2008 ElectroMind Ltd 36

37. Organisational Evaluation
500
Organization
400 Process
Teaming
Evaluation
Ability
300
200
100
Q1/00 Q2/00 Q3/00 Q4/00 Q1/01 Q2/01 Q3/01
Date
SoftTest Ireland Copyright 2008 ElectroMind Ltd 37

38. Components of a centre of excellence
l Tes
Project 1 p oo t l ab
e
o urc
s
Re Tool Experts
Project 2
Test Managers SIG
Project 3
e
Knowledge Bas
C
on
Project 4
re
su
ent
lta
DC
nt
R&
s
SoftTest Ireland Copyright 2008 ElectroMind Ltd 38

39. Benefits of a centre of excellence
Independent testing Variety of work
Breadth & depth Identity, pride
Fewer product recalls Morale, professionalism
Fewer production failures Extra revenue stream
Central reference point Consistent approach
Knowledgeable staff Multi-methodologies
Expertise across org. Earlier involvement
Structured career path
SoftTest Ireland Copyright 2008 ElectroMind Ltd 39