EuroSTAR Software Testing Conference 2010 presentation on Testing Package Solutions: Business as usual? by Tim Koomen. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/

1. 1
Testingpackage solutions: business as usual?
Tim Koomen EuroSTAR2010

2. Seite 2
Start
Package solutions: business as usual?
Testing
QA
End
Test / QA staff

3. Wanted:package solution!
Business processes
Support processes
IT-policy
Standards
Other applications
3
Testing package solutions: business as usual?
Architecture
Infrastructure

4. Found:package solution
Testing package solutions: business as usual?
4
Business processes
Support processes
IT-policyStandardsOther applications
Architecture
Infrastructure

5. Implemented: customized package
Testing package solutions: business as usual?
5
Parameters
Standard
software
Custom
made
Business processes
Support processes
IT-policy
Standards
Other applications
ArchitectureInfrastructure

6. Life-cycle
Packageselection
Implementation
Exploitation& maintenance
Testing package solutions: business as usual?
6
Changes in
system
management
Acceptance

7. There are no risks, because ...
The salesguytellsme thispackageis perfect forourorganisation ...
The packagesupplieralreadyfoundall defects ...
My implementationpartner takescare of everything...
We usethe packageas is ...
The design of the modifiedbusiness processesis clearto the (super)usersand implementationpartner ...
The packageshouldworkon ourinfrastructure...
The packageis veryuser-friendly, solittletraining is required...
Ifthereare anyproblems, we justquicklychangea few settings ...
Oncein production, we canveryeasilybringout newreleases ...
7
Testing package solutions: business as usual?

8. Or are there, because...
Does the package functionality cover most of our requirements?
Has the package been parameterised correctly?
Is any custom made software built? Does it work correctly?
Does the package (still) work with business processes, existing systems, operational procedures, infrastructure, etc.?
Have the authorisations been defined properly?
Can the user work with the changed business processes and with the package?
Is the performance of the package sufficient?
Can the old data be converted correctly?
Can system managers install and operate the package?
8
Testing package solutions: business as usual?

9. Somenewsitems ...
[July 2010] IBM says Queensland Health SAP failure is not its faultBut the start date for the project expanded from 8 to 26 months and when it finally went live on March 14, a litany of major problems ensued. Thousands of health staffers were incorrectly paid and the cost for the project blew out from an initial $6.19 million to $64.5m
[June 2010] In a important case, Marin County, California filed a complaint against Deloitte Consulting for its role in an over-budget SAP implementation. [.. ] still not working four years after it initially went live […] damages of at least $30 million
[February, 2009] Another High-Profile SAP Failure: State Of California Yet another black eye for German software giant SAP (SAP). Now the State of California --already in dire financial straits --is giving up on its SAP implementation after sinking $25 million into the project and seeing nothing out of it.
[March, 2008] Waste Management Inc.'s lawsuit against SAP for the "complete failure" of a $100 million software implementation could bruise the credibility of SAP's vertical-market strategy. Waste Management claims SAP duped it into purchasing untested software that wasn't ready to handle the complexities of the U.S. waste hauling market. ”
Testing package solutions: business as usual?
9

10. And not only SAP ...
[August 2008] “Prone to Failure: Why CRM and Billing Systems Implementations Are High Risk”
Testing package solutions: business as usual?
10
(Dr. Raul Katz, Columbia Business School, presents findings from research on why certain telecom implementations are doomed from the start)

11. Siebel ... ...
Testing package solutions: business as usual?
11

12. Risk control
Package solutions, both in implementation and maintenance, differfrom custom made software
Myth: “packages are low risk solutions”
Underestimationof many (other) risks
“Changing software is peanuts compared to changing people”
Testing package solutions: business as usual?
12

13. Seite 13
StartPackage solutions: business as usual?
Testing
QA
End
Test / QA staff

14. From Wikipedia ...
“SAP implementation...”
Testing is very important before going live with any system. Before going live with a SAP system, it is vital to do many different kinds of testing, since there is often a large, complex infrastructure of hardware and software involved. Both requirements as well as quality parameters are to be tested. Important types of testing are:
Functional testing: to test using functional use cases, i.e. a set of conditions or variables under which a tester will determine if a certain business process works
Integration testing
Regression testing
All tests should be preceded by creating solid test plans.”
Testing package solutions: business as usual?
14

15. Steps risks-driven testing
15
Testing package solutions: business as usual?

16. Product riskRisk
Frequency of Use
Chance of fault
Chance of
failureDamage
*
*
16
Testing package solutions: business as usual?

17. Determining participants
Risk
Frequency of Use Chance of fault
Damage
*
*
• (End) users
• (Functional and
system) managers
• Line managers
• Project manager
• Client
Chance of
failure
17 Testing package solutions: business as usual?
• Process (re)designers
• Architects
• Package consultants
• Developers
• DBA
• QA
• Test manager
• Project manager

18. PRA: from business to ITBusiness
IT
test goals
characteristics /
test goal
object parts /
characteristicsrisk indication per object part / characteristic
packageunder test
business processes, change requests or risks (to be covered) ,
... suitability, functionality, performance, usability, security, …
packagemodules(funct.),
workprocesses(suitability)
online, batch (perf.)
user screens(usability),
application, infra(security)
Chanceof failure
High
Medium
Low
Damage
High
A
B
B
Medium
B
B
C
Low
C
C
C
Testing package solutions: business as usual?

19. Quality attributes
More (+) orless(-) risk?
+Suitability(=fit in organisation / business processes)
+ Functionality(custom-made)
+Security
+Operability
+Infrastructure
+Performance
-Functionality(package)
-User friendliness, reusability, flexibility, ...
20
Testing package solutions: business as usual?

20. Risksregarding“as is” x “customized”
21
Testing package solutions: business as usual?
High
Low
Packageas-is
Amountof custom-madechanges

21. Test strategyMasterTest Plan
Characteristic/ object part
PRA- RiskClass
Review
Devel- oper.T
Sys. T / Funct. T
User Acc.T/
FinalInt.T
Prod
Acc.T
suitability
-module 1
Medium




-module 2
High




-totalintegrated
Medium


security
-infrastructure
Medium


-application
Medium



performance
-online
Medium
I

-batch
Low
I
S
functionality
High




manageability
Low

Test strategy
PRA
24
Testing package solutions: business as usual?

22. Test types
Some examples:
Proof-of-concept testing (selection process)
Testing individual business processes
Testing integration between processes / package
Interface tests
End-to-end (business or final integration) testing
Data migration testing
(Scalable) Regression testing
Stress testing
25
Testing package solutions: business as usual?

23. ... with a structured test process
27
Testing package solutions: business as usual?
TMap is a registered trademark of Sogeti BV

24. Example: TMap life-cycle and activities
Control
Planning
Preparation
Specification
Execution
Completion
Setup and maintain infrastructure
Collect test basis
Testability review
Assignment
PRA
Test strategy
…
Consolidate plan
Specify tests
Define starting points
Specify intake test object
Management
Monitoring
Reporting
Adjusting
Intake/pretest
Prepare starting points
(Re)test
Check & assess
Preserve Testware
Evaluate process
28
Testing package solutions: business as usual?
Specifying
Realising
Specifying intake
Intake
Maintain
Conserve

25. Test basis
 Process design (blue print)
 Change requests
 Specifications for custom-made
software
 Release notes (from package vendor)
 Informal:
 Knowledge of users
 Existing processes
 Existing software systems
29 Testing package solutions: business as usual?

26. Seite 31
Start
Package solutions: business as usual?
Testing
QA
End
Test / QA staff

27. Possible (QA) measures
Product
Process
Project
procedures: configuration / change / release / defect management
monitoring
tasks & responsibilities
project structure
communication plan
standards& templates
reviews & inspections
workshops
testing
proof of concept
acceptance
criteria
33
Testing package solutions: business as usual?

28. Seite 34
Start
Package solutions: business as usual?
Testing
QA
End
Test / QA staff

29. StaffingQA and testing
Involveddisciplines:
Packageconsultants
Business users
Infrastructure/ system management
Test/QA professionals!
Required roles:
Test manager
Testers
Test support
QA manager
35
Testing package solutions: business as usual? Discussion
•More important: test/QA or package expertise?
•Independent?

30. Seite 36
Package solutions have different risks: -Business risk instead of IT risk
-Organizational complexity
-Low risk awareness
Demands risk-based and transparent QA and Testing
(Independent) test/QA expertise and method is requiredConclusion
tak!

31. Finally ...
37 Testing package solutions: business as usual?
spørgsmål,
vraag,
question,
pregunta,
Frage,
interrogação,
domanda
frågal

32. E.
info@timkoomen.nl
M.
+31 (0)6 34139260
I.
www.timkoomen.nl
Copyright Tim Koomen Testmanagement en -advies