Contenu connexe Similaire à Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs (20) Plus de STAG Software Private Limited (20) Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs1. Hypothesis Based Testing - Application and
Adaptation for testing Enterprise Software
Satvik Kini, Suite Test Center, SAP Labs India Pvt Ltd., Bangalore
December 19, 2013
Public
2. Agenda
Hypothesis Based Tests – Overview
Traditional Test Approach vs. HBT
Stages in HBT Approach – (Detailed Explanation with a case study)
Challenges and Benefits
Key factors for success
Summary
Q&A
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
2
4. Hypothesis Based Tests - Overview
Hypothesis :
Tentative explanation for a certain behavior, phenomenon, events that have occurred or will occur
•
Null Hypothesis – H0 – Eg: Heavy Smoking is the cause of Lung Cancer
•
Alternate Hypothesis – H1 – Eg: Heavy Smoking is not the cause of Lung Cancer
Hypothesis Based test:*
Goal focused approach, test methodology based on solid scientific principles built on the core theme of
hypothesizing potential defects.
* HBT is the IP of STAG Software.
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
4
5. Hypothesis Based Tests - Overview
The Scientific Method
Form a Question
Does my software work ?
Research
Understand Context
Hypothesis
Hypothesize Defects
Experiment
Develop & Execute Tests
Analyze Results
Pass or Fail ?
Draw Conclusions
Does my software work ?
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
5
7. Traditional Test Approach Vs. HBT
Goal
Activities
Manual Test Cases
Automated Test Cases
---------------------------------------------------------------------------------Hopefully results in
Defects
Determines
Risks/
Potential
Defects
Drives
Activities
Test Levels
Test Design
Test Execution
------------------------------------------------------Defect Elimination
Focus: Coverage, Automation,
Test Scripts
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Clean
Software
Focus: Defect Elimination based
approach for cleanliness
Public
7
8. Traditional Test Approach - Limitations
Limitations of Traditional Approach
•
•
•
•
Often Software is passed with X% OK Rate
Assumption – existing test cases cover desired parts of the software
Hopeful encounter of bugs
Focus on Quantitative reporting
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
8
10. HBT – An Example
Example 1 : Clean Water
Goal: Having Clean Water
Potential Defects in water: color, odour, suspended particles, bacteria
Cleaning Process:
• Conduct activities to first determine if each of the potential defects exists
• If a potential defect exists, then take steps to clean that defect
• Once all defects are removed, re-test the water for all potential defects and certify that the water is
clean
Result: Clean Water (goal is achieved)
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
10
11. HBT - Stages
Define
Cleanliness
Criteria
Define
Potential
Defect Types
(PDT)
Perform Testing
Analyze and
Conclude
Understand End
user expectation,
System Behavior
Consider All
possible different
defect types and
identify each with
test quality levels
Use of different
test techniques
and tools to test
each of the PDTs
Document and
Analyze the
results and get
an overall picture
on Quality
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
11
12. Stages in HBT – Case Study
Scenario – Student Course Registration and Completion
User Types
Equivalency
Determination
•
Course Registration
Student
•
Admin
•
Record External
Achievements
Academic
Advisor/Instructor
Attendance Tracking
Market Places
Produce Transcript
Grading
Examination
US Region
•
Europe Region
•
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
•
APJ Region
Public
12
13. HBT – Cleanliness Criteria (Goal)
Example 2 : End to End (E2E) Scenario tests
Define
Cleanliness
Criteria
Goal :
Define
Potential
Defect Types
Perform Testing
Analyze and
Conclude
The E2E Scenario Student Course Registration and Completion works fine
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
13
14. HBT – Define Potential Defect Types
Define
Cleanliness
Criteria
Define
Potential
Defect Types
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Perform Testing
Analyze and
Conclude
Public
14
15. HBT – Test Levels
Level 5
Expert Level E2E Tests
Level 4
Advanced E2E Scenario
Tests
Level 3
Integration tests
Level 2
Primary E2E Use Cases
Level 1
Transactional Coverage
workflows; role based
tests;Accessibility
Integration across
scenarios
Multiple Users;
Timezone based;
Less Used features
Integration defects;
Reporting
usability
Key E2E Requirements
Browser compatibility
Performance
Obvious Errors
Runtime Errors
Commonly used E2E
Use cases
•
•
•
Each level corresponds to a set of potential Detect Types (PDTs)
A purposeful detection of PDTs
Only once all PDTs are tested for, a level is called to be reached
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
15
16. HBT – Test Matrix
Creation of Report Card - Derive Test Matrix
Test levels
Potential Defect Types
Level 1
Runtime Errors
Obvious Functional Errors
Commonly used E2E usecases
Level 2
Broswer Compatibility
Calculations
Performance
Level 3
Integration
Reports
Usability
Level 4
Currency conversion
Less used features
Timezone based
Level 5
Workflows
Multiple Users
Accessibility
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Integration to other scenarios
Public
16
17. HBT – Perform Testing
Define
Cleanliness
Criteria
Define
Potential
Defect Types
Perform Testing
Analyze and
Conclude
I. Use of Existing Test Techniques/Tools :
1.
Test case driven testing or scripted testing
Automated test cases (only scripted tests)
b. Manual test cases (also giving the tester liberty of ad-hoc tests)
a.
2.
All pairs testing
3.
Exploratory testing
Usage of Specific Test tools
4.
II. Preparation of Charters
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
17
18. HBT – Analyze and Conclude
Define
Cleanliness
Criteria
Define
Potential
Defect Types
Perform Testing
Analyze and
Conclude
•
Document the results using charters/protocols
•
Check the test results for each of the PDTs
•
Update Report Card/Test Matrix - get a overall picture on Product Quality
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
18
19. HBT – Test Matrix
Update Report Card
Test levels
Potential Defects Types
Level 1
Runtime Errors
Obvious Functional
Errors
Commonly used E2E
usecases
Level 2
Broswer Compatibility
Calculations
Performance
Level 3
Integration
Reports
Usability
Level 4
Currency conversion
Less used features
Timezone based
Level 5
Workflows
Multiple Users
Accessibility
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Integration to
other scenarios
Public
19
20. Summary - HBT Realization
Define Cleanliness Criteria
Prepare PDT List
Derive Test Matrix
Prepare Charters
Test Using Charters
Protocol Logging
Update Report Card
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
20
22. Challenges
Challenges
•
Understanding different user contexts –
market places
•
Existence of custom code on top of
standard
Ads
•
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Highly configurable software –
customization and Business Functions
Public
22
23. Benefits
Benefits
• Confidence on what is tested
• Exploration of new scenarios as we do
the tests
• Reduction in customer messages
Ads
• Test Matrix gives an overall picture of
quality
• Higher Probability of finding defects as
the goal is to check existence of PDT
• Improvement - Code Coverage
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
23
24. Key Factors for Success
Key Factors for Success
• Success mainly depends on
Understanding internal and external
factors and list of PDTs
• Identifying each PDT and its
corresponding level
Ads
• For each of the PDT, choosing right set
of scenarios/risks
• Choosing right set of tools and test
techniques to test each PDT
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
24
26. Summary
Goal focused approach, test methodology based on solid scientific principles built
on the core theme of hypothesizing potential defects.
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Public
26
28. Thank you
Contact information:
Satvik Kini
Associate Quality Manager
SAP Labs India Pvt. Ltd, Bangalore
+91 98863 89654
satvik.kini@sap.com
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
30. Stages in HBT - Realization
BACK
Example of Charter :
Scope
PDT
Risks To Manage
How
Test Techniques
Tester
All Business
Processes
Common Used E2E
use cases
Check the straight
forward E2E scenario
is working
Run Automated Test
cases
Automated test
cases
Garbage Collector
Tour
Browser
Compatibility
Check Attendance
Tracking UI is
compatible with IE9,
Chrome Version 30,
Firefox, Safari
Check All the Tabs,
Popups, F4 Helps,
Result List, Status
Icons, Calendar
Display
Exploratory Tours :
Super Model Tour,
Garbage Collector
Tour
Mr. B
Equivalency
Determination
Obvious Functional
Errors
Mapping of External
Achievements to
Internal subjects
Try different
achievements with
different status and
validity dates
Exploratory Tour :
Intellectual Tour
Additional Comments
Mr. A
Attendance Tracking
UI
Messages
Mr. C
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
All pair Testing
Public
30
31. Stages in HBT - Realization
BACK
Update each Charter after testing each of the PDTs
Scope
PDT
Risks To Manage
How
Test Techniques
Tester
Messages
Additional Comments
All Business
Processes
Common Used E2E
use cases
Check the straight
forward E2E scenario
is working
Run Automated Test
cases
Automated test
cases
Landmark Tour
Mr. A
None
All Business
processes are
working fine
Attendance Tracking
UI
Browser
Compatibility
Check Attendance
Tracking UI is
compatible with IE9,
Chrome Version 30,
Firefox, Safari
Check All the Tabs,
Popups, F4 Helps,
Result List, Status
Icons, Calendar
Display
Exploratory Tours :
Super Model Tour,
Garbage Collector
Mr. B
23348743
32476764
Checked browser
compatible for the
IE9,chrome,Safari .
Yet to check for
Firefox
Equivalency
Determination
Obvious Functional
Errors
Mapping of External
Achievements to
Internal subjects
Try different
achievements with
different status and
validity dates
Exploratory Tour :
Intellectual Tour
Mr. C
43234234
Runtime Error while
clicking on the Sort
Button
Blocked
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
All pair Testing
Public
31