Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs

Hypothesis Based Testing - Application and
Adaptation for testing Enterprise Software
Satvik Kini, Suite Test Center, SAP Labs India Pvt Ltd., Bangalore
December 19, 2013

Public

Agenda
 Hypothesis Based Tests – Overview

 Traditional Test Approach vs. HBT
 Stages in HBT Approach – (Detailed Explanation with a case study)

 Challenges and Benefits
 Key factors for success

 Summary
 Q&A

© 2013 SAP AG or an SAP affiliate company. All rights reserved.

Public

2

Hypothesis Based Tests - Overview

Hypothesis :

Tentative explanation for a certain behavior, phenomenon, events that have occurred or will occur
•

Null Hypothesis – H0 – Eg: Heavy Smoking is the cause of Lung Cancer

•

Alternate Hypothesis – H1 – Eg: Heavy Smoking is not the cause of Lung Cancer

Hypothesis Based test:*
Goal focused approach, test methodology based on solid scientific principles built on the core theme of
hypothesizing potential defects.
* HBT is the IP of STAG Software.


Public

4

The Scientific Method
Form a Question

Does my software work ?

Research

Understand Context

Hypothesis

Hypothesize Defects

Experiment

Develop & Execute Tests

Analyze Results

Pass or Fail ?

Draw Conclusions

Does my software work ?


Public

5

Traditional Test Approach Vs. HBT

Traditional Test Approach Vs. HBT
Goal

Activities
Manual Test Cases
Automated Test Cases
---------------------------------------------------------------------------------Hopefully results in
Defects

Determines
Risks/
Potential
Defects

Drives
Activities
Test Levels
Test Design
Test Execution
------------------------------------------------------Defect Elimination

Focus: Coverage, Automation,
Test Scripts

Clean
Software

Focus: Defect Elimination based
approach for cleanliness
Public

7

Traditional Test Approach - Limitations
Limitations of Traditional Approach
•
•
•
•

Often Software is passed with X% OK Rate
Assumption – existing test cases cover desired parts of the software
Hopeful encounter of bugs
Focus on Quantitative reporting


Public

8

HBT – An Example
Example 1 : Clean Water
Goal: Having Clean Water
Potential Defects in water: color, odour, suspended particles, bacteria
Cleaning Process:
• Conduct activities to first determine if each of the potential defects exists
• If a potential defect exists, then take steps to clean that defect
• Once all defects are removed, re-test the water for all potential defects and certify that the water is
clean
Result: Clean Water (goal is achieved)


Public

10

HBT - Stages
Define
Cleanliness
Criteria

Define
Potential
Defect Types
(PDT)

Perform Testing

Analyze and
Conclude

Understand End
user expectation,
System Behavior

Consider All
possible different
defect types and
identify each with
test quality levels

Use of different
test techniques
and tools to test
each of the PDTs

Document and
Analyze the
results and get
an overall picture
on Quality


Public

11

Stages in HBT – Case Study
Scenario – Student Course Registration and Completion

User Types

Equivalency
Determination

•

Course Registration

Student

•

Admin

•

Record External
Achievements

Academic
Advisor/Instructor

Attendance Tracking

Market Places
Produce Transcript

Grading

Examination

US Region

•

Europe Region

•


•

APJ Region

Public

12

HBT – Cleanliness Criteria (Goal)
Example 2 : End to End (E2E) Scenario tests
Define
Cleanliness
Criteria

Goal :

Define
Potential
Defect Types

Perform Testing

Analyze and
Conclude

The E2E Scenario Student Course Registration and Completion works fine


Public

13

HBT – Define Potential Defect Types
Define
Cleanliness
Criteria

Define
Potential
Defect Types


Perform Testing

Analyze and
Conclude

Public

14

HBT – Test Levels
Level 5
Expert Level E2E Tests

Level 4
Advanced E2E Scenario
Tests
Level 3
Integration tests
Level 2
Primary E2E Use Cases
Level 1
Transactional Coverage

workflows; role based
tests;Accessibility
Integration across
scenarios

Multiple Users;
Timezone based;
Less Used features

Integration defects;
Reporting
usability

Key E2E Requirements
Browser compatibility
Performance

Obvious Errors
Runtime Errors
Commonly used E2E
Use cases

•
•
•

Each level corresponds to a set of potential Detect Types (PDTs)
A purposeful detection of PDTs
Only once all PDTs are tested for, a level is called to be reached


Public

15

HBT – Test Matrix

Creation of Report Card - Derive Test Matrix

Test levels

Potential Defect Types

Level 1

Runtime Errors

Obvious Functional Errors

Commonly used E2E usecases

Level 2

Broswer Compatibility

Calculations

Performance

Level 3

Integration

Reports

Usability

Level 4

Currency conversion

Less used features

Timezone based

Level 5

Workflows

Multiple Users

Accessibility


Integration to other scenarios

Public

16

HBT – Perform Testing
Define
Cleanliness
Criteria

Define
Potential
Defect Types

Perform Testing

Analyze and
Conclude

I. Use of Existing Test Techniques/Tools :
1.

Test case driven testing or scripted testing
Automated test cases (only scripted tests)
b. Manual test cases (also giving the tester liberty of ad-hoc tests)
a.

2.

All pairs testing

3.

Exploratory testing
Usage of Specific Test tools

4.

II. Preparation of Charters


Public

17

HBT – Analyze and Conclude
Define
Cleanliness
Criteria

Define
Potential
Defect Types

Perform Testing

Analyze and
Conclude

•

Document the results using charters/protocols

•

Check the test results for each of the PDTs

•

Update Report Card/Test Matrix - get a overall picture on Product Quality


Public

18

HBT – Test Matrix
Update Report Card

Test levels

Potential Defects Types

Level 1

Runtime Errors

Obvious Functional
Errors

Commonly used E2E
usecases

Level 2

Broswer Compatibility

Calculations

Performance

Level 3

Integration

Reports

Usability

Level 4

Currency conversion

Less used features

Timezone based

Level 5

Workflows

Multiple Users

Accessibility


Integration to
other scenarios

Public

19

Summary - HBT Realization
Define Cleanliness Criteria

Prepare PDT List

Derive Test Matrix
Prepare Charters

Test Using Charters

Protocol Logging

Update Report Card


Public

20

Challenges, Benefits and Key Factors for
Success

Challenges
Challenges
•

Understanding different user contexts –
market places

•

Existence of custom code on top of
standard

Ads
•


Highly configurable software –
customization and Business Functions

Public

22

Benefits
Benefits
• Confidence on what is tested
• Exploration of new scenarios as we do
the tests
• Reduction in customer messages

Ads

• Test Matrix gives an overall picture of
quality

• Higher Probability of finding defects as
the goal is to check existence of PDT
• Improvement - Code Coverage


Public

23

Key Factors for Success
Key Factors for Success
• Success mainly depends on
Understanding internal and external
factors and list of PDTs
• Identifying each PDT and its
corresponding level

Ads

• For each of the PDT, choosing right set
of scenarios/risks
• Choosing right set of tools and test
techniques to test each PDT


Public

24

Summary

Goal focused approach, test methodology based on solid scientific principles built
on the core theme of hypothesizing potential defects.

Public

26

Thank you
Contact information:
Satvik Kini
Associate Quality Manager
SAP Labs India Pvt. Ltd, Bangalore
+91 98863 89654
satvik.kini@sap.com

APPENDIX


Stages in HBT - Realization

BACK

Example of Charter :
Scope

PDT

Risks To Manage

How

Test Techniques

Tester

All Business
Processes

Common Used E2E
use cases

Check the straight
forward E2E scenario
is working

Run Automated Test
cases

Automated test
cases
Garbage Collector
Tour

Browser
Compatibility

Check Attendance
Tracking UI is
compatible with IE9,
Chrome Version 30,
Firefox, Safari

Check All the Tabs,
Popups, F4 Helps,
Result List, Status
Icons, Calendar
Display

Exploratory Tours :
Super Model Tour,
Garbage Collector
Tour

Mr. B

Equivalency
Determination

Obvious Functional
Errors

Mapping of External
Achievements to
Internal subjects

Try different
achievements with
different status and
validity dates

Exploratory Tour :
Intellectual Tour

Additional Comments

Mr. A

Attendance Tracking
UI

Messages

Mr. C


All pair Testing

Public

30

Stages in HBT - Realization

BACK

Update each Charter after testing each of the PDTs
Scope

PDT

Risks To Manage

How

Test Techniques

Tester

Messages

Additional Comments

All Business
Processes

Common Used E2E
use cases

Check the straight
forward E2E scenario
is working

Run Automated Test
cases

Automated test
cases
Landmark Tour

Mr. A

None

All Business
processes are
working fine

Attendance Tracking
UI

Browser
Compatibility

Check Attendance
Tracking UI is
compatible with IE9,
Chrome Version 30,
Firefox, Safari

Check All the Tabs,
Popups, F4 Helps,
Result List, Status
Icons, Calendar
Display

Exploratory Tours :
Super Model Tour,
Garbage Collector

Mr. B

23348743
32476764

Checked browser
compatible for the
IE9,chrome,Safari .
Yet to check for
Firefox

Equivalency
Determination

Obvious Functional
Errors

Mapping of External
Achievements to
Internal subjects

Try different
achievements with
different status and
validity dates

Exploratory Tour :
Intellectual Tour

Mr. C

43234234

Runtime Error while
clicking on the Sort
Button
Blocked


All pair Testing

Public

31

Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

Similaire à Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs

Similaire à Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs (20)

Plus de STAG Software Private Limited

Plus de STAG Software Private Limited (20)

Dernier

Dernier (20)

Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs