כיצד ניתן לצמצם את כמות הבדיקות מבלי לפגוע (ואף לשפר) בכיסוי המערכת?

IBM Haifa Research Lab

Combinatorial Test Design and the FoCuS Tool

Itai Segall
Software Testing Verification and Reviews
Group

© 2010 IBM Corporation


About us

Software Testing, Verification and Review
Methodologies group at IBM Haifa Research Labs
– Concurrent testing
– Code review methodology and tool support
– Code coverage analysis
– Test selection
– Functional modeling and test planning



Outline

Combinatorial Test Design (CTD) Concepts and
Overview
The FoCuS Tool and the Modeling and Reduction
Process

3 © 2010 IBM Corporation


Observation - Omission is a Major Risk in Testing

Choosing what to test out of an enormous test space
– What is a test space? The set of parameters/values that define the
scenarios, inputs, configurations or conditions to be tested, such as
• Customer type (business, residential, government, …)
• Customer geography (provinces)
• Transaction type (add service, delete service, request credit …)

When you have a list of 3000 tests, can it be reviewed?
How do you know what’s there and what is missing?
– Coverage (in retrospect)
– Preventive techniques using CTD



An Example of a Test Design

The configuration space for a cellphone application
– Bluetooth – Yes / No
– GPS – Yes / No
– GPS software
– Support for different GSM frequencies
– Support for different 3G frequencies
– Support for EMS and MMS

Total number of combinations
– 2 x 2 x 5 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 = 5120 Too many!



Do we really need to test all 5120 configurations?

So, to balance cost and risk, we look to select a subset of tests that covers the
interactions of configuration variables at some level of interaction (pairs, three-
way, etc.)

Source http://csrc.nist.gov/groups/SNS/acts/ftfi.html used in slides 6,8,9


How do we reduce the 5120 potential test cases?

Interactions !
– Interaction level of every 2 means that for any two variables all
the combinations of their values have been tested together.
– Every three is the same for every three variables

IBM’s FoCuS tool models the variables, and allows us
to adjust the level of interaction/coverage to achieve
acceptable cost/risk tradeoff
Explicit identification of what is tested and what is not
tested



The configuration space for a cellphone application
– Bluetooth – Yes / No
– GPS – Yes / No
– GPS software
– Support for 3 different GSM frequencies
– Support for 2 different 3G frequencies
– Support for EMS and MMS
Total number of combinations
– 2 x 2 x 5 x 2 x 2 x 2 x 2 x 2 x 2 x 2 x 2 = 5120
Question: How many tests are required for covering
every combination of every 2 parameters?
Answer: only 16 ! (99.7% reduction from 5120)


Interaction level of every 2 requires 16 tests – 99.7% reduction from 5120
GPS_Soft
Bluetooth GPS ware Wi_Fi GSM850 GSM900 GSM1800 UMTS850 UMTS2100 EMS MMS

FALSE TRUE Destinator TRUE TRUE FALSE TRUE FALSE TRUE FALSE FALSE
FALSE TRUE uDrive FALSE TRUE FALSE FALSE TRUE TRUE FALSE FALSE
TRUE FALSE None TRUE TRUE FALSE TRUE TRUE FALSE FALSE TRUE
FALSE TRUE mioMap TRUE TRUE FALSE FALSE FALSE TRUE FALSE TRUE
FALSE TRUE iGo TRUE TRUE FALSE TRUE TRUE FALSE TRUE TRUE
TRUE TRUE iGo TRUE FALSE TRUE TRUE TRUE TRUE FALSE FALSE
TRUE TRUE mioMap FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE
TRUE TRUE Destinator FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE
TRUE TRUE Destinator FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE
FALSE TRUE mioMap TRUE TRUE TRUE TRUE TRUE FALSE FALSE TRUE
TRUE FALSE None TRUE FALSE TRUE FALSE FALSE TRUE FALSE FALSE
TRUE TRUE uDrive FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE
FALSE TRUE iGo FALSE FALSE TRUE FALSE FALSE FALSE TRUE TRUE
TRUE TRUE uDrive TRUE TRUE TRUE TRUE TRUE FALSE FALSE FALSE
TRUE TRUE uDrive FALSE FALSE TRUE TRUE FALSE TRUE TRUE TRUE
FALSE FALSE None FALSE FALSE TRUE FALSE TRUE FALSE TRUE TRUE


Outline

Combinatorial Test Design (CTD) Concepts and
Overview
The FoCuS Tool and The Modeling and Reduction
Process



A Comment on the Use of CTD in the Industry

Many CTD tools were created
– The mathematics are very nice to play with
– The problem is solved with advanced combinatorics
However, most are almost toy-tools
– Real world problems are not toy examples
– Real world problems may be hard to model
– Not all value combinations are possible – there are restrictions
– Existing tests need to be evaluated or taken into account
– Advanced requirements such as distribution requirements on the
values in the resulting tests
Most tools miss many things testers need



CTD within our FoCuS tool

Tackles complexity of modeling by Cartesian product methodology
Fully supports restrictions on the valid combinations
Highly scalable
Supports advanced features
– Distribution requirements
– Good path and bad path test planning
Considers existing tests
– Report uncovered areas
– Suggest complementary test plans



The cell phone example in FoCuS



Example – cont.
Cartesian product consists of 5120 possible
configurations:



Example – Cont.
Some combinations do not make sense, and should
be removed:



Example – Cont.

Now the model contains 2560 legal tasks:



Example – Cont.

A test plan can now be created:



Example – Cont.
16 tests suffice:



Example – Cont.
Suppose I have only a few Bluetooth components



Example – Cont.
The new test plan:



Example – Cont.

More complex coverage requirements may be stated:



Example – cont.

And the resulting test plan:



Example – cont.

“.. But I already have some tests implemented, I want
to use them”



Example – cont.

FoCuS can analyze what is not covered:



Example – cont.



Example – cont.

.. and suggest a complementary test plan:



Comparing Manually Created Test Plans to Usage of
Modeling and CTD

Manual test plans will usually have many large
holes
– And it is not known where they are

Manual test plans are more expensive
– Take a lot of time to design
– Manual test plans contain more tests, so longer time to
implement the tests and longer time to execute them

Known quality situation
– With CTD you can get the desired quality based on risk
– With CTD you can compare alternatives, as once the model is
created, creating alternatives takes minutes



Additional Advantages of Systematic Test Plan Design
Once the methodology is learned, it is much less work – less tests to execute
The tool tells you how many test configurations are needed and specific
configurations that fulfill your requirements
– You can discuss alternatives for how the budget will be used
Having a model instead of a spreadsheet test list is much more useful
– Less effort to maintain the model
– From the model you can generate lists after changes are made
• A new operating system was added
• Need to add a new variable that interacts with the current ones
• Need to remove variables or values and make sure that the test plan
maintains quality
– The model is much easier to review than traditional test plans as the
generation scheme explicitly describes what is tested



Thank You

Trial download:
http://www.alphaworks.ibm.com/tech/focus
Contact me at: itais@il.ibm.com


כיצד ניתן לצמצם את כמות הבדיקות מבלי לפגוע (ואף לשפר) בכיסוי המערכת?

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (11)

Similaire à כיצד ניתן לצמצם את כמות הבדיקות מבלי לפגוע (ואף לשפר) בכיסוי המערכת?

Similaire à כיצד ניתן לצמצם את כמות הבדיקות מבלי לפגוע (ואף לשפר) בכיסוי המערכת? (17)

כיצד ניתן לצמצם את כמות הבדיקות מבלי לפגוע (ואף לשפר) בכיסוי המערכת?