For decades, software development tools and methods have evolved with an emphasis on modeling. Standards like UML and SysML are now used to develop some of the most complex systems in the world. However, test design remains a largely manual, intuitive process. Now, a significant opportunity exists for testing organizations to realize the benefits of modeling. Adam Richards describes how to leverage model-based testing to dramatically improve both test coverage and efficiency—and lower the overall cost of quality. Adam provides an overview of the basic concepts and process implications of model-based testing, including its role in agile. A survey of model types and techniques shows different model-based solutions for different kinds of testing problems. Explore tool integrations and weigh the pros and cons of model-based test development against a variety of system and project-level factors. Gain a working knowledge of the concepts, tools, and techniques needed to introduce model-based testing to your organization.
AWS Community Day CPH - Three problems of Terraform
Model-Based Testing: Concepts, Tools, and Techniques
1. W14
Concurrent Class
10/2/2013 3:00:00 PM
"Model-Based Testing:
Concepts, Tools, and
Techniques"
Presented by:
Adam Richards
Critical Logic
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
2. Adam Richards
Critical Logic
Adam Richards is the director of consulting services at Critical Logic, a consulting firm
specializing in model-based testing tools and services. With nearly two decades in the industry,
Adam has experience as a developer, business analyst, requirements manager, test analyst,
and QA manager. He brings first-hand experience implementing model-based testing
approaches on systems in a wide array of industries, across the entire gamut of development
methodologies.
3. 9/20/2013
Model-Based Testing
Concepts, Tools, and Techniques
Presented by Adam Richards
Starwest, 2013
2013 Critical Logic
If the shoe fits…
Model-Based Testing offers something for everyone.
2013 Critical Logic
2
1
4. 9/20/2013
Survey says…
Who uses MBT, how, and for what
2013 Critical Logic
3
Domains
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
4
2
5. 9/20/2013
Interfaces
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
5
Methodologies
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
6
3
6. 9/20/2013
Test Level
At what scope do you use MBT?
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
7
Test Aspects
Which aspects of the SUT do you test with MBT?
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
8
4
7. 9/20/2013
Benefits
On average, respondents report MBT:
• Reduced escaped bugs by 59%
• Reduced testing costs by 17%
• Reduced testing duration by 25%
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
9
Okay…
But what is it???
2013 Critical Logic
10
5
8. 9/20/2013
Model Soup
What we Mean by “Model”
2013 Critical Logic
11
Structural Models
•
•
•
•
•
2013 Critical Logic
Static
ERD, Class, Component, Data Model…
Can define test environment (e.g. UTP)
Foundation for Behavioral models
Used for both Rqmts & System Models
12
6
9. 9/20/2013
Behavioral Models
• Describe flow
• BPM, Activity, State, Cause-Effect…
• Can specify abstract test scenarios
(no data, test environment)
• Used for both Rqmts & System Models
2013 Critical Logic
13
Test Models
•
•
•
•
2013 Critical Logic
Behavioral model + inputs + environment
Specify tests at executable level
Automatable
May or may not include test generator
14
7
10. 9/20/2013
MBT Process Menu*
Hanging the model types together
*Substitutions welcome
2013 Critical Logic
15
Flavor 1: Model-Centric
Rqmts
Model
Rqmts
Model
•
•
•
•
Rare today (yet)
Generate all
Big investment
Big returns
•
•
•
•
2013 Critical Logic
Maint. costs
Human error
Defect escapes
Defect
introduction
16
8
11. 9/20/2013
Flavor 2a: Model-Driven
• Less uncommon
• No rqmts model
• Generate
executable tests
from test model
2013 Critical Logic
17
Flavor 2b: Model-Driven (Agile)
• Test model only
• Generate
acceptance criteria
• Maintain model,
not tests
• Supports CIT
2013 Critical Logic
18
9
12. 9/20/2013
Flavor 3: Model-Based
•
•
•
•
No test model
Walk rqmts model
Validation later
Add data, env.
2013 Critical Logic
19
MBT Flavor Rainbow
Model “Centric”
• Model-generated tests & code
Model “Driven”
• Model-generated tests
Model “Based”
• Tests based on rqmts models, built by hand
2013 Critical Logic
20
10
13. 9/20/2013
Tool Ingredients
What Test Modeling Tools Can Do
2013 Critical Logic
21
Test Model Tool Functions
Test Data Generators
Test Case Editors
Test Case Generators
>> Test Coverage Criteria
2013 Critical Logic
22
11
14. 9/20/2013
Test Coverage Criteria
Sys/Rqmt Model Coverage
• Control Flow, Data Flow
• Transition-Based (FSM, statecharts)
• UML-Based (AD, Class, Seq, etc.)
Data Coverage
• Boundary Value Analysis
• Classification Tree
• Statistical, Pairwise
Fault-Based Coverage
• Mutation, Stuck-At Fault Model, MC/DC
Requirements-Based Coverage
• All Requirements Coverage
Utting, M.; Legeard, B.: "Practical Model-Based Testing - A Tools Approach". Morgan Kaufmann Publ, Amsterdam, 2007
2013 Critical Logic
23
Test Modeling Tool Examples
Commercial
• RBT/DTT (Bender RBT, Critical Logic)
• Rhapsody ATG (IBM)
• Others (Conformiq, AETG, Leirios…)
Open Source
• ParTeG (Eclipse Plugin)
• Spec Explorer (Visual Studio)
• Others (JUMBL, Torx, Uppaal TRON…)
2013 Critical Logic
24
12
15. 9/20/2013
More Survey
What kind of MBT tools were used?
• 17% Open Source
• 30% Custom
• 53% Commercial
Testing modes used on project (avg)?
• 33.6% MBT
• 36.7% Traditional
• 29.7% Manual
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
25
Coverage
What kind of coverage is measured for your MBT test
runs?
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
26
13
16. 9/20/2013
More Survey (last one)
How do you connect MBT output, SUT control
interfaces, & SUT observation interfaces?
• 06% Manual input & observation
• 15% Manual scripting required
• 15% Tool generates code/data + manual
processing
• 26% Tool generates code/data for the SUT
interface
• 38% Tool generates code/data for an adapter that
abstracts the SUT interface
Binder, Robert V. 2011 Model-based Testing User Survey: Results and Analysis. 2012
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
2013 Critical Logic
27
Tool Challenges Opportunities
Tool Integration
• Automation, RMS, TMS, SDLC, TestIF
Coverage limited to source model
• Can it synthesize sources to generate tests?
Constraints & infeasibles
• Good coverage <> executable tests
Environment Independence
• What if the scripting language changes?
2013 Critical Logic
28
14
17. 9/20/2013
“Test Model”
Model-Centric Example - CEM
1. Multi-Source
Synthesis
2. RMS
Traceability
3. Manual &
Auto
4. TMS
integration
5. MCDC tests
that run
6. Language
independence
2013 Critical Logic
29
Yeah but…
Advantages, Challenges, & Benefits
2013 Critical Logic
30
15
18. 9/20/2013
MBT Advantages
1. Flexibility (env, interface, model basis)
2. Low barrier to entry (80-100 hrs)
[Binder]
3. Measurable, repeatable test coverage
4. Basis for effort estimation
5. Review models, not tests
6. Move validation forward in SDLC
7. Store test definitions as data
8. Integrated test automation
9. Maintain models, not tests
2013 Critical Logic
31
MBT Challenges
•
•
•
•
•
Education of test modelers
MBT strategy is an intellectual task
Expectations: models not “automatic”
Selecting and integrating tool suite
Integration with existing SDLC
•
•
•
2013 Critical Logic
People
Tools
Processes
32
16
19. 9/20/2013
MBT Benefits
1.
2.
3.
4.
5.
6.
7.
8.
9.
Requirements improvement
Efficient test generation, automation
Reduce maintenance costs
Models as basis for effort estimation
Reduce technical debt
Lower cost of quality from rework
Reduce defect escapes
Reduce defect introduction
Data-based result analysis and reporting
2013 Critical Logic
33
Tips to Get Started
• Start small
•
•
•
Pilot low-risk app/change
Rev. eng. an old complex function
Parallel = ideal
• Use a vendor
•
•
Planning, objectives, strategy
Delivery, training & mentoring
• Identify demonstration objectives
•
•
•
•
2013 Critical Logic
Test Coverage improvement
Requirements improvement
Reduced defect introduction/escape rates
Scalability, strategy
34
17
21. 9/20/2013
References & Further Reading
Ambler, Scott W. Agile Modeling: Effective Practices for Extreme Programming and the Unified Process.
2002
Utting, M.; Legeard, B.: "Practical Model-Based Testing - A Tools Approach". Morgan Kaufmann Publ,
Amsterdam, 2007
Web site of Robert V. Binder, software engineer, expert on testing and automation, source of important
research information in this presentation: http://www.robertvbinder.com/
http://www.robertvbinder.com/open-source-tools-for-model-based-testing/
http://www.robertvbinder.com/docs/arts/MBT-User-Survey.pdf
Web site of Stephan Weissleder, MBT theoretician, expert, and innovator: http://model-based-testing.de/
Please feel free to contact me directly if you have any questions or simply to further the discussion. ModelBased Testing is the future!
2013 Critical Logic
37
19
