Contenu connexe Similaire à Test Automation Strategies and Frameworks: What Should Your Team Do? (20) 2. !
!
!
!
!!!!! !
!!!!
Gene Gotimer is a senior architect at Coveros, Inc., a consulting company that uses
agile methods to accelerate the delivery of secure, reliable software. Gene is an
experienced software developer who focuses on continuous integration, static code
analysis, automation, and any tool he can find to do his work for him. For the past few
years, he has been on a team that is bringing a continuous delivery process into the US
Department of Defense, showing that higher quality software can be delivered quicker
and with more security by using agile techniques.
Gene Gotimer
Coveros, Inc.
3. 6/25/15&
1&
© Copyright 2015 Coveros, Inc. All rights reserved.© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Strategies and
Frameworks
What Should Your Team Do?
Max Saperstone
© Copyright 2015 Coveros, Inc. All rights reserved.
Gene Gotimer
Gene Gotimer is a senior architect at Coveros, Inc., a consulting company that
uses agile methods to accelerate the delivery of secure, reliable software. Gene
is an experienced software developer who focuses on continuous integration,
static code analysis, automation, and any tool he can find to do his work for him.
For the past few years, he has been on a team that is bringing a continuous
delivery process into the US Department of Defense, showing that higher quality
software can be delivered quicker and with more security by using agile
techniques.
4. 6/25/15&
2&
© Copyright 2015 Coveros, Inc. All rights reserved.
About Coveros
• Coveros helps organizations accelerate the delivery of secure, reliable
software
• Our consulting services:
– Agile software development
– Application security
– Software quality assurance
– Software process improvement
• Our key markets:
– Financial services
– Healthcare
– Defense
– Critical Infrastructure
Development Capabilities
© Copyright 2015 Coveros, Inc. All rights reserved.© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Planning
5. 6/25/15&
3&
© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Planning
● SMART
○ Specific (What to accomplish, why, who, where – specific reqs)
○ Measurable (How much/many – Quantifiable way to determine
completeness)
○ Achievable (answers question of “How will this be accomplished?”)
○ Relevant (Valued/Worthwhile, consider timing/priorities)
○ Time Bound (target completion)
● Some typical objectives:
○ Testing Consistency/Repeatability
○ Unattended test execution
○ Regression defect identification
○ Frequent Test Execution
○ Software Quality Improvement
Test Automation Objectives
© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Planning
● Usability
● Accuracy
● Integrity
● Interoperability and Compatibility
● Maintainability
● Availability
● Performance
Areas to consider for Test Objectives
6. 6/25/15&
4&
© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Planning
● What is going to be considered in scope for testing?
● What will be out of scope?
● At which level of testing will we increase or decrease our
scope to cover these?
● What is in scope for automation vs
manual testing
Test Automation Scope
© Copyright 2015 Coveros, Inc. All rights reserved.
● Primarily only done for non-
functional requirements,
smoke tests, business
process
● Acceptance test-driven
development Story testing
● Test-driven development
Unit testing
Test Automation Planning
Test Types to Include
7. 6/25/15&
5&
© Copyright 2015 Coveros, Inc. All rights reserved.© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Framework
Selection
© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Framework Selection
● Frameworks provide test infrastructure for automating
various types and levels of tests.
○ Capabilities for common automation requirements
○ Complexity Abstraction
○ Extend Coverage
○ Reduce Maintenance
● Framework Features
○ Test Driver
○ Object Repository Management
○ Data Management
○ Recovery Mechanisms
What is a Framework?
8. 6/25/15&
6&
© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Framework Selection
● Record/Playback (Linear)
● Modular / Structured
● Data Driven
● Keyword Driven
● Action Based
● Hybrids
Automation Framework Types
© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Framework Selection
● There is no ‘Silver Bullet’
● Tool Evaluation
○ Motivation
○ Project Technical constraints
■ OS/Language
■ Development IDE
■ Project Architecture
○ Team skillsets/expertise
○ Budget
○ Best tool fit selection
Test Automation Tool Selection
9. 6/25/15&
7&
© Copyright 2015 Coveros, Inc. All rights reserved.© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Strategy
© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Strategy
Limited Time + Initial Effort required = MoSCoW method
● What MUST be Automated?
● What SHOULD be Automated?
● What COULD be Automated?
● What WON’T be Automated?
Selecting Tests for Automation
10. 6/25/15&
8&
© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Strategy
● Value vs Effort evaluation
● Frequency of Use
○ Repetitive tests that run for multiple builds
○ Tedious Manual Tasks
● Features that historically problematic/prone to defects
● Tests that are highly subject to human error
● Tests that require multiple data sets
● Tests that run on several different HW/SW platforms and
configurations
● Tests that take a lot of effort and time when doing manual
testing
Determining what to Automate
© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Strategy
● Should data start empty, and be cleaned each time?
● What data is needed in QA for automated testing? What
about manual testing?
● How can we keep QA in a state with known test data?
● How do we handle multiple users/automated tests
potentially changing and needed the same data?
● Where is it acceptable to use production data?
● Should production data be sanitized before being used?
● Should production data be analyzed before being used?
Data Management
11. 6/25/15&
9&
© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Strategy
● Frameworks should include take screenshots methods
which can be taken on any failed steps
○ These can also be included for additional information
● Automated frameworks should provide readable reports,
and be used for documentation
● Results from frameworks should be automated to provide
traceability into requirements
○ Multiple tools provide hooks into test management systems
○ Public APIs can allow automated updates for tooling
Traceability through Automated Testing
© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Strategy
● Your automated tests should be treated as code
● When created smartly, steps are simple to find, update, and
reuse
● Create a separate class for each area of functionality
● Instantiate important classes or objects in your base class
Base Functionality
Help FunctionalityLogin Functionality Library Functionality
Best Practices
12. 6/25/15&
10&
© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Strategy
● You can't test all possible combinations of parameters
● Using parameterized inputs to a method allows for a simple
expansion of test suites
○ This allows one test to run with multiple data sets
○ Generally, these can run in parallel
● Multiple existing frameworks support
parameterized testing
● Take your smoke and quicker
executing tests and expand
on them using parameters
Parameterized Testing
© Copyright 2015 Coveros, Inc. All rights reserved.
Test Automation Frameworks
● Abbot
● Anteater
● Canoo WebTest
● Chorus
● Concordion
● Crosscheck
● DejaGnu
● Dogtail
● EMOS
● ItiN
● ItsNat
● JBehave
● Maveryx
● MozUnit
● Phoronix Test Suite
● QAliber
● Rasta
● Robot
● Robotium
● TestNG
● Selendroid
● WAAT
What Testing Framework Might Be Right For Me?
