Agile Testing - Challenges

1. Challenges in Agile Testing

2. List of Challenges
 Volume and Speed of requirement change
 Over-commitment and Sprint durations
 Accidental Broken Code due to frequent builds
 Inconsistent and Inadequate Unit Testing
 Inadequate Test Coverage
 Inadequate Testing for published API
 Poor, Changing and Missing Test Oracles
 Detecting defects early, When they’re easier and cheaper to fix
Challenges in Agile Testing……

3.  Volume and Speed of Requirement
Challenges in Agile Testing……
Caused by:
Changing requirements, even late in development
Changes in definition of product and its correct behavior
Resolved by:
Risk-based Testing
Automated Testing
Lightweight reactive Testing
Keep test team informed of such changes
Account for differential impact of changes on testing

4.  Over-commitment and Sprint durations
Challenges in Agile Testing……
Caused by:
Some projects ritualize the rules, particularly sprint time deadlines
Ex: if a project follows four week sprints and continually over-commits, the test team gets squeezed
on the last weekend of every sprint
Resolved by:
Requires team and management maturity
Risk-based testing helps the test team deal with over-commitment
 Reduce the scope of testing based on risk
 Slip low-risk tests into next sprint

5.  Inconsistent and Inadequate Unit Testing
Challenges in Agile Testing……
Caused by:
Unit testing has limited bug-finding effectiveness, averaging 25 to 30%, while good system testing averages
around 85%
Not all programmers do unit testing
Resolved by:
Good Unit Testing is the only solution

6.  Broken code due to frequent builds
Caused by:
Daily build introduces the rick of breaking code/ existing functionality
Resolved by:
Perform continuous integration – automated process for creating software build
Employ an automation tool that performs regression testing whenever you launch a new build
Employ Test-Driven development [TDD]
Ensure that Automated Test team is working optimally
 Review Automated Test metrics
 Use Source Control for reviewing changes to automated tests and protection
 Continue to build on your Automated Tests
Track metrics are most important for successful automated testing:
Metrics for successful automated testing:
Feature Coverage
Requirement/Feature blocked
Daily Test Run trending
Daily Test Runs by Host
Challenges in Agile Testing……

7.  Inadequate Test Coverage
Caused by:
Few test cases written for User story
Lack of visibility of code check-ins for regular corrections to code
Resolved by:
Creating test cases at the level of User-story
Design sufficient number of tests:
 Positive Tests/ Negative Tests/ Performance Tests/ Ancillary Tests
Branch code using SCM; any code outside the target always remains untouched
Adding triggers to SCM to inform team of code changes
Integration with source check-in to find changed code that was not anticipated or planned for
Analyzing specific metrics to identify traceability and missing test coverage
Metrics for traceability and test coverage:
Traceability Coverage
Blocked Tests
Test Runs by Requirement [User Story] & Configuration
Daily Test Run Trending
Defects by Requirement/ Daily Defect Trending/ Defect Duration
Challenges in Agile Testing……

8.  Inadequate Testing for published API
Caused by:
API testing often is omitted because of the misperception that it takes programming skills to call the properties
and methods of your API.
Resolved by:
Dedicated resource
High Use functions
Usability Tests
Security Tests
Stopwatch-level performance Tests
API metrics to be watched:
Function Coverage
Blocked Tests
Coverage within Function
Daily API Test Run Trending
Ensure API testing team is working optimally:
Review API Run Metrics
Continue to build on your API Tests
Challenges in Agile Testing……

9.  Poor, changing and missing Test Oracles
Challenges in Agile Testing……
Caused by:
Agile de-values documentation
Change the definition of correct behavior after test execution has started
Change the definition of correct behavior without involving the test team
Resolved by:
Embracing change
Conveying information [in] face-to-face conversation
Change Management needs to be in-place

10.  Detecting defects early
Challenges in Agile Testing……
Problems from late-detection of defects:
 More time to fix
 More money to fix.
Resolved by:
Performing peer reviews of source code and test artifacts to find early stage defects
 Peer Code Review Metrics:
 Lines of Code Inspected
 Requirement Review Coverage
 Manual Test Review Coverage
 Automated Test Review Coverage
 Code Review Coverage
 Static Analysis Issues found
 Defects discovered by Peer review
Using static analysis tools to identify early stage defects
Analyzing peer review statistics and defect aging to address defects early when they’re least costly to fix
 Perform Peer reviews
 Automatically run static analysis
 Review Defect metrics

11. Challenges in Agile Testing……
Testers and Test managers must help their organizations understand and manage
the challenges to manage expectations