The Survey Says: Testers Spend Their Time Doing...

9/11/2014
1
© 2013 IBM Corporation
@Al_Wagner
IBM Rational
Testers spend
their time doing…
The
1 © 2013 IBM Corporation
Session Abstract
How can testers contribute more to the success of their project and their
company? How can they focus on asking the right questions, improving test
planning and design, and finding defects so the business releases a quality
product―even though there’s always one more fire to extinguish or one more
request to fulfill? There aren’t enough hours in the day to do it all.
Join Al Wagner as he reveals recent survey results showing where testers actually
spend their time and where testers think their time would be better spent.
Compare your own experience with what 250 test professionals from around the
world reported. You may be surprised how prevalent testing challenges really are.
Learn what techniques and technologies are available to help today’s test
professionals execute what they were actually hired to do—test software.
Return to your organization with an increased understanding of how other testers
are dealing with their testing bottlenecks and what activities your peers view as
the best use of their valuable time.

9/11/2014
2
Why run a survey?
Aside from talking to directly to customers, we were…
 watching forums and blogs
 reading papers
 following social media channels
 listening to the Analysts
What we found was a small sampling of data harvested from
anecdotes and discussions between small groups.
We wanted to hear directly from the Testers!
We wanted to hear from a lot of Testers!
Agenda – Survey Says
 Setting the stage
 Challenges testers face
 Help is available
 Putting it in context

9/11/2014
3
Who responded?
250 Testing Professionals
from around the world!
Testing Experience
Testing Roles
Quality Managers, Test
Managers, and Test Leads
with more than 6 years
experience!
What and how are they testing?
250 Testing Professionals
dedicated to improving
software quality
Types of applications being tested
Development methodology followed
43% cite Scrum as their main
development methodology.
56% of organizations are
using some variation of
iterative development.

9/11/2014
4
What tools and techniques are being used?
61% of Testers surveyed use
documents & spreadsheets to manage
software quality and test execution.
Is this effective? You decide!
Agenda

9/11/2014
5
What’s The Rush?
Better – Cheaper - Faster
In The Past
• Organizations looked to find a
balance between speed, cost &
risk.
• Development teams were not as
agile as they are today; testing
never seemed to have enough
time.
Today
• Organizations are asking teams to
release higher quality software
sooner.
• Development teams are becoming
more and more agile; testing still
never seems to have enough
time.
Change in Release Frequency
How have Release Cycles changed?
…over last 3 years
32% of Test Professionals surveyed have
seen an increase in release frequency
over the past 3 years.
34% of those who
expressed no change were
already releasing quarterly
or better.

9/11/2014
6
What is the ratio of Testers to Programmers on a
delivery team in your organization?
Testing takes as long as it takes! Really?
Testing will take
longer when you are
not “actually” testing.
How Much of Your 40-Hour Workweek Is
Spent on Unplanned, Non-Testing Activities?
Most Disruptive
non-Testing Activities!
35% of Testers are
spending at least ½ of their
week not testing!

9/11/2014
7
How Confident Are You in Your Product's Quality?
44% of Test Professionals
surveyed have little to no confidence
in the quality of the software
being released.
Where and How Often Do You Find Defects?
40% of Testers shared that defects are
frequently found in production!
Shouldn’t it be a priority to find defects sooner?

9/11/2014
8
What change do you feel will make the biggest
difference in your organization?
Agenda

9/11/2014
9
Testers want to spend
more time on…
1
2
3
4
5
6
7
8
We asked 250 Testers where they would like to
spend less time and where they felt their time
would be better spent?
1
2
3
4
5
6
7
8
Survey Says!
I want to Spend MORE time

9/11/2014
10
Things to remember when automating tests…
Not everything should be automated!
Not everything can be automated!
Test automation is not a silver bullet!
Are you testing what’s under the surface?
Build and Deploy
Software components
built by developers.
Software being
checked by Testers

9/11/2014
11
Test Automation
Are we automating the right tests?
Are we automating the tests rights?
Traditional Test Automation
*
* http://www.agilecoachjournal.com/index.php/2014-01-28/testing-2/the-agile-testing-pyramid/
User Interface
tests
Service
tests
Unit
tests
Prioritizing your test automation effort
Focus on…
Manual /
Exploratory
Testing
pushing
tests
lower
Unit Tests
Tests written before code (TDD);
instant feedback to the developer on
code being written; defects isolated to
specific line(s) of code; run as part of
build automation process
Integration Tests
Tests message exchange, behavior, and
performance between integrated and dependent
components/systems; defects isolated to
specific application component, continuously
executed as part of deployment automation;
lower effort to maintain as application changes
User Interface tests
Checks functionality via the user interface without
understanding the application architecture (black box);
most fragile of automated test types; lacks ability to
isolate defect; most time consuming to create and
maintain; dependent on user interface being available
and stable

9/11/2014
12
Unattended test execution
Teams need to automate tests at every layer
 Separate what the test accomplishes from how it is executed
–Keywords abstract the details of how to execute a logical unit of function
–Business scenarios compose keywords into a flow
 Better leverage the skills in your organization
–Subject matter experts can focus on the business scenarios independent of
implementation
–Test automation engineers can focus on automating
 Reduce test (scenario and automation) maintenance
–Keyword behavior isn’t reproduced in multiple tests; update fewer places
when application changes
–Reduce test cases because overlap becomes apparent when abstraction is
raised to keyword level
Durable, maintainable, resilient to change
Testers want maintainable test automation

9/11/2014
13
 A business process may be available
through multiple channels
– The goal is the same across channels
– The flow is similar across channels
– The UI gestures differ depending on channel
 A business process may span
multiple channels
– Different steps in the flow occur on different
channels
– Consider applications taking advantage of
mobile capabilities such as location and near
field communication
 Multichannel testing is about
effectively creating, managing, and
executing tests of these kinds of
processes
Validating business flows across multiple channels
Effectively creating, managing and executing tests
Keywords make supporting channels easy!
Testers want to spend
LESS time on…
1
2
3
4
5
6
7
We asked 250 Testers where they would like to
spend less time and where they felt their time
would be better spent?

9/11/2014
14
1
2
3
4
5
6
7
8
Survey Says!
I want to Spend LESS time
Standing up Test Environments is a key inhibitor
Applications are getting more complex
• Increasing expense of standing up
complex environments
• Delays caused by lack of system
availability
• Third-party services can be
especially costly
• Testers spend more time setting up
tests than testing
30-50%
The average amount of time testing teams
spend on setting up test environments,
instead of testing
$5-30 million
The typical investment to build a single
test lab for a Fortune 500 company. Most
have dozens …

9/11/2014
15
How do you mitigate delays?
Beware… these approaches are risky and
potentially damaging to the business.
Solving the Test Environment dilemma
Making production-like test environments a reality
Today’s test environments must be:
 Realistic and complete (production-like)
 Inexpensive to stand up
 Available on demand
Agile test environments require four enabling capabilities
 Test Data Management
 Service Virtualization
 Deployment Automation
 Orchestration

9/11/2014
16
Time Spent on Test Data Management
(more than one response allowed)
64% creating test data manually!
34% cloning production and manually cleansing!
Refresh test data
speeding testing and
application delivery
Reduce Risk
Mask sensitive
information for
compliance and
protection
Reduce Cost
Automate creation of
realistic “right sized” test
data to reduce the size
of test environments
• Understand what test data is needed for test cases
• Create “right-sized” test data by subsetting
• Ensure masked data is contextually appropriate to the data it
replaced so as not to impede testing
• Easily refresh & maintain test data by developers and testers
• Automate test result comparisons to identify hidden errors
• Support custom & packaged ERP applications in heterogeneous
environments
Improve application quality and delivery
Agile testing needs continuous access to test data
Speed Delivery

9/11/2014
17
Dealing With Unavailable Software & Services
(more than one response allowed)
Survey says… “Top 2 tactics”:
• Manual creation and maintenance of mocks
• Expensive; lack of scalability; takes time away from testing
• Defer testing until dependent software is available
• Adds risk; decreases test coverage; potential project slip
What is Service Virtualization?
Agile teams need to make the unavailable available for testing
 Virtual components simulate the behavior
of a service or application during testing
– System dependencies are a key challenge in
setting up test environments
– Unavailable/inaccessible services slow down
the testing process
– Costly 3rd party access fees can significantly
impact testing availability
 Virtual components run on commodity
hardware, private cloud, public cloud
 Each developer and tester can easily have
their own test environment
 Developers and testers can continue to use
current testing procedures and tools
 Streamline creation of production-like
environments and negative testing
Heterogeneous Environments
Public Cloud
Private Cloud
Data Warehouse Mainframe
Enterprise
Service Bus
Directory
Identity
File
systems
Collaboration
App Under TestRouting
Service
Third-party
Services Portals
Content
Providers
EJB
Shared
Services
Archives
Business
Partners
Messaging
Services
Databases Mainframe
applications
App Under Test
Third-party
Services
Packaged apps, messaging services, etc.
Virtual Components

9/11/2014
18
Deployment Automation
Agile teams need to accelerate application delivery across the lifecycle
 Automated, consistent deployments and rollbacks of applications
 Orchestration of changes across servers, tiers and components
 Configuration and security differences across environments
 Clear visibility: what is deployed where and who changed what
 Integrated with middleware, provisioning and service virtualization
Key Features:
• Multi-Tier Application Models
• Easy Process Designer
• Track what is where:
Inventory
• Scalable Distributed
Automation
• Quality Gates and Approvals
• Integrations Replace Custom
Scripting
Deployment Automation
Agenda

9/11/2014
19
Dev-Test-Ops (aka. DevOps)
Enterprise capability for continuous software delivery that enables organizations to seize
market opportunities and reduce time-to-customer feedback
Accelerate
software delivery –
for faster time to value
Balance speed, cost,
quality and risk –
for increased capacity
to innovate
Reduce time to
customer feedback –
for improved customer
experience
Continuous
Customer
Feedback &
Optimization
Collaborative
Development
Continuous Release
and Deployment
Continuous
Monitoring
Continuous
Business Planning
Continuous
Testing
Operate Develop/
Test
Deploy
Steer
DevOps
Continuous
Feedback
All the world’s a “test” stage
and all Testers merely players; They have their exits and their entrances;
The Continuous Delivery Pipeline
Development Testing Staging Production
Business
Owners Customers
From ideation to implementation…
• everyone contributes to quality!
• everyone contributes to a positive end user experience!
• everyone contributes to the success of the business!

9/11/2014
20
Continuous Testing:
• helps extend agile and lean practices across
the delivery lifecycle
• removes traditional testing bottlenecks such as
unavailable test environments to increase
efficiency
• enables teams to test earlier and with greater
coverage at lower cost
• provides project teams with continuous
feedback on software quality to reduce
business risk
A comprehensive Continuous Testing solution
covers quality management, functional UI and
integration test automation, performance testing,
and service virtualization.
Organizations need Continuous Testing
Accelerating the delivery of higher quality software
Resolve today’s testing
challenges with confidence
Orchestration and automation for the deployment of applications,
middleware configuration, and database changes into development,
test and production environments.
Databases Internal
Messages
Third-party
Services
virtual components
Simultaneously
test across
multiple test
stages
Dev QA
Service
Virtualization
Combining continuous testing withContinuous Testing enables Continuous Delivery
Repeatable, & traceable test and deployment automation
Test
Automation
Test Environments
Dynamic Infrastructure
 Virtualize unavailable
yet dependent software
& systems
 Continuously test in
production-like env.
Quality
Management
 Manage quality across
the lifecycle
Deployment
Automation
 Deploy private, secure
right sized test data
Test Data
Management
 Deploy just built
applications
Continuous
Delivery
Orchestrate the
delivery pipeline
integrated with

9/11/2014
21
Top 6 take aways
1
2
3
4
5
Not every test can be automated.
Not every test should be automated.
At least 1/3 of the delivery lifecycle
is testing. Testers do matter!
Test what matters earlier.
Test it continuously!
Deploy what you have.
Virtualize the rest!
6
Everyone contributes to quality!
You can’t test quality in.
Continuous Testing enables
Continuous Delivery
The
Download your copy today!
ibm.biz/TestingSurveyResults
How do you stack up to what
the others are saying?

9/11/2014
22
42
ibm.biz/continuoustesting
43

The Survey Says: Testers Spend Their Time Doing...

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (17)

Similar to The Survey Says: Testers Spend Their Time Doing...

Similar to The Survey Says: Testers Spend Their Time Doing... (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

The Survey Says: Testers Spend Their Time Doing...