1. Cloud Testing Bill of Rights
Top 10 Decision Criteria for a Web &
Mobile Application Testing Platform
By Scott Barber, Chief Technologist, President & CEO, PerfTestPlus, Inc.
Why a Cloud Testing Bill of Rights?
My collaboration with When the governmental structure of the United States was initially put into place,
SOASTA began in the sum-­ it did not include the Bill of Rights. While many of the Founding Fathers were far
mer of 2011 when I was from pleased about the absence of explicit citizens’ rights in the Constitution, they
asked to preview a beta ver-­ knew that if they didn’t get the basic governmental structure in place quickly, the
sion of CloudTest Lite. The new country wouldn’t last long enough for citizens’ rights to matter. This sounds
positive experience I’ve had very much like how development and IT groups were formed, with quick and dirty
with both the product and organizational and governmental structures that made sense to someone in the
the SOASTA team has en-­ moment.
abled our work together to
continue. This begs the question: In your development or IT group, do you feel like you have
the right to voice dissatisfaction and organize to champion change? If your experi-­
If you are familiar with my
work then you know that I what change we are championing.”
don’t commonly write about
vendors or tools aside from I can think of no techniques, ideas or practices related to testing, which require sup-­
deliberately neutral educa-­ port from management, that get shut down more quickly than those that include the
tional pieces. cloud. The cloud is gaining a degree of credibility when it comes to operations, but
when it comes to testing, there are a disproportional number of vociferous detrac-­
SOASTA has proven to be tors as compared to champions. I have to wonder if the detractors are being driven
one of those rare vendors by fear: fear of change and fear of being rendered irrelevant by advancement.
speak freely about without This observation leads me to see the Cloud Testing Bill of Rights as a series of
violating what mom taught proposed revisions to commonly accepted software development practices. They
me about refraining from ensure citizens of the nation of software development the freedom of expanded
saying anything at all if you options and the right to pursue betterment of their current situation. These revi-­
don’t have anything nice to
say. simplify application performance testing.
Your Cloud Testing Rights
I’m hard pressed to imagine many individuals who would object to having the right
to pursue these items appropriate to the needs and context of their project, applica-­
tion or organization without fear of being labeled a troublemaker.
© Soasta, Inc. 2012 1

2. II. Provides an elastic, globally distributed test environment on demand and
within minutes
III. Has dynamic and fully automated test grid provisioning
IV. Has the ability to build and modify complex tests within hours
V. Supports dynamic web and mobile technologies and protocols
VI. Controls test sessions to start, pause, stop, modify, re-­start and analyze
VII. Has a single integrated view of performance and monitoring metrics
VIII. Rapidly isolates performance issues in real time at any scale
IX. Allows functional and load testing from lab to production
X. Is affordable enough for everyday testing
In case these rights don’t strike you as important or it’s not immediately obvious to
you what problems they address, read on.
Load Generation
A cornerstone of a business creating value by adding application performance test-­
ing is the ability to exercise the application under test with realistic loads. “Realistic”
doesn’t only refer to the ability to deliver the correct volume, but also to deliver
that volume from the places where actual users will be using your application. The
before deciding which tool to use for your application performance testing needs.
-­
ternally and from an end-­user’s perspective. Each of these perspectives add value
that the other two don’t, meaning that failing to consider and account for each of
these will ultimately lead to unwanted surprises.
Internal load generation is the most effective option for early performance assess-­
ments and for continually assessing and tuning hardware and software that will live
in your hosted environment. External load generation is necessary to determine
of your site or application and geographic proximity to the data center. Taking mea-­
surements from a true end-­user perspective, or “Last Mile” testing, is necessary
to assess and tune the front-­end aspects of the site or application and determine
actual user experience.
When it comes to load generation, in almost all situations you’ll want the ability to
generate load both internally and externally. If you’re part of an organization that
hosts every component internally and access to those sites or applications will be
available only to folks on the same internal network, then there is no need for ex-­
ternal testing.
© Soasta, Inc. 2012 2

3. There is rarely a need to generate load from actual client machines. Generating exter-­
nal load from the cloud while observing a sampling of end-­user interactions is entirely
-­
ready part of your overall testing program.
II. Elastic, globally distributed test environment
-­
ters is that you have the right to be able to generate realistic, production volume loads
from geographies representative of your actual user base. While generating load ex-­
ternally from across the street is better than only being able to generate internal load,
it’s not nearly as valuable as concurrently generating load from a variety of geographic
locations.
It’s far more challenging today to predict performance from a particular geography than
-­
able to the producers of sites and applications to improve performance in a particular
geography. Implementing a cloud-­based technology for load generation means that
you no longer have to wait for users to complain before becoming aware of a problem.
Cloud-­based load generation can enable you to assess performance by user location
by clicking a couple of extra buttons and running a couple of extra of tests.
III. Dynamic and fully automated test grid provisioning on demand
and within minutes
Very much related to the previous two rights, you have the right to mix and match the
generation points for your load quickly and simply. Having the ability to generate load
from anywhere isn’t enough, and isn’t unique to cloud-­based solutions. I, for one, was
doing that ten years ago. Of course, it always seemed to involve a month to ship hard-­
ware with the right software pre-­installed to remote locations, or to talk people working
of, coordinate network connectivity and the other hundred details. And all this before
-­
gested such a solution today!
Speed and Completeness
enough to be relevant without needing to learn a new tool each time a new technology
is implemented. Performance engineers have long been begging for rapid testing tools
that can be used with a wider range of technologies.
Today, traditional testing tools continue to take weeks and months and only test older
technologies. Two rights speak to today’s performance testers not being condemned
to telling those same war stories ten years from now.
© Soasta, Inc. 2012 3

4. Clearly, this right is not exclusive to cloud-­based performance testing technologies.
Rather, you have the right to demand this from your tool or service provider no matter
don’t have days, let alone weeks, to wait for performance assessments. They have
the right to results in enough time to do something more useful than shrug, mumble
something about wishing they’d known sooner and logging an action item for the next
release. The only hope we have of providing results on their timeline is if we can build
and modify realistic tests, complex tests, in hours or less.
V. Support for dynamic web and mobile technologies and
protocols
One could say that we have the right to demand that our tools provide support for new
technologies and protocols. At the time of this writing, the most challenging of those
from a load generation perspective is related to asynchronous yet inter-­related use of
both client-­ and server-­side communications within a single site or application. Next
year, it may be something entirely different. What will remain the same, however, will
be our desire to not purchase and learn a new tool or service due to a technological
advance.
Of course there will always be bleeding-­edge technologies that tools don’t support.
And when one of those technologies catches on, vendors need not charge us obscene
amounts of money for updates. The fact that some vendors still expect their loyal cus-­
tomers to pay exorbitant fees to AJAX or SOAP support, protocols that crossed from
have the right to be treated better than that.
Control
Following the ability to create and implement tests quickly, and building tests against
leading technologies without purchasing new tools, tool extensions are the ability to
performance test effort.
VI. Control test sessions to start, pause, stop, modify, re-­start and
analyze
There was a time when it was a rare tool that allowed expert performance testers to
write custom code to give them the kind of detailed control they needed to maximize
a tool so they can accomplish tasks like changing load levels on demand during test
runs. There are plenty of tools on the market that come with this kind of control built in.
This speaks to performance testers having the right to spend the majority of their time
doing performance testing instead of working around tool limitations. Any organization
disabling their performance testing program by forcing their testers to work around
their tools today is spending far more money in terms of wasted time, unfound perfor-­
© Soasta, Inc. 2012 4

5. mance issues and under-­tuned sites and applications than it would cost to switch to a
tool or service that just works. We most certainly deserve the right to ask our compa-­
nies to stop wasting money.
Analytics
When I was starting out as a performance tester, virtually all performance analysis was
into the same spreadsheet of data that was manually exported from the load genera-­
tion tool to manually create charts and graphs so the team could manually analyze the
modify the next test.
VII. Single integrated view of performance and monitoring metrics
Year after year my pleas for this capability fell on deaf ears. Today, most load gen-­
eration tools come with built-­in monitoring. Many even come with real-­time, integrat-­
ed monitoring on a single, customizable dashboard. To me, this is nothing short of a
dream come true;; for any performance tester this is at least a right, if not an expecta-­
tion. Aside from saving an unbelievable amount of time, the increase in value per test
rights, it’s the right to have monitoring software that works as part of your performance
testing program that matters.
VIII. Rapidly isolate performance issues in real time at any scale
-­
ect. Today, these tools are commonplace, affordable, approachable, and even easy to
-­
ties. The ability to isolate performance issues in real time not only saves time, as the
right’s title implies, it all but eliminates the “it’s your tool, not my application” response,
enabling the team to get right to work on resolving issues instead of wasting time –
The overwhelming majority of my clients’ performance testing efforts are completely
hamstrung by tools, services and license agreements that make it impossible to do
continual performance testing. This situation leads to the propagation of the belief that
performance testing can be effective as a last-­minute task. The reality is that treating
performance testing as an isolated afterthought leads to unnecessary heroic efforts
and frequently to epic failures in production. This is a sad and decidedly unhelpful state
of affairs.
© Soasta, Inc. 2012 5

6. IX. Test platform that allows functional and load testing from lab to
production
Good performance testing begins when a project is conceived and ends when a site
or application is retired. Sites and applications that achieve consistent acceptable (or
better) performance integrate performance testing into every aspect of design, devel-­
opment, deployment and maintenance of their product. It used to be that doing this
took a large and expensive toolbox as well as a small army of tool specialists. That is
no longer the case.
With modern cloud-­based performance testing tools, activities that used to be excep-­
tionally painful, like moving a test from the lab environment to a staging or production
environment, take minutes instead of days or weeks. Combine this with capabilities of
individual tools to perform multiple functions means that fully and seamlessly integrat-­
ing performance testing is entirely viable. Once again, we have the right to demand this
kind of simple and seamless integration of performance testing into the entire product
lifecycle and the right to demand that vendors support our efforts in this regard.
X. Affordable enough for everyday testing
This right simply cannot be overstated. For the majority of my career as a performance
tester, only the richest and most elite software development organizations were able
to justify the expense of buying or leasing the software and infrastructure necessary to
run tests at will. Shared licenses, temporary environment provisioning, and administra-­
tive overhead has been the bane of a performance testers existence for years. Any
organization that is still facing this situation is doing so out of negligence.
Advances in cloud-­based technologies and pricing models have all but eliminated the
need for additional hardware and infrastructure to be procured and maintained in-­
ternally. This has reduced the total cost of enabling on-­demand, daily performance
testing, conducted by any member of any team, without the need to cross-­coordinate
hardware or license availability. Add that to the savings that an organization will realize
by deploying and maintaining well-­performing applications, and it seems nothing short
of irresponsible to continue paying so much more for so much less value.
Conclusion
Cloud-­based performance testing technologies have converted the dreams of many
old-­school performance testers into the rights of today’s performance testers. Failure
to take advantage of these rights wastes time and money. It ensures acceptable site
expensive, or result from sheer dumb luck. In short, not knowing and taking advan-­
tage of your rights as described by the Cloud Testing Bill of Rights is nothing short of
irresponsible.
About the Author:
A prominent throught-­leader in the area of software system performance and testing software systems, Scott
Barber, founder and Chief Technologist of PerfTestPlus & Director of the Computer Measurement Group,
makes his living writing, speaking, consulting, and coaching with the goal of advancing the understanding
and practice of software testing. Scott has contributed to four books, composed over 100 articles and papers,
the Association for Software Testing, and co-­founded the Workshop on Performance and Reliability.
© Soasta, Inc. 2012 6