1. Confidential
Production Environment Data to
Improve Testing
Test based on established facts instead of assumptions
Rev PA1 2011-10-26 1

2. Introduction
 According to many voices the future of testing is to move test
into a production environment [1][2][3]
 Defects are no longer cheaper to find early in development, but
just as cheap to fix in production, according to some [1]
 Testing in production finds problems at a scale that most
groups can’t afford to duplicate with a test environment [2]
 Without being overly dramatic, one can assume that data from
a production environment will be an important input to testing
in the future

3. Rapid Deployment & Product Environment
Feedback
 Deployment of software has become more and more rapid[4]
 Companies move towards more and more agile development
[5]
 Multiple releases of software ensures that customer feedback
for one release can be used to influence testing of future
releases
 A framework for gathering this information is needed, and it is
important to understand what product environment feedback
is needed to improve testing

4. Production Environment Information
 If a framework is in place to collect data from a production
environment, the important question is what information is
actually needed to improve testing
 This information varies dependant on the software, and it is a
question everyone who is releasing software will have to ask
themselves in the future
 What information is important to extract from the production
environment?

5. Production Environment Information
Example
Resource
usage
What different
variants and
Software
configurations
degrade over
of the
time
software are
used
Production
Environment
Information
Stability in Performance
production in production
environment environment
Most used
applications
and services

6. How to use the information
 The information gained can be used in a multitude of ways
 If you know what applications and services the customer uses
most, you can focus your testing to those areas
 If you find degrades over time, you know to focus on that area
 Performance and stability issues in specific areas can give input to
additional testing in those areas
 If you know which variants and configurations of the software is
used most, then you can focus on using those variants during
testing
 The actions taken depends on the information, which as we said
previously in turn depends on the software

7. Conclusion
 Of course having a good production environment feedback
structure in place will replace some testing, but the real gain
will be in improving the testing that is still being executed
 It is important to identify the information needed from the
production environment early, even before the feedback
structure is in place – this will give important insight into what
gain the production environment information will yield to your
test execution
 Getting this product environment feedback structure in place
will be key for almost all software development in the future

8. Reference
[1] The Liberation of Testing
http://marlenacompton.com/?p=2514
[2] Testing services in production
http://www.pnsqc.org/2011-conference/technical-paper-abstracts-and-bios-2#T-11
[3] Test is Dead
http://googletesting.blogspot.com/2011/06/test-is-dead.html
[4] Software G Forces: The Effects of Acceleration
http://www.youtube.com/watch?v=KIkUWG5ACFY
http://www.slideshare.net/KentBeck/software-g-forces
[5] Has Agile Peaked?
http://drdobbs.com/architecture-and-design/207600615