With the increasing market demand for “always on” high performance applications, many organizations find that their traditional load testing programs have failed to keep pace with expectations and competitive pressures. Agile development practices and DevOps concepts of continuous delivery cause old load testing approaches to become unacceptable bottlenecks in the delivery process. Although it remains true that the only way to know for certain how a system will respond to load is to subject it to load, much of what load testing has traditionally accomplished is rooting out performance issues that are detectable and resolvable without actually applying load. The trick is knowing when and how to look for these issues. With specific examples from recent client implementations, Scott Barber shares the T4APM™ approach, a simple and universal process to detect and manage performance issues—with or without applying load—throughout the lifecycle.

1. T5
Performance Testing
5/8/2014 9:45:00 AM
Next-Generation Performance
Testing with Lifecycle
Monitoring
Presented by:
Scott Barber
SmartBear
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Scott Barber
SmartBear
Chief performance evangelist for SmartBear Scott Barber is a respected leader in the
advancement of software testing practices, an industry activist, and load testing celebrity of
sorts. Scott authored several books―Performance Testing Guidance for Web Applications,
Beautiful Testing, How to Reduce the Cost of Testing, and Web Load Testing for
Dummies―and more than 100 articles and blog posts. Founder/president of PerfTestPlus, Scott
co-founded the WOPR, served as director of the AST and CMG, and is a founding member of
ISST. His industry writing, speaking, and activism focus on improving the effectiveness and
business alignment of software development practices. Learn more about Scott Barber.

3. 4/26/2014
1
Next-Generation Performance Testing
with Lifecycle Monitoring
By:
Scott Barber
Product Owner – Load Testing Tools
SmartBear Software, Inc.
Revised for:
Next-Generation Performance Testing
with Lifecycle Monitoring

4. 4/26/2014
2
Product Owner – Load Testing Tools
SmartBear Software, Inc.
scott.barber@smartbear.com
@sbarber
Co-Founder: Workshop On Performance and Reliability
www.performance-workshop.org
Co-Author:Author: Contributing Author:
Books: www.perftestplus.com/pubs
About me: about.me/scott.barber
System or application characteristics related to:
Speed:
- responsiveness
- user experience
Scalability:
- capacity
- load
- volume
Stability:
- consistency
- reliability
- stress
What is Performance?

5. 4/26/2014
3
What is a Performant System?
A system optimized to balance:
- Cost
- Time to Market
- Capacity
…while remaining focused on:
Quality of Service to System Users
What is Performance Testing?
Testing designed to:
Inform software system optimization by
balancing:
- Cost
- Time to Market
- Capacity
while remaining focused on the quality of
service to system users.

6. 4/26/2014
4
Performance vs. Load Testing?
Conception to Headstone
Not
Cradle to Grave
The Performance LifeCycle is…

7. 4/26/2014
5
© PerfTestPlus, Inc.
Prevent Poor Performance…
… don’t just react when it happens.
Who is Responsible?

8. 4/26/2014
6
Where does it fit in the SDLC?
Delivering Performant Systems;
Commonly, Y2K -> Today
SPE (Software Performance Engineering)
Math-centric Architect/Design method
+ Profiling
Context-free code optimization
+ Load Testing
Demonstrate (too late) poor performance characteristics
+ Capacity Planning
Math-centric guessing to determine hardware needs
+ APM (Application Performance Management)
Proactively react to production issues
=> Inefficient & Reactionary Delivery & Maintenance

9. 4/26/2014
7
Delivering Performant Systems;
Moving Forward
Minimize Need for Complex Mathematics
Few teams have mathematicians w/ advanced degrees
Disconnect Accuracy from Actual Production Data
Because we can’t depend on what we don’t have yet
Remove Socio-Political Barriers
Collaboration improves information sharing & efficiency
Keep/Adapt most Valuable Practices
Many good concepts have been poorly implemented
Distribute Valuable Practices Throughout LifeCycle
Many valuable practices have suffered from stove-piping
We need one model for the whole ‘Performance Journey’
© PerfTestPlus, Inc.

10. 4/26/2014
8
“DevOps” Perf Team Individual
• Code Profiling
• Requirements Revision
• Design Validation
• Budgets
• Units & Components
• Monitoring
• Tuning
• Load/Stress Testing
• Environment Config
• Capacity Planning
• Concurrency Checks
• Situation Analysis
• Snapshots
• Comparisons
• Niche Concerns
• Mini-Field Studies
(UAT)
• “What-if…”
Key Points
• Entire Lifecycle
• Entire Team Focused
• Individual, Role, & Team Accountability & Responsibility
• Continuous & Collaborative
© PerfTestPlus, Inc.
© PerfTestPlus, Inc.
Preventing Poor Performance with
a little work…
every day…
from every one.

11. 4/26/2014
9
© PerfTestPlus, Inc.
Units
Stories
Tiers
Resources
Goals
Assess
Compare
Investigate
Accept
Answer
Dev & Prod
Times
Resources
Sizes
Frequencies
Dashboard!
Proactive
Micro & Macro
Establish Goals
Update Targets
© PerfTestPlus, Inc.

12. 4/26/2014
10
© PerfTestPlus, Inc.
An Ounce of Prevention…

13. 4/26/2014
11
© PerfTestPlus, Inc.
© PerfTestPlus, Inc.

14. 4/26/2014
12
Load Tests
Stress Tests
Endurance Tests
Infrastructure/Configuration Verification
Load Variability Analysis
Background Load for Acceptance Tests…
What is it?
Or any other deliberate approximation of
Production Conditions
© PerfTestPlus, Inc.

15. 4/26/2014
13
RPT is:
Inspired by Rapid Software Testing
Consistent with Rapid Software Testing themes
Sanctioned by James Bach, Michael Bolton & the RST
instructors to as a specific implementation of the
Rapid Testing Methodology
For more information about RST, visit:
http://www.satisfice.com/info_rst.shtml
“What have we got?
What do we want?
How do we get there…?”
--Bob Barber (Scott’s dad)
… as quickly, simply, and
cheaply as possible?
Evolved from:

16. 4/26/2014
14
…a common man’s way of
expressing the problem solving
approach that classical engineers
employ.
• Given: “What have we got?”
• Find: “What do we want?”
• Solve: “How do we get there?”
An approach to respond to a specific
performance-related question after 4 or fewer
hours of team effort with 1 or more of:
A) The answer
B) A partial answer
• To determine the value of additional effort
• The level of effort to provide the answer
C) Better questions to address the underlying
concern
What is it?

17. 4/26/2014
15
1. Receive Question
• Clarify the question
• Understand the driver(s) behind the question
2. Generate Test Coverage Outline (TCO) (~20 min)
• Simplest path to (partial) answer(s)
• Comprehensive path to (partial) answer(s)
3. Transform TCO into Rapid Strategy (~20 min)
• Only tasks that fit in time box
• Stick to tasks requiring available resources
4. Execute Strategy (~2.5 hrs)
• Snapshots are your friends
• Anecdotal is sufficient
5. Consolidate/Analyze Data (~30 min)
• Identify patterns
• Confirm patterns (time permitting)
6. Report Results (~20 min)
• Answer(s) / Time or Effort to get answer(s)
• Follow-on questions of interest
~4 hr
Conceptual Approach
© PerfTestPlus, Inc.
© PerfTestPlus, Inc.

18. 4/26/2014
16
© PerfTestPlus, Inc.
The Bottom Line
© PerfTestPlus, Inc.

19. 4/26/2014
17
Addendum
Questions?

20. 4/26/2014
18
Contact Info
about.me/scott.barber
Product Owner – Load Testing Tools
SmartBear Softwear, Inc.
E-mail:
Scott.barber@smartbear.com
Blog:
scott-barber.blogspot.com
Web Site:
SmartBear.com
Twitter:
@sbarber