Two key challenges to continuous load testing are provisioning a test system to handle the load and accessing load generators to drive the traffic. In this webinar from SOASTA & CloudBees, you will learn how to: Build realistic automated web performance tests and run them in Jenkins Architect and launch a test environment that auto-provisions in the cloud Manage a load generation grid to drive load tests in a lights-out mode Establish a performance baseline in your daily Jenkins reports

1. and
Present
Automating
the hard
parts, made
easier
Webinar
Copyright © 2014 SOASTA, Inc. - All rights reserved.

2. Continuous Load Testing
TODAY’S PRESENTERS
Mark Prichard: CloudBees, Dir. Product Management
Mike Ostenberg: SOASTA, Dir. Systems Engineering
Brad Johnson: SOASTA, VP Product
TWO COMPANIES, A SINGLE SOLUTION
- the leader in Mobile and Cloud Testing
- the home of Jenkins in the Cloud
Together, we offer a complete cloud-based service:
Continuous Delivery for Web & Mobile
2
Copyright © 2014 SOASTA, Inc. - All rights reserved.

3. 3
Copyright © 2014 SOASTA, Inc. - All rights reserved.

4. •
•
•
•
Why should we load test earlier?
Building tests and preparing the test environment
Connecting your load tests to your CI (Jenkins)
Establishing and managing to a performance
baseline
• Q&A
Questions:
Submit in the question box during event
4
Copyright © 2014 SOASTA, Inc. - All rights reserved.

5. 5
Copyright © 2014 SOASTA, Inc. - All rights reserved.

6. 6
Copyright © 2014 SOASTA, Inc. - All rights reserved.

7. 7
Copyright © 2014 SOASTA, Inc. - All rights reserved.

8. “Because it’s hard.”
oTools require coding skills we don’t have (C, VB?)
oWe don’t have a scalable test environment
oWe don’t have hardware for load generation
oIt’s a different tool
oThere are too many other tests to do
8
Copyright © 2014 SOASTA, Inc. - All rights reserved.

9. “Because it’s hard.”
oTools require coding skills we don’t have (C, VB?)
oWe don’t have a scalable test environment
oWe don’t have hardware for load generation
oIt’s a different tool
oThere are too many other tests to do
“Because it takes too
long.”
oScripting takes forever
oResults are slow
oFast setup of environments is impossible
oWe’re dependent on IT or other teams
oWe’re too agile
9
Copyright © 2014 SOASTA, Inc. - All rights reserved.

10. “Because it’s hard.”
oTools require coding skills we don’t have (C, VB?)
oWe don’t have a scalable test environment
oWe don’t have hardware for load generation
oIt’s a different tool
oThere are too many other tests to do
“Because it takes too
long.”
oScripting takes forever
oResults are slow
oFast setup of environments is impossible
oWe’re dependent on IT or other teams
oWe’re too agile
“Because no one ever told us
to.”
oThere are no “performance coverage” requirements
oThere isn’t a performance baseline for reference
oThere isn’t a defined performance target
oNo one is managing to a performance metric
10
Copyright © 2014 SOASTA, Inc. - All rights reserved.

11. “Because it’s hard.”
oTools require coding skills we don’t have (C, VB?)
oWe don’t have a scalable test environment
oWe don’t have hardware for load generation
oIt’s a different tool
oThere are too many other tests to do
“Because it takes too
long.”
oScripting takes forever
oResults are slow
oFast setup of environments is impossible
oWe’re dependent on IT or other teams
oWe’re too agile
“Because no one ever told us
to.”
oThere are no “performance coverage” requirements
oThere isn’t a performance baseline for reference
oThere isn’t a defined performance target
oNo one is managing to a performance metric
Until Now.
11
Copyright © 2014 SOASTA, Inc. - All rights reserved.

12. 12
Copyright © 2014 SOASTA, Inc. - All rights reserved.

13. Iteration 1
Iteration 2
Automated unit tests
Automated unit &
functional tests
Iteration 3
Iteration n
Regression test suite Regression test suite
Load &
Performan
ce Testing
Deploy
Issues
?
The Issue:
•
High expectations from tests run with diminishing time
windows
•
Code-level performance bugs found late set entire project
back
•
Separate team and schedule conflict with delivery goals
•
Major issues escape to production
13
Copyright © 2014 SOASTA, Inc. - All rights reserved.

14. Iteration 1
Automated unit tests
Small perf test
Iteration 2
Automated unit &
functional tests
Small perf tests
Iteration 3
Regression test suite
Incl:
Iteration n
Regression test suite
Incl:
Perf smoke
tests
Perf smoke
tests
Deploy
End to End
Production Scale Test
Production perf tests
DevOps
Development/Test team
Performance/Ops team
The Solution:
•
Run incremental performance tests automatically with each iteration
•
Establish a trend line for performance early
•
Developers find code and architecture issues early
•
Performance teams focuse on end-to-end infrastructure and user
related testing
•
Minimize production escapes and implement production testing
14
Copyright © 2014 SOASTA, Inc. - All rights reserved.

15. Manage with a new metric!
I JUS
COD T MADE
E CH
A
AN G
E
Jenkins launches
suitable cloudbased test
environment
Identify
bottlenecks
& inform
teams, just
like unit and
functional
tests in
Jenkins
Daily (or more frequent) updates
keep teams “performance focused”
CloudTest runs
load tests via
Jenkins
15
Copyright © 2014 SOASTA, Inc. - All rights reserved.

16. 16
Copyright © 2014 SOASTA, Inc. - All rights reserved.

17. Mark
Prichard
@mqprichard
17
Copyright © 2014 SOASTA, Inc. - All rights reserved.

18. Have you met Jenkins?
o #1 OSS CI server
o Written in Java
o Highly Extensible
o About 8 years old
o Easy to install/use
o Extensible via 1000+ plugins
o Widely adopted
o 70K+ installations
o CloudBees knows Jenkins
o On-premise, cloud and hybrid
o Expert support
o Enterprise plugins

19. Continuous Integration and Automated
Testing
o Let developers focus on developing
o Leave building, packaging, testing, distributing to a computer
o Make the right thing the easiest thing – don’t think about it
o Many, widely-adopted plugins
o Safety in numbers – hugely active community
o Dashboard view of the state of your builds
o You don’t have to be an expert at everything
o Test Anywhere
o Deploy on-premise or in the cloud for testing
o Spin up VMs or EC2 Instances as build executors
o We can do it for you, if you like…

20. Setting up the Test Envrionment
• CloudBees ClickStarts – A complete project in just two clicks!
• Bees Shop Demo
• Git repo, database, build and deploy in seconds
• Online examples: https://partnerdemo.ci.cloudbees.com

21. o Demo:
• Bees Shop Overview
• SOASTA Cloud Test: Setting up a load test up for unattended runs
• Jenkins Performance Data:
• Reviewing Performance Trending Data: Track performance of each build
• Reviewing Performance Pass/Fail Data: Failing
• Setting up Continuous Load Testing
• Install Plot Plugin
• Download GitHub: https://github.com/mostenberg/SOASTAPERFPERL
• Add steps to plot build data for any metrics of interest.
• Publish Junit test results to PASS/FAIL based on performance thresholds:
SOASTA/5_PERF_THRESHOLD_RESULTS.xml
21
Copyright © 2014 SOASTA, Inc. - All rights reserved.

22. Mike Ostenberg
22
Copyright © 2014 SOASTA, Inc. - All rights reserved.

23. TouchTest
CloudTest
Continuous
Testing
Load &
Performance
mPulse
Real User
Monitoring
23
Copyright © 2014 SOASTA, Inc. - All rights reserved.

24. SOASTA Jenkins Plug-in
• Open Source: online at jenkins-ci.org
• Automatically available on CloudBees DEV@cloud
• Complete control over CloudTest/TouchTest Automation from within your
Jenkins builds:
• Totally integrated into Jenkins environment

25. Try it out yourself!
• CloudBees ClickStarts – A complete project in just two clicks!
• Example configurations all online:
• https://partnerdemo.ci.cloudbees.com/
• Bees-Shop-PerformanceTest

26. Thanks & Q&A
RESOURCES
www.SOASTA.com
Knowledge Center
•White Papers
•Webinar
Recordings
•Case Studies
CloudLink
Community
•Support
•Tutorials
•Video
Contact SOASTA:
www.soasta.com/cloudtest/
info@soasta.com
866.344.8766
Follow us:
www.CloudBees.com
blog.cloudbees.com
•Look for Mark’s post on this
topic
developer.cloudbees.com
•Platform Services
•Frameworks
•ClickStarts
Contact CloudBees
www.cloudbees.com
info@cloudbees.com
1.781.404.5100
Follow us:
twitter.com/cloudtest
twitter.com/cloudbees
facebook.com/cloudtest
facebook.com/CloudBees
26
Copyright © 2014 SOASTA, Inc. - All rights reserved.