Presentation on Pacific Northwest Drupal Summit, Vancouver 2013 about load testing Drupal sites using Apache JMeter and Amazon AWS.

1. Load Test Drupal Site
Using Apache JMeter & Amazon AWS
By Vladimir Ilic
@burgerboydaddy

2. About me
• twitter: @burgerboydaddy
• http://burgerboydaddy.com

3. Quote
• “The designer is concerned with what happens
when 1 user presses a button and the architect
is concerned with what happens when 10,000
users press a button.”
• Sun Certified Enterprise Architect for J2EE Technology Study Guide. Page 6. Mark Cade, Simon Roberts.
2007 JavaOneSM Conference | Session TS-9235 | 2

4. 4
Test Types
• Performance
• Load
• Stress
• Scalability
• Endurance
• Functional

5. Performance Testing
• Performance testing determines or validates the
speed of the application
(X per T).
• Used for finding bottlenecks and establish
baseline for the system.
• In other words it’s solo purpose is to determine
the response and effectiveness of a system.

6. Load Testing
• Load testing identifies the maximum operating
capacity of an application as well as any
bottlenecks that might interfere with its
operating capacity.
(or, when does it blow up?)

7. Stress Testing
• Stress testing is focused on determining an
application’s robustness, availability, and
reliability under extreme conditions
– Heavy loads
– High concurrency
– Limited computational resources
• An attempt to break the system by
overwhelming its resources

8. Scalability
• Scalability testing determines or validates
whether adding another Y resource (database,
memory, disk, CPU, etc) increases speed of X
proportionally
Endurance Testing
This type of testing is used to check that the system can withstand
the load for a long or large number of transactions.

9. Preparation for a Test
• Mission
• Network
• Hardware
• Software
• Metrics

10. Mission
• What is the testing intended to achieve?
• What are basic assumptions like:
– What is our anticipated average number of users
(normal load)?
– What is our anticipated peak number of users?
• When is a good time to load-test our
application (i.e. off-hours or weekends),
bearing in mind that this may very well crash
one or more of our servers?

11. Environment Prep - Network
• Performance testing is usually network
intensive operation and can affect others in the
organization
• Testing should be done on a separated /
segregated network
• Amazon AWS – virtually unlimited
in/out speed

12. Environment Prep – Hardware
• Is your machine ready to receive full load?
• Are multiple machine available (for distributed
testing)?
• Do you have enough resources
• Again – Amazon to the rescue – AWS gives you
as many test machines as you need.

13. Metrics
• Performance testing is all about numbers and
metrics
• Determine which metrics you are concerned about
and how to get them.
• Some simple tests / benchmarks can be done using
Apache Bench “ab” command:
•
Suppose we want to see how fast our site can handle 100 requests, with a
maximum of 10 requests running concurrently:
• ab -n 100 -c 10 http://example.dev/

14. JMeter – Software of Choice
• Open source desktop / server application
• Designed for functional/load/performance/stress testing
• Extensible… write your own test
• Simulate heavy load (application, server and network)
• Gives instant visual feedback
• Distributed testing
• Various protocols - HTTP, FTP, JDBC, JMS, LDAP, SOAP
• Multi-platform
• Full multithreading framework
• Caching and offline analysis/replaying of test results.
• JMeter is not a web browser!

15. JMeter vs. Real World
Real World JMeter World
One User Browser request HTTP Request Sampler
One HTML page display with JavaScript
Execution
View Tree Listener with basic HTML
display.
No JavaScript execution
Multiple Users Requesting Pages
Simultaneously
Thread Group Configured for Number of Users,
uses the same HTTP Request Sampler to
simulate multiple users
No Equivalent
Difficult to do
Measuring performance like min, max, and
average (avg) time for processing using
Summary Report Listener

16. JMeter - Terminology
JMeter Term Meaning
Test Plan - You keep your test plan under me
- Only one per one JMeter window
- Save it for a future use
Thread Group - Represent one set of action – one scenario
- You add actions that single user will do
- JMeter will use them to simulate multiple users
HTTP Request Sampler - Record the request to web server
- Also receive the response from Web server
- Provide all data received for analysis
View Tree Listener - Show the test data in details for each item
- For HTTP Request it show’s Request, Response and Status of
Transaction
Summary Report Listener - Show aggregated values for all users
- Useful when multiple users are simulated
- Provide performance information

17. 17
JMeter Testing Tools
• Test Plan
• Thread Group
• Controllers:
– Samplers &
– Logical Controllers
• Listeners
• Timers
• Assertions
• Configuration Elements
• Pre-Processor Elements
• Post-Processor Elements

18. • Pre-Processor Elements
• A Pre-Processor executes some action prior to a Sampler Request
being made.
• If a Pre-Processor is attached to a Sampler element, then it will
execute just prior to that sampler element running.
• A Pre-Processor is most often used to modify the settings of a
Sample Request just before it runs, or to update variables that aren't
extracted from response text.
• Post-Processor Elements
• A Post-Processor executes some action after a Sampler Request has
been made.
• If a Post-Processor is attached to a Sampler element, then it will
execute just after that sampler element runs.
• A Post-Processor is most often used to process the response data,
often to extract values from it. See the scoping rules for more details
on when Post-Processors are executed.

19. Execution order
1. Configuration elements
2. Pre-Processors
3. Timers
4. Sampler
5. Post-Processors (unless SampleResult is null)
6. Assertions (unless SampleResult is null)
7. Listeners (unless SampleResult is null)
• Timers, Assertions, Pre- and Post-Processors are only processed if there is a
sampler to which they apply.
• Logic Controllers and Samplers are processed in the order in which they appear in
the tree.
• Other test elements are processed according to the scope in which they are found,
and the type of test element

20. JMeter – Basic Elements

21. 21
JMeter – Basic Elements
• “number of threads” - In other words, this variable is the number of users
executing a “real life” use case on your system.
• This number is not the number of concurrent / parallel users executing a “real
life” use case on your system:
• the concurrency of the users depends on both the duration of your scenario
and the ramp up time configured on the thread group.
• The “ramp up time” in a thread group is the actual time taken by JMeter to
spawn all the threads.
• A rough estimation of the throughput (number of requests per second) during
the ramp up period of your test plan is:
number of threads / ramp up time (in seconds).

22. Tips & Tricks
• You should try to have a constant throughput during a run:
It is often very difficult to “control” the throughput particularly
during the ramp up period
• If your objective is to simulate a “peak”:
You should have a “high” number of threads and a “low” ramp
up time and number of loops
• If your objective is to simulate a “long run”:
You should have a “medium” number of threads, a “higher”
ramp up time and a “high” number of loops
• Note: The terms “high”, “higher”, “medium” and “low” are
voluntary qualitative in the 2 bullets above as they depend on
the system you are testing.

23. JMeter – HTTP Request Defaults

24. JMeter – Adding HTTP Requests
• The “HTTP Request Default” elements does not tell
JMeter to send an HTTP request. It simply defines
the default values that the HTTP Request elements
use.
• In our test plan we need to add at least one “HTTP
Request Sampler”.
• JMeter sends requests in the order that they
appear in the tree.
• Add HTTP Request to the JMeter Users element
(Add->Sampler->HTTP Request).

25. JMeter – HTTP Request

26. JMeter - Listener
• The final element you need to add is a
Listener.
• This element is responsible for storing all of
the results of your HTTP requests in a file and
presenting a visual model of the data.
• Select the JMeter Users element and add a
Graph Result Listener and Summary Report
Listener. (Add->Listener->Graph Results).

27. JMeter - Listener

28. Distributed Testing Using JMeter

29. 29
The Cloud and Load Testing
• One of the things the Cloud is useful for is load testing;
very large amounts of hardware can be used to generate
load at minimal cost.
• Added benefit, if your application you are testing is
external to your corporate network, your tests will be run
from a realistic location which prevents any problems
with artificial bottlenecks occurring on your LAN.
• This type of testing, using externally located hosts, is
increasingly common and JMeter is a superb open-source
solution for facilitating this.

30. Easy Amazon AWS & JMeter
• Thanks to Oliver from http503.com and his
automated JMeter on EC2 script
http://www.http503.com/2012/run-JMeter-on-amazon-
ec2-cloud
• Run with up to 20 concurrent EC2 instances
(default max number)
• Run when you want it and how you want it!

31. JMeter – http503.com script
• It does things like:
– Using Amazon’s API to launch instances
– Installing JAVA & JMeter
– Copying test files to each server
– Adjusting thread counts to ensure the load is evenly
distributed over each host
– Editing the jmx for file paths to external data files
– Displaying real-time aggregated results from the test as it
is running
– Downloading and collating all jtl files
– Terminating instances.

32. 32
JMeter-EC2
• Prerequisites
– Test plan to be run has a Generate Summary Results listener.
– Your JMeter test plan has a duration or loop count value set.
• Prerequisites specific to using Amazon:
– That you have an Amazon AWS account.
– You have Amazon’s API tools installed on your machine.
• Note: you can control execution from your local machine,
but you will need to open few ports for data return. Difficult
if you are behind corporate firewall/proxy.
– Solution: Run controller on Amazon AWS (another EC2 instance)

33. 33
Installation Process
• Go to the:
http://www.http503.com/2012/run-jmeter-on-amazon-ec2-
cloud/#example
• Small tips
– Create security group with port 22 open to the world (or
your IP address).
– Also allow all machines inside security group to access
each other.

34. 34
JMeter-EC2
• Setup your machine image key, and your
secret key pair.
– Call it:
project="myblogtestplan" count="3" owner="Vlad”
./jmeter-ec2.sh

35. DEMO TIME

36. 36
Results Analysis
• Running a test plan is only 50% of the
performance testing task.
• The most challenging part of the performance
testing process is the analysis of test results
and the identification of bottlenecks.
• Think of the load testing reports as the
evidentiary support to prove your innocence to
a crime in court.

37. Just a moment please…
• Before going any further, we should spend some time
on the measurable outcomes of a stress test. There are
mainly 2 interesting measures that you can record
when you run a stress test on a web application:
• The throughput: is the number of requests per unit of
time (seconds, minutes, hours) that are sent to your
server during the test.
• The response time: is the elapsed time from the
moment when a given request is sent to the server until
the moment when the last bit of information has
returned to the client

38. ... moment please …
• The throughput is the real load processed by
your server during a run but it does not tell
you anything about the performance of your
server during this same run.
• This is the reason why you need both
measures in order to get a real idea about your
server’s performance during a run.
The response time tells you how fast your
server is handling a given load.

39. Results Analysis – Interpreting Results
• What do we want to find inside our reports?
• Kind of Reports
– Summary Report*
– Graph Results
– View Results in Tree
– View Results in Table
• Extra report types
– Response Times vs Threads**
– Transaction Throughput vs Threads**
• * - must have report listener!
• ** - Thread == user (in JMeter world)

40. 40
Response Times vs Threads
• This graph shows how
Response Time changes
with amount of parallel
threads.
• Naturally, server takes
longer to respond
when a lot of users
requests it
simultaneously. This
graph visualizes such
dependencies.

41. 41
Transaction Throughput vs Threads
• This listener is very similar to Response Times vs
Threads, except it shows total server's transaction
throughput for active test threads.
• The formula for total server transaction throughput is
<active threads> * 1 second / <1 thread response
time>
• So basically, it shows the statistical maximum possible
number of transactions based on number of users
accessing the application.

42. 90% line of response time

43. Learn to Read Graph
• Load speed
• Throughput
• Deviation
• Hits

44. Results Graph - Average Load Time
• Page load speeds in
milliseconds.
• Lower is better
• On a stable system, it
should go flat

45. Results Graph - Deviation
• The deviation
(variability) of the load
speed in milliseconds.
• Lower is better
• On a stable system, it
should go flat

46. Results Graph - Throughput
• Throughput in pages per
second.
• Higher is better
• On a stable system, it
should go flat

47. • The black dots are
individual sample
times

48. Good sign
• When the values on the
graph begin to flatten
out, it shows that the
system has become
stable at that load.
– Speed flattening
– Throughput flattening
– Deviation dropping
– No exceptions ;-)

49. Connect Results With Logs
• Learn to relate problems on the server to its effect on the
graph.
• Big spikes indicate that you have a problems
• You may see the effects of:
– Exceptions
– Garbage collection

50. JMeter & New Relic

51. Tips & Tricks
• A clear name for each performance test
• non-GUI mode is more stable than GUI mode
• Do not use listeners if not needed
• Ramp-up is needed for heavy load
• Assertion is needed to simulate a virtual user
• Unstable tests: think of data while an user run its scenario
• If one user can not login, its later steps should not be counted
• Backup after every important step you made to your script easily by
cloning the jmx file
• Speedup JMeter script modifying with text editors which support
regex

52. Simulate User Behavior in JMeter
• Only Once Controllers
• Cache Management
• Cookie Management
• Header Management
• Think Times

53. Gaussian Random Timer
• This timer pauses each thread request for a random
amount of time, with most of the time intervals occurring
near a particular value.
• The total delay is the sum of the Gaussian distributed value
(with mean 0.0 and standard deviation 1.0) times the
deviation value you specify, and the offset value.

54. References
• Apache JMeter
http://jmeter.apache.org/
• jmeter-ec2 | Run JMeter on Amazon’s ec2 Cloud
http://www.http503.com/2012/run-jmeter-on-amazon-ec2-cloud
• Some thoughts on stress testing web applications with JMeter
http://nico.vahlas.eu/2010/03/17/some-thoughts-on-stress-testing-web-applications-with-jmeter-part-1/
• JMeter tips
http://www.javaworld.com/javaworld/jw-07-2005/jw-0711-jmeter.html
• Response Times: The 3 Important Limits
http://www.nngroup.com/articles/response-times-3-important-limits/
• Apache JMeter Custom Plugins
http://jmeter-plugins.org
• JMeter Wiki
http://wiki.apache.org/jmeter/JMeterLinks
• Amazon AWS EC2 Command Line Toolkit
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/SettingUp_CommandLine.html
• Bayo Ernie - Performance Testing With JMeter 2.9 [Kindle Edition]

55. Thanks for your time


56. Q & A