2. AppLabs Confidential 2
Purpose & Pre-requisites
Purpose of this presentation: The purpose of the presentation is to give an overview of the
ubiquitous Cloud and explain the latest trends and techniques in Cloud Performance Testing.
Pre-requisites: It helps if the intended audience has a basic understanding of
Cloud terminologies
Virtualization
Traditional Performance Testing Life Cycle (PTLC)
3. AppLabs Confidential 3
Agenda
Cloud 101
Cloud offerings
Cloud deployment models
Cloud vs. Non-cloud solutions
Cloud Performance Testing
Challenges of Cloud computing
Tools at each phase of PTLC
AppCloud
AppCloud - PoC
AppCloud - The road ahead
Key takeaways
Recommended reading
4. AppLabs Confidential 4
Cloud 101
Characteristics of a Cloud
» On-demand self service
» Elasticity
» Broad geographical network access
» Pool of resources
» Pay-as-you-go
» Customizable platforms
» Universal access
» Multi-tenancy
Cloud
On-demand
Software
NetworkInfrastructure
5. AppLabs Confidential 5
Cloud offerings
1. Software as a Service (SaaS) - On-demand software that is accessed via a web browser and
is paid on a subscription basis. Allows users to run existing online applications.
2. Infrastructure as a Service (IaaS) Infrastructure (which includes storage, hardware, servers
and networking components) is provided as a Service to the end-user on a Pay-per-use model.
3. Platform as a Service (PaaS): Externally hosted service providing a complete platform to
create, run, and operate users own applications.
6. AppLabs Confidential 6
Cloud deployment models
1. Public Cloud
(Amazon EC2, Intel, Cisco)
Resources are available to general
public on a self-service billing model
2. Private Cloud
(Amazon EC2)
Resources are available solely to a
specific organization
3. Community Cloud
(Healthcare & Media )
Created for a specific community
catering to the needs of a group
4. Hybrid Cloud
(Microsoft Azure, GoGrid)
Combination of Public , Private or
Community Clouds
7. AppLabs Confidential 7
Cloud vs. Non-cloud solutions
Cloud Solutions
Pay-by-usage
Access anywhere
Customizable Infrastructure
Reduced human costs
Eliminates depreciable expenses
Multi tenancy
Self service model
Effective hardware utilization
Non-cloud Solutions
Huge cost and effort
Geographical location limitation
Physical server maintenance
Human resources management
Static machines
Static platforms
8. AppLabs Confidential 8
Cloud Performance Testing
As shown above, there are 3 possibilities of Performance Testing in the Cloud.
9. AppLabs Confidential 9
Why to choose Cloud Performance Testing?
Increased
Expectations
Faster LG’s
setup
Isolate Perf
bottlenecks
faster
Elastic Load
Generation
On-demand
environment
Pay-per-use
model
Customizabl
e SLA’s
No geo.
limitation
for load
generation
Performance Testing over a Cloud
offers vital benefits such as:
1. Faster Load Generators setup
2. Load can be generated from any
location
3. Elastic load generation
4. Reduced costs
5. Performance bottlenecks can be
isolated faster
6. No physical maintenance of the
servers
10. AppLabs Confidential 10
Challenges of a Cloud
While there are many tangible benefits with the cloud, there are some challenges in adopting a
cloud, some of the key problems faced by a cloud include -
Privacy
Security
ManageabilityAvailability
Performance
1. Data Integrity
2. Data Theft
3. Privacy issues
4. Infected Application
5. Data loss
6. Data Location
7. Security on Vendor level
8. Security on user level
11. AppLabs Confidential 11
Tools at each phase of PTLC
Phase Applicable Tools
Discovery phase AppCloud Manager web application
Test Script development phase
OpenSource Apache JMeter, HP LR or Borland
SilkPerformer
Test Execution phase
AppCloud Manager, Amazon cloud, Rackspace or CSC
cloud
Test Reporting phase
AppCloud Manager, Amazon Cloudwatch, Native
Performance Monitoring tools or Cloud Analytics
13. AppLabs Confidential 13
AppCloud – An Introduction
• Requirements gathering through
questionnaire
• Discover performance
characteristics of the application
• Establish performance goals and
team
Discovery Phase
• Setup test environment
• Setup master data
• Create performance test scripts
• Identify & Deploy monitoring tools
Test Design &
Execution Phase
• Collect performance metrics
• Analyze performance metrics
• Prepare performance test report
Test Analysis &
Reporting Phase
Discovery + Planning + Test Executions + Test Analysis & Reporting
=
AppCloudoffered as a TaaS
14. AppLabs Confidential 14
AppCloud management console
Discovery
Phase
• Questionnaire module
Test Design &
Execution
Phase
• Execution module
Reporting
Phase
• Reporting module
AppCloud Management Console has the following 3 modules
15. AppLabs Confidential 15
Features
Benefits
AppCloud console – Questionnaire module
AppCloud management console’s Questionnaire module helps uncover the Customer requirements during the
initial discovery phase of the project.
• Helps the customer to fill the questionnaire form over the internet
• Accordingly the team can plan for the Project resources and take it forward henceforth
• Provides customized questionnaire to each
customer
• Globally accessible without any limitations
• Immediately notifies the Customer and Admin on
the status of filled-in questionnaire
• An easy to understandable GUI
• One can create, edit and customize the questions
as per the need
Reporting module
16. AppLabs Confidential 16
Features
Benefits
AppCloud console – Reporting module
AppCloud management console’s Reporting module generates elegant final Performance Test summary reports
within a fraction of seconds from the huge log files
• The major benefit is one can drastically reduce the time consumed in generation of elegant final
summary reports.
• Globally accessible web application
• User friendly GUI.
• The tool can spawn any number of records, transcending Excel’s 32
lakh row limitation.
• Generates elegant MS-Word reports.
•Allows the user to print, save, merge and zoom the graphs.
• Saves productive time.
Reporting module
19. AppLabs Confidential 19
Proof of Concept (PoC) Overview
» An Open Source e-commerce application available under GNU General Public License
was tested with both Commercial tool and an Open Source tool Apache JMeter on the
same Cloud infrastructure for 10,000 users
» Both Load Runner and JMeter used the Load Generators available in the Amazon
Cloud.
» Both the tests were designed to simulate equal number of transactions.
JMeter HPS LoadRunner HPS
21. AppLabs Confidential 21
PoC - Key metrics
Regardless of the tools, both the tool response times were comparable and within the
acceptable limit (The variation is in fraction of milliseconds).
S.No. Parameter Name HP Load Runner Apache JMeter
1 Ramp-up
4 vusers every 2 seconds (83
minutes)
4 vusers every 2 seconds (83
minutes)
2 Vusers 10000 9996
3 Ramp-down
15 vusers every 2 seconds (22
minutes)
15 vusers every 2 seconds (22
minutes)
4 Test Duration 2 hours, 20 minutes 2 hours, 20 minutes
5
Average Throughput (Bytes per
second)
43,806 42,118
6 Total transactions 154030 156194
7 Errors 0 0
8 Think time 60 seconds 60 seconds
9
Average response time (all
transactions) in the final test
0.027 0.027
22. AppLabs Confidential 22
PoC - Performance Testing Costs
0
50000
100000
1
Physical Data Center 67213
Amazon Cloud 48
Cost(in$)
Cost - Physical Data Center vs. Amazon Cloud
Source: Amazon Cloud
Environment Parameter Cost per unit Total Cost
Physical Data Center
(LR)
Machines $10 per hour 900
Bandwidth $677 per 100Mbps 677
Tool License For 10,000 users license 65636
Amazon Cloud (JMeter)
Machines $ 0.48per hour 48
Bandwidth $0.10 per GB usage 0
Tool License 0 0
23. AppLabs Confidential 23
AppCloud - The road ahead
Integrate the Test
execution interface
into AppCloud
console
Integrate AppCloud
console with CSC
cloud
Market in Pre-sales
brochures
24. AppLabs Confidential 24
Key takeaways
» Cloud is here to stay – Get tuned to it
» By 2012, 80% of Fortune 1000 enterprises will pay for
some cloud-computing service (Gartner)
» By 2011, CIOs Must Answer the Question, “Why Not Run
In The Cloud” (Forrester)
» Customers has to pay only for what they use
» Enroll in Amazon sandbox
» Market about the AppCloud services in the Pre-sales
brochures
AppCloud
On
Demand
Zero tool
licensing
cost
Reduced
infra costs
Managed
testing
25. AppLabs Confidential 25
Recommended reading
Wikipedia
http://en.wikipedia.org/wiki/Cl
oud_computing
Refer to Cloud webinars and
latest happenings
http://aws.amazon.com/ec2/p
ricing/
