Contenu connexe Similaire à Combining Performance Testing And Modelling For Easy Jet.Com V 1.0 (20) Combining Performance Testing And Modelling For Easy Jet.Com V 1.01. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Combining Performance Testing and Modelling for
easyJet.com
Danny Quilton, COO, Capacitas
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Agenda
• Business Context
• easyJet’s Goals
• Challenges with Agile
• How can we Reduce Risk?
• Performance Testing
• Performance Modelling
• What you need
• Costs
• Benefits
© Capacitas 2002-2009 2
2. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
About Capacitas
• UK-based Capacity and Performance Management specialists
• Training and professional services
• Hardware-vendor independent
• Tools independent
© Capacitas 2002-2009 3
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Business Context
© Capacitas 2002-2009 4
3. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Business Context
• Capacity planning service www.easyJet.com
• Business critical service
• 98% of revenue earned through the e-commerce service
• Highly competitive market
• Frequent innovation by adopting agile development
• Monthly software releases
• Legislative changes
• Consistent growth; typically 20% p.a.
• Low-cost operating model
• Requirement to manage risk
© Capacitas 2002-2009 5
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
easyJet’s Goals
© Capacitas 2002-2009 6
4. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
easyJet’s Goals
© Capacitas 2002-2009 7
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Agile Development Process
Scrum
Product (Daily
Meeting)
Backlog
OUT Sprint IN
Pre-release
Sprint Sprint Sprint Activities
Planning
Backlog (15-30 days) Review (e.g. Training,
Meeting Documentation)
LESSONS LEARNED Sprint
Retrospective Release
Meeting
© Capacitas 2002-2009 8
5. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Challenges Associated with Agile
• Frequent change introduces risk
• Window available for performance testing is typically 1-2 weeks
• Final production candidate code often not available at the start of
testing
• Multiple code releases within the testing window
• Increased probability of functional and environmental defects
• Frequent rework of code necessitates modifications to test scripts
• Cumulative effect on capacity from 12 releases per annum
• Narrow time window to resolve performance defects
© Capacitas 2002-2009 9
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Challenges Associated with Agile
© Capacitas 2002-2009 10
6. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Typical Causes of Degraded Service Performance
© Capacitas 2002-2009 11
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
How Can We Reduce Risk?
Cause of degraded performance Mitigated by performance Mitigated by modelling?
testing?
Changes in user patterns No Yes
Poor performance of 3rd parties May be limited by Yes
environment
Physical capacity bottlenecks May be limited by size of Yes
the environment
Logical bottlenecks Yes If known
Software configuration changes Yes Yes, if tested
Increases in demand May be limited by size of Yes
the environment
© Capacitas 2002-2009 12
7. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Why Test?
• Performance testing is used to understand the software’s
performance characteristics
• Do unknown time and growth related defects exist?
• Memory leaks
• Poor database design/indexing
• Do unknown concurrency related defects exist?
• Database locking issues
• Memory locking issues
• Race conditions
• Obtain model inputs
© Capacitas 2002-2009 13
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Why Model?
• Modelling uses our understanding of the software to predict end to
end service performance under different scenarios
© Capacitas 2002-2009 14
8. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Scenario Modelling
© Capacitas 2002-2009 15
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Coverage of Testing and Modelling
Testing Modelling
Business Scenarios
Application Infrastructure
Technical Scenarios
© Capacitas 2002-2009 16
9. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
High Level Process
© Capacitas 2002-2009 17
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Risk Assessment
© Capacitas 2002-2009 18
10. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Risk Assessment
• Assess the potential performance impact on different elements of
infrastructure
• Plan tests to help mitigate that impact
• Focus on areas of highest risk due to limited timeframes
Risks
Description ID Web DB Other Risk Mitigation
Project X --- Medium Medium Low Test 1
Project Y --- Low High Medium Tests 2 & 3
Project Z --- Low Low Low None
© Capacitas 2002-2009 19
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Performance Tests Employed
© Capacitas 2002-2009 20
11. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Baseline Tests
• Used to determine the impact Example service time comparison between two
fictional baseline tests
of the planned release
1400
• Comparative performance 1200
between releases
Service Time (ms)
1000
• May be Load, Stress or Soak 800
Tests 600
• Service times and response 400
times are compared to each 200
other 0
Database Web
Server Server
Service Service
Time Time
Test 1.1 Release X 752 1240
Test 1.1 Release Y 813 1192
% Difference 8% -4%
© Capacitas 2002-2009 21
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Single Transaction Type Tests
• To determine the impact on the Milliseconds Web CPU per page
(fictional data)
infrastructure of individual user
transactions
• Every business process which is to 132
be modelled will have a calculated 328
service time for each hardware 233
element
221 156
122
Home Search Select
Options Login Purchase
© Capacitas 2002-2009 22
12. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Modelling
Performance Test Results
© Capacitas 2002-2009 23
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Performance Modelling
Business
Service
Component
Virtual Infrastructure
© Capacitas 2002-2009 24
13. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Performance Modelling
Aircraft Business
Home Page Search Booking Service
HTTP1 APP1 APP2 DB1 Component
Virtual Infrastructure
© Capacitas 2002-2009 25
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Performance Modelling
600 Business
100 80 5 Service
160 70 10 80 Component
Virtual Infrastructure
3.5 transactions/s
© Capacitas 2002-2009 26
14. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Model Scope
• What infrastructure to include?
• Major contributor to the performance experienced by end-user
• Highly utilised
• Critical
• Load growth
• Frequently reviewed
Virtual Infrastructure
© Capacitas 2002-2009 27
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Model Scope
• What transactions to include
• Business critical
• Importance to revenue
• High-volume
• High component cost
Change
Home Page Search Booking
Password
Service
© Capacitas 2002-2009 28
15. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Demand Model Validation
• The model should be validated on
an ongoing basis against actual
600
values
• The first stage is to validate the
demand part of the model to ensure 100 80 5
that arrival rates are being
calculated correctly
• Following this the true demand can
be used to validate the utilisation 610
calculations
80 80 6
© Capacitas 2002-2009 29
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Capacity and Performance Model Validation
Comparison of Actual Database Load to Predicted Database Load
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Actual Database CPU Predicted Database CPU
© Capacitas 2002-2009 30
16. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
What you need – People
• Capacity and performance management skills:
• Demand forecasting
• Modelling
• Load testing
• A strong understanding of risk management
• A good understanding of the business model
• Flexibility
• Project management skills
© Capacitas 2002-2009 31
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
What you need – Process
• Interface with the release team
• Participation in the agile development ‘scrums’
• Access to developers to understand application functionality
• Interface with the test environment management team for test
planning
• Interface to the business to access business forecasts
• Interface to operations to access production measurements
© Capacitas 2002-2009 32
17. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
What you need – Tools & Environment
• An appropriately sized test environment
• Production-like environment is not required
• Consider the ratio of capacity between presentation, business and
data tiers
• Toolset
• Performance monitoring tools (production and test)
• Load generation tool
• Modelling tool
© Capacitas 2002-2009 33
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
What you need – Environment
© Capacitas 2002-2009 34
18. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
What you need – Data
• Software/hardware configuration (production and test)
• Performance measurements from production and test environments
• Benchmark data (Spec, TPC, etc.)
• Service times derived from performance test results
• SLAs
• Internal
• SLAs with third-parties
• Historical and forecast business demand
• Historical and forecast service demand
• Measured error of previous forecasts
© Capacitas 2002-2009 35
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Historical Business Demand
Fleet Plan to April 2009
180
y = 0.0461x - 1673.1
R² = 0.9668
160
140
GB
Airways
120 acquisition
Number of Owned Aircraft
100
80
60
40
20
0
Sep-04
Sep-05
Sep-06
Sep-07
Sep-08
Mar-04
Jun-04
Dec-04
Mar-05
Jun-05
Dec-05
Mar-06
Jun-06
Dec-06
Mar-07
Jun-07
Dec-07
Mar-08
Jun-08
Dec-08
Mar-09
Delivery Date
© Capacitas 2002-2009 36
19. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Historical Service Demand
© Capacitas 2002-2009 37
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Costs
© Capacitas 2002-2009 38
20. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Benefits
“Capacitas give us the confidence in making the correct release decision”
James Mackay, Test Manager, easyJet
Test Phase Status
Risk Assessment Complete
Performance Test Plan Complete
Targeted Performance and Stress Testing Complete
Regression Performance and Stress Testing Complete
All Defects: Closed Complete
Capacity Models Updated Complete
Capacity Plan updated and communicated Complete
© Capacitas 2002-2009 39
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Benefits
• Testing and modelling complement one another
• The combined approach provides effective risk-mitigation in the
context of:
• Frequent change
• Narrow performance test window
• Business growth
• The approach enables us to rapidly answer business and technical
‘what if’ scenarios
• Costs are limited through the use of a scaled-down performance test
environment and low-cost modelling tools
• Provides performance assurance in a highly competitive market place
© Capacitas 2002-2009 40
21. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing and Modelling for easyJet.com
Questions?
• Presentation will be available for download at www.capacitas.co.uk
• Please visit us at our stand for any further questions
© Capacitas 2002-2009 41