SOASTA's tens-of-thousands of tests in customer labs and production environments have uncovered issues that range from code level bugs to issues in 3rd party services. Testing early, often and at real scale is the only way to be fully prepared.
2. The Test Lab
Goals:
• Isolate bugs in code
• Optimize page performance
• Plug memory leaks
• Tune databases
• Establish performance expectations
Limitations:
• Production-quality hardware
• IT support and resources
• Bandwidth and network infrastructure
•3rd party application components
• Time
3. The Staging Environment
Goals:
• Verify capacity
• Establish configurations
• Stress available resources
• Verify performance expectations
Limitations:
• Availability / scheduling
• Not full-scale
• Excludes key components
• Time
4. The Full Production Environment
Goals:
• Test to and beyond expected limits
• Verify entire infrastructure
• Confirm load balancing and firewall integrity
• Certify performance in normal and extreme conditions
Benefits:
• All components
• 3rd party apps
• All integrations
• Continuous
• Global latencies
• Reality
6. Performance is Multi-Dimensional
Find problems fast when metrics are viewed in context, at scale
External and Internal user load
Complex Web Applications : AJAX, HTML5, video, mobile, web services
User Experience
Multi-Dimensional Views
7. Performance is Multi-Dimensional
Find problems fast when metrics are viewed in context, at scale
External and Internal user load
Complex Web Applications : AJAX, HTML5, video, mobile, web services
Application Deployment Deployment
Component & Infrastructure Infrastructure Network User Experience
Code (Internal) (External)
Extract
Transport
Load
Real-Time
End to End
The CloudTest Performance Data
platform is built for
scalability and real-
In-Memory
time identification
OLAP
of issues
Multi-Dimensional Views
8. Extending Performance Beyond the Lab
Discovering issues across teams, time and scale
Stage / Team
Debugging code
Garbage collection
Memory leaks
Memory leaks Database thread counts
Dev
& Test Slow pages Inefficient database queries
Method-level tuning
Test Lab (a few thousand users)
Scale of Test
9. Extending Performance Beyond the Lab
Discovering issues across teams, time and scale
Stage / Team
Configuration Setting & Sizing
Auto-scaling failures
Release Max sockets exceeded Global latency variance
& Deploy
Slow third-party plug-ins Security bottlenecks
Default configuration settings Search technology limits
Inadequate server resources
Garbage collection
Memory leaks
Memory leaks Database thread counts
Dev
& Test Slow pages Inefficient database queries
Method-level tuning
Test Lab Staging (thousands to 10’s of thousands)
Scale of Test
10. Extending Performance Beyond the Lab
Discovering issues across teams, time and scale
Tuning for full-scale…and well beyond
Latency between systems
Network Network configuration
& Ops
Network bandwidth CDN file placement
Conflict with other apps DNS routing
Stage / Team
Load Balancer configuration
Firewall max capacity
Auto-scaling failures Unbalanced web servers
Release
& Deploy
Max sockets exceeded
s Global latency variance
Security bottlenecks
Slow third-party plug-ins
Default configuration settings Search technology limits
Inadequate server resources
Garbage collection
Memory leaks
Memory leaks Database thread counts
Dev
& Test Slow pages Inefficient database queries
Method-level tuning
Test Lab Staging Production (100% +++)
Scale of Test
11. CloudTest Live
Real-world Issue Resolution
Company Profile Project Profile
Company Toy Manufacturer Environment Lab and Staging
Size Fortune 500 Application Drupal (Apache, PHP, MySQL)
Location Private Hosted Cloud
Objective Identify Breaking Points
Project Length Five Days (non-consecutive)
Lab Staging
12. CloudTest Live
Real-world Issue Resolution
• First test cycle in the lab • Some servers hitting 100% CPU Lab
• 239 hits per second • Bottleneck in throughput
• 40.5 MBit/second bandwidth
13. CloudTest Live
Real-world Issue Resolution
• Second test cycle in the lab • Some servers hitting 100% CPU Lab
• Higher concurrency target • Performance problem over time
• Bandwidth stayed the same • Load balancer flapping servers
16. CloudTest Live
Real-world Issue Resolution
• 767 Home pages served at 650 megs total transfer.
• That’s an 850kb page weight! Yikes!
Staging
17. CloudTest Live
Real-world Issue Resolution
Mission Debriefing - Fortune 500 Toy Manufacturer
Lab Staging
• Poor throughput on the initial tests • Massive page weight on the home page
• CPU hits 100% • Application unresponsive at ~2000 users
• Bandwidth cap identified in the lab
• Database connections leaking over time
• Load balancer dropping servers
Outcome
• Customer happy with the breaking point identified
• Optimized web assets in the home page
• Offloaded assets to the CDN
• Load balancer algorithm optimizations
18. CloudTest Live
Real-world Issue Resolution
Company Profile Project Profile
Company Media and Broadcasting Environment Production
Size Fortune 500 Application Java
Location Private Data Center
Objective 25,000 Concurrent
Project Length Three Days
Production
19. CloudTest Live
Real-world Issue Resolution
• Steadily increasing response times
• Constant stream of errors (< 1%) Production
• Variety of error types
21. CloudTest Live
Real-world Issue Resolution
• Connection reset errors starting the video stream
• Entitlement license server calls failing
Production
22. CloudTest Live
Real-world Issue Resolution
Mission Debriefing - Fortune 500 Media Company
Production Outcome
• Connection issues with CDN • Worked with CDN provider to reinforce certain geographies
• One problem page • Added capacity to the license server cluster
• Massive bandwidth usage
23. CloudTest Live
Real-world Issue Resolution
Company Profile Project Profile
Company Advanced Technology Environment Production
Size Fortune 50 Application Java (Struts), Oracle
Multiple HostedPrivate
Location
Data Centers
Objective 172,000 Concurrent
Project Length Two Months
Production
26. CloudTest Live
Real-world Issue Resolution
Mission Debriefing - Fortune 50 Technology Company
Production Outcome
• Load balancer firmware bug with 2048 bit cert • Worked with hardware LB manufacturer on a fix
• Database issues with the login process • Really low error rate
• Successful launch!
27. Q&A
Please join us for the next webinar in this series:
Testing Mobile Applications
May 3rd at 10AM PDT, 1PM EDT
Register at: http://bit.ly/ezq3fP
Contact SOASTA:
www.soasta.com/cloudtest/
info@soasta.com
866.344.8766
Follow us:
twitter.com/cloudtest
facebook.com/cloudtest
Notes de l'éditeur
Good Morning and welcome to the second in our new series, Best Practices in Performance Testing. This series is designed to share with you the best practices our performance engineering team is developing as they work with world-class customers of all sizes. Today we will be exploring what issues are uncovered as customers test applications starting a small scale in development and moving to full-scale testing in production. My name is Betsy Zikakis, I’m the VP, Marketing for Soasta and I’ll be your MC today.
Thank you so much for attending this webinar. The archive will be available on our website by the end of this week. Please tell others about it. Our next webinar in this series is focused on testing mobile applications. You can go to the link on this slide to register now. It will be up on the SOASTA home page to register tomorrow morning. And you will receive an email invitation next week.