Web and mobile performance has become business critical in today’s world. Website failures are costly in terms of lost revenue and tarnished reputation. Quality user experience has taken center stage. Are you ready to deliver? To help prepare for web traffic spikes associated with seasonal readiness, promotions or special online events, proper planning and execution is a must. This means that the combination of technology tools, implementation best practices and expertise needs to be carefully managed and optimized. Since 2008, SOASTA has conducted tens-of-thousands of tests, and logged over a million hours in the cloud. During this time we have accelerated web and mobile testing with scale and affordability for the world’s leading brands. During our upcoming webinar, we will share key SOASTA testing strategies needed to avoid epic web fails. They include the need to: - Test continuously - Test end-to-end - And, test at scale Join SOASTA experts Peter Galvin, SVP SOASTA Marketing and Brad Johnson, SOASTA VP Product Marketing as they share the best practices you need to follow to ensure your web and mobile apps are real-world ready. We will also show you how SOASTA’s industry leading CloudTest platform enables you to streamline and automate performance tests, so you can deliver quality user experiences every time.

1. 8
Webinar
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
0

2. What have we learned from recent epic fails?
TODAY’S SOASTA PRESENTERS
Peter Galvin:
SVP Marketing
Brad Johnson:
VP Product Marketing
Agenda:
• Poll question
• Lessons Learned
• Some Best Practices
Questions:
Submit in chat box during event
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
1

3. Poll Question

4. © 2013 SOASTA CONFIDENTIAL - All rights reserved.
3

5. © 2013 SOASTA CONFIDENTIAL - All rights reserved.
4

6. “We didn’t have enough
testing for a very
complicated project”
US Health and Human Services
Secretary, Kathleen Sebelius
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
5

7. •
•
•
•
•
•
•
Millions of uninsured citizens
Healthcare program mistrust
President’s approval rating dropped
Congress tied up in legislation
Public cynicism
Press and social media circus
Uncertainty for the future
Brought QUALITY and TESTING to the public forefront!
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
6

8. In March 2013
RECOMMENDATIONS
WHAT HAPPENED
―Name a single implementation
leader‖
―No single leader‖
―Lock down‖ requirements by April 30
―Streamline decision making‖
―End-to-end integrated operations and
IT testing‖
http://www.businessweek.com/news/2013-11-19/obamacareconsultants-warned-of-health-website-failure-in-march
―Shifting requirements‖
―Multiple definitions of success
―Insufficient time and scope of end-toend testing‖
http://www.npr.org/blogs/alltechconsidered/2013/11/19/246132770/this
-slide-shows-why-healthcare-gov-wouldnt-work-at-launch
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
7

9. © 2013 SOASTA CONFIDENTIAL - All rights reserved.
8

10. © 2013 SOASTA CONFIDENTIAL - All rights reserved.
9

11. © 2013 SOASTA CONFIDENTIAL - All rights reserved.
10

12. ―Twitter’s down…and it has cost the economy
$1 billion already?‖
Source: Venture Beat & Ray Wang at
Constellation Research 2012
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
11

13. OK...reduce Twitter usage assumptions 50%.
Then, only $400,641,025.00 was lost.
―Twitter’s down…and it has cost the economy
$1 billion already?‖
Source: Venture Beat & Ray Wang at
Constellation Research 2012
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
12

14. © 2013 SOASTA CONFIDENTIAL - All rights reserved.
13

15. ―Mass Panic As Common App Crashes‖
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
14

16. o
Aug. 14: New York Times
• 2 hours - Stock dropped - opened paywall (lost $$) –
Twitter storm
o
Aug 17: Google
• Less than 5 minutes – lost $500,000 – Internet traffic
dropped 40%
o
Sept 2: HSBC (UK)
• Internal and external systems – “Weekend of
disruption”
o
Nov 12: Kayazoonga.com (India)
• Last cricket match of Sachin Tendulkar - 19.7M
hits/hour – crashed all day – “Chaos”
o
Nov 18: YouTube
• 35 minutes – est. 100,000 users affected
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
15

17. © 2013 SOASTA CONFIDENTIAL - All rights reserved.
16

18. © 2013 SOASTA CONFIDENTIAL - All rights reserved.
17

19. 500 Tests
400,000
“We began working with SOASTA
CloudTest six months prior to the
games.”
Paul Bunnell,
Lead Architect,
London 2012
concurrent users
431 Million Visitors
4.73 Billion Page Views
Access from 201 Countries
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
18

20. Prep for…
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
19

21. Plan based on
Tested &
“ We were on very
tight timelines…the
SOASTA team was
able to run multiple
tests very quickly
6-8 million TV viewers
over a two-day
period and get us
tuned for 10 million hits going on issues to
immediately fix.”
Kunal Sarda,
VerbalizeIt
250 to 30,000 visits/day
COO and CoFounder
7x customer growth
Zero downtime
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
20

22. © 2013 SOASTA CONFIDENTIAL - All rights reserved.
21

23. “One of the best
investments to our website
this year .”
Nancy Jordan,
Tested to 7x Initial load test goals
IT Middleware
& Integration
Reached 9x previous year’s peak
Manager
Validated dynamic caching investment
Reduced hardware capacity needs
Validated Akamai to 4x beyond peak
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
22

24. © 2013 SOASTA CONFIDENTIAL - All rights reserved.
23

25. Tested to 120% of expected
Saved 50% on testing
“Website performance really
affects your brand image
when you have
problems…The inherent
savings to our brand by
using CloudTest is
load
priceless.”
—Cheryl Davis,
expenses
digital technical
operations manager
Hallmark Cards, Inc.
Reduced staff costs by 66%
Eliminated test software and hw costs
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
24

26. ―No amount of
testing within
reasonable time
limits can adequately
replicate a live
environment of this
nature‖
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
25

27. ―No amount of
testing within
reasonable time
limits can adequately
replicate a live
environment of this
nature‖
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
26

28. © 2013 SOASTA CONFIDENTIAL - All rights reserved.
27

29. o 55 Contractors
o Not enough time in schedule to conduct adequate performance
testing
o No Integration testing between systems
o No end-to-end testing simulating real life experiences of end users
o Followed waterfall development with parallel/disconnected projects
o Locked at 2000 users at the 1st step
http://www.washingtonpost.com/politics/house-panel-grills-contractorson-troubled-health-insurance-web-site/2013/10/24/8f42c748-3ca711e3-b7ba-503fb5822c3e_story.html
http://www.cnn.com/2013/10/29/politics/obamacare-warning/
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
28

30. o Test Continuously
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
29

31. Identify bottlenecks & inform
teams, just like unit and functional
tests in CI
CI system runs load tests
against a pre-prod site
…while unit, functional and
mobile device tests run in parallel
Daily (or more frequent) updates
keep teams ―performance focused‖
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
30

32. o Test End To End
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
31

33. Credit
Card
Web
analytic
s
CDN
Other
3rd pty
svcs
SUT/AUT
Web
Server
App Server
Cache
Web
Server
App Server
Cache
Web
Server
App Server
Web
Server
App Server
Load
Balancer
Databas
e
Customers
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
32

34. CloudTest
Credit
Card
Analytics
Main
Databas
e
Web
analytic
s
CDN
Other
3rd pty
svcs
SUT/AUT
Web
Server
App Server
Cache
Web
Server
App Server
Cache
Web
Server
App Server
Web
Server
App Server
Load
Balancer
―Conductor‖
Databas
e
Perf Tester 1
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
33

35. CloudTest
Credit
Card
Analytics
Main
Databas
e
Analytics
Load
Generator
Load
Web
analytic
s
CDN
Other
3rd pty
svcs
SUT/AUT
Generator
Load
Generator
Web
Server
App Server
Cache
Web
Server
App Server
Cache
Web
Server
App Server
Web
Server
App Server
East Coast (AWS)
Load
Balancer
Databas
e
Perf Tester 1
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
34

36. CloudTest
Credit
Card
Analytics
Main
Databas
e
Analytics
Load
Generator
Load
Web
analytic
s
CDN
Other
3rd pty
svcs
SUT/AUT
Generator
Load
Generator
Web
Server
App Server
Cache
Web
Server
App Server
Cache
Web
Server
App Server
Web
Server
App Server
East Coast (AWS)
Load
Balancer
Databas
e
Perf Tester 1
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
35

37. CloudTest
Credit
Card
Analytics
Main
Databas
e
Analytics
Load
Generator
Load
Web
analytic
s
CDN
Other
3rd pty
svcs
SUT/AUT
Generator
Load
Generator
Web
Server
App Server
Cache
Web
Server
App Server
Cache
Web
Server
App Server
Web
Server
App Server
East Coast (AWS)
Load
Balancer
Databas
e
Perf Tester 1
―Conductor‖
Data sources:
Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
36

38. Analytics
Load
Generator
Load
CloudTest
Analytics
Main
Analytics
Load
Generator
Load
Generator
Amsterdam (Azure)
Generator
Load
Generator
Analytics
Load
Generator
Load
Tokyo (AWS)
Generator
Load
Generator
Analytics
Load
Generator
Load
Web
analytic
s
CDN
Other
3rd pty
svcs
Generator
San Francisco (GoGrid)
Databas
e
Analytics
Load
Generator
Load
Credit
Card
Analytics
Load
Generator
Load
Virginia (IBM)
SUT/AUT
SOASTA
Generator
Load
Global
Generator
Test Cloud East Coast (AWS)
App Server
Cache
Web
Server
App Server
Web
Server
App Server
Web
Server
Chicago
(Rackspace)
Web
Server
Cache
Generator
Load
Generator
App Server
Load
Balancer
Databas
e
Perf Tester 1
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
37

39. Analytics
Load
Generator
Load
CloudTest
Analytics
Main
Analytics
Load
Generator
Load
Generator
Amsterdam (Azure)
Generator
Load
Generator
Analytics
Load
Generator
Load
Tokyo (AWS)
Generator
Load
Generator
Analytics
Load
Generator
Load
Web
analytic
s
CDN
Other
3rd pty
svcs
Generator
San Francisco (GoGrid)
Databas
e
Analytics
Load
Generator
Load
Credit
Card
Analytics
Load
Generator
Load
Virginia (IBM)
SUT/AUT
SOASTA
Generator
Load
Global
Generator
Test Cloud East Coast (AWS)
App Server
Cache
Web
Server
App Server
Web
Server
App Server
Web
Server
Chicago
(Rackspace)
Web
Server
Cache
Generator
Load
Generator
App Server
Load
Balancer
Databas
e
Perf Tester 1
―Conductor‖
Data sources:
Native (SSH), JMX, PerfMon, CA
Introscope, AppDynamics, New
Relic, CloudWatch,…. 38
© 2013 SOASTA CONFIDENTIAL - All rights reserved.

40. Analytics
Load
Generator
Load
CloudTest
Analytics
Main
Analytics
Load
Generator
Load
Generator
Amsterdam (Azure)
Generator
Load
Generator
Analytics
Load
Generator
Load
Tokyo (AWS)
Generator
Load
Generator
Analytics
Load
Generator
Load
Web
analytic
s
CDN
Other
3rd pty
svcs
Generator
San Francisco (GoGrid)
Databas
e
Analytics
Load
Generator
Load
Credit
Card
Analytics
Load
Generator
Load
Virginia (IBM)
SUT/AUT
SOASTA
Generator
Load
Global
Generator
Test Cloud East Coast (AWS)
Access
for all
App Server
Cache
Web
Server
App Server
Web
Server
App Server
Web
Server
Chicago
(Rackspace)
Web
Server
Cache
Generator
Load
Generator
App Server
Load
Balancer
Databas
e
Perf Tester 1
―Conductor‖
Data sources:
Native (SSH), JMX, PerfMon, CA
Introscope, AppDynamics, New
Relic, CloudWatch,…. 39
© 2013 SOASTA CONFIDENTIAL - All rights reserved.

41. o Test to Any Scale
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
40

42. Frequent Testing from Lab to Production
Scale is relative – bugs are everywhere
Tuning for full-scale…and well beyond
Latency between systems
Network
& Ops
Network configuration
Network bandwidth
CDN file placement
Stage / Team
Conflict with other apps
Load Balancer configuration
Auto-scaling failures
Release
& Deploy
Max sockets exceeded
Global latency variance
Security bottlenecks
Search technology limits
Default configuration settings
Inadequate server resources
Database thread counts
Memory leaks
Dev
& Test
Firewall max capacity
Unbalanced web servers
s
Slow third-party plug-ins
Garbage collection
DNS routing
Inefficient database queries
Slow pages
Method-level tuning
Test Lab
Staging
Production (100% +++)
Scale of Test
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
41

43. Top 10 Performance Problems
Where do we typically find the issues?
Architecture
Network
& Ops
Bandwidth
Stage / Team
Load Balancer
Third Party
Services
Connectivity
Release
& Deploy
s
Database Performance
Shared
Environments
Configuration Settings
Application Issues
Dev
& Test
Culture and Mindset
Test Lab
Staging
Production (100% +++)
Scale of Test
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
42

44. o HHS mistakes have quality at the forefront (again)
o Risks to revenue and reputation are measurable and high profile
o Speed and quality do coexist
o Best practices are well established (follow them)
o There IS time, and you CAN test realistically
o Start now
© 2013 SOASTA CONFIDENTIAL - All rights reserved.
43

45. Thanks
RESOURCES
Case studies: soasta.com/customers
Upcoming webinars: soasta.com/knowledge-center
- This Friday, Nov. 22: ―Getting Started with CloudTest‖
- Dec. 11: ―Continuous Testing for Optimal Performance‖
Need a test, fast? CloudTest On-Demand.
- email info@soasta.com
- You’ll hear from us by tomorrow
Contact SOASTA:
www.soasta.com/cloudtest/
info@soasta.com
866.344.8766
Follow us:
twitter.com/cloudtest
facebook.com/cloudtest

46. www.SOASTA.com