Having problems with your website’s performance? Does it take too much time and effort to determine the cause of a particular page’s poor performance? Would you like to find the root cause of client-side issues in an automated way? If you answered yes to any of these questions, then this session is for you. At GSI Commerce, an eBay company, Ron Woody manages a large team of performance engineers working on nearly nearly 100 eCommerce websites. Ron and his team have developed cutting-edge approaches for automating client- and server-side performance testing. Learn the specific approaches Ron’s team uses today for pre-release performance tests, production performance management, and website optimization. Find out the ways they’ve automated cross-browser performance testing—and analysis—to increase productivity and efficiency. Covering these and additional topics Ron shares a toolkit of performance testing ideas and approaches your team can use to ensure optimal application performance and a better user experience.

1. W11
Performance Testing
5/1/2013 1:45:00 PM
Cutting-edge Performance Testing on
eCommerce Websites
Presented by:
Ron Woody
GSI Commerce
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Ron Woody
Senior manager of QA performance at GSI Commerce (an eBay company), Ron Woody manages a team
of twenty-one performance engineers dedicated to ensuring that web-based performance is within
targeted requirements for all client sites and launches. Ron brings more than fifteen years of automation,
performance, QA management, and process experience to his current role. He heads the effort to improve
performance for all web stores released and supported by GSI while providing performance testing and
engineering vision to the company. Ron has served as a consultant to companies such as Comcast,
Bloomberg, Vanguard, and Bank of America.

3. 4/16/2013
Automating Client- and ServerClientServerSide Web Performance Testing
and Optimization
Ron Woody
Director of Performance
GSI Commerce, an
eBay Inc. company
|1
Who is GSI Commerce?
|2
1

4. 4/16/2013
About GSI Commerce
■ Companies of all sizes have selected GSI
Commerce® to help deliver an exceptional
and engaging eCommerce shopping
experience to their customers.
|3
GSI Commerce: QA Performance Processes
■ Performance Testing
– Platform/Product
– Projects
– Store Launches
■ Client-Side Testing
■ Server-Side Testing
■ Performance Analysis
|4
2

5. 4/16/2013
Business Impact of Performance
■
Performance impacts sales!
■ Poor performance increases the risk of site
abandonment
■ Slow performance may impact brand
– People remember their experience when a site is
slow; Users may be reluctant to return
■ Our clients’ competitors are only a click away
|5
Recent Performance Testing Objectives
■ Phase 1: Improve the end-user experience
via client-side and server-side testing
■ Phase 2: Increase productivity via Automation
■ Phase 3: Create more realistic client-side test
scenarios and enhance our testing further
|6
3

6. 4/16/2013
Phase 1 – Improve end-user experience via
endclientclient- and server-side testing
server■ Client-Side Testing
– dynaTrace AJAX Edition
– Improve the End-User Experience
■ Server-Side Testing
– dynaTrace Development & Test Center Editions
– Diagnosing server-side bottlenecks
■ End-to-End visibility
|7
ClientClient-Side Testing
■ Started with dynaTrace’s free AJAX Edition
– More accurate rendering times; greater visibility
|8
4

7. 4/16/2013
ServerServer-Side Testing
■ Added dynaTrace Development Team & Test
Center Editions
|9
Phase 2 – Increase Productivity via
Automation
■ Server-side Automation
– LoadRunner with dynaTrace Test Edition
– dynaTrace AJAX Edition + Test Center Editions
provide in-depth views
■ Client-side Automation
■ Use of dynaTrace AJAX API’s to extend our
reporting
| 10
5

8. 4/16/2013
Benefits of using dynaTrace: Visibility
■ dynaTrace provided visibility we hadn’t
had before
– But testing browser use-cases was manual,
and very time-consuming
■ Increasing Productivity via Automation
– LoadRunner / Performance Center
– QuickTest Professional / Selenium
– Quality Center / ALM
| 11
Solution: Automated Browser Lab
■ Virtualized browser lab to capture the end-user
experience with the touch of a button
■ Cutting-edge automated home-grown solution
–
–
–
–
–
dynaTrace AJAX Edition
Selenium
Google Page Speed
Webpagetest.org
Shunra network emulation tools
■ Real-Time reporting system to show results
| 12
6

9. 4/16/2013
GSI Commerce’s Browser Lab
| 13
GSI Commerce’s Browser Lab
| 14
7

10. 4/16/2013
Why dynaTrace?
■ Visibility into Rendering
– As well as other Web 2.0 challenges
– Was not available in HTTPWatch
■ IE 7,8,9 support
– Firebug didn’t support IE
■ Now dynaTrace supports Firefox as well
| 15
Basic
Browser
Lab
Diagram
| 16
8

11. 4/16/2013
GSI Commerce’s Browser Lab Front-End
Front-
| 17
A Sample Browser Lab Report
| 18
9

12. 4/16/2013
A Sample Browser Lab Report
| 19
Using Automation to Save Time and Money
■ With automation, GSI saved hours per test
– Reducing test time from 20 hours to ~1 hour
■ Nearly a 20x time savings
■ Reproducible results
■ Shortens project time
– Allows for more iterative testing
| 20
10

13. 4/16/2013
Additional Benefits of our Browser Lab
■ Pinpoint problems quickly
– Problems that were difficult / impossible to see
before
– Full Waterfall / timeline
– Rendering
– DOM Manipulations
■ Allows us to be proactive
■ Leads to a faster site
■ Enables clients to make informed decisions
based on the results of our testing in QA
| 21
Benefits of dynaTrace: Better Collaboration
■ Dev and QA collaboration
– dynaTrace AJAX Edition sessions shared with
Development
– Now Development sees exactly what Test sees
– Less back & forth
– Faster resolution / optimization
■ Dev Teams can now use dynaTrace prior to QA
■ Developed Performance Requirements that
include dynaTrace rankings
– Dev teams now design to these requirements
| 22
11

14. 4/16/2013
Phase 3 – Create more realistic client-side
clienttest scenarios and enhance our testing further
■ Network emulation for more realistic traffic
■ Larger samplings with automation
■ Browser weighting to generate more accurate
timings across browsers
■ Automatic removal of outliers
| 23
Performance Objectives Revisited (ROI)
■ Phase 1: Improve the end-user experience
– ROI: We now have visibility into performance issues and how to
resolve them quickly and more efficiently thus saving us time and
money
■ Phase 2: Increase productivity via Automation
– ROI: ~20x savings with our Browser Lab solution
– can now do testing and analysis simultaneously
– faster time to market; more complete testing
■ Phase 3: Create more realistic test scenarios
– ROI: More accurate timings through browser weighting and network
emulation; increased consistency by reducing noise in our results
| 24
12

15. 4/16/2013
Recent additions
■ Leveraging dynaTrace browser agent (AJAX
Edition Premium Version)
– Ability to see end-to-end performance in QA
(PurePath) via Performance Center scripts
■ Use of dynaTrace (Production Edition) to
become even MORE proactive
– Ability to find issues that exist in Production
(including issues that clients haven’t reported yet)
– Determine where users are abandoning our sites
– Ability to monitor Apache
| 25
What’s Next?
■ Browser Lab aspirations
–
–
–
–
Mobile testing
Cached vs. Non-Cached pages
Film strip
Integration with webpagetest.org
■ Expand to Continuous Integration
– Enable Dev to find/fix problems before they reach
QA
| 26
13

16. 4/16/2013
GSI Commerce’s Performance Best Practices
■ Performance matters!
■ Define Performance Targets
■ Test both client-side AND server-side
performance
– Cross-browser testing
– Add server-side testing
– Tie everything together
| 27
GSI Commerce’s Performance Best Practices
■ Automate!
■ Get Test and Development on the same page
■ Get Proactive
■ Benchmark your site against your competition
| 28
14

17. 4/16/2013
QUESTIONS?
| 29
15