1.
Making More Happen –
How to Measure Revenue in Milliseconds

2.
Cliff Crocker
@cliffcrocker
SOASTA
VP Product, mPulse

3.
Performance is a Business
Problem

4.
Funnel Optimization is Everything
• Merchandizing
• SEO
• Multivariate testing
• Pricing
• Ad Campaigns & targeting
• User Experience
Performance IS User
Experience!!

5.
Yahoo! - 2008
Increase of 400ms
causes 5-9% increase in
user abandonment
http://www.slideshare.net/stubbornella/designing-fast-websites-presentation

6.
Shopzilla - 2009
A reduction in Page Load
time of 5s increased site
conversion 7-12%!
http://assets.en.oreilly.com/1/event/29/Shopzilla%27s%20Site%20Redo%20-
%20You%20Get%20What%20You%20Measure%20Presentation.ppt

7.
Walmart - 2012
http://minus.com/msM8y8nyh#1e
SF WebPerf –
2012
Up to 2%
conversion
increase for every
second

8.
SIMULATION

9.
Unfortunately, it’s not always
that straightforward

10.
Not All Pages are Created Equal
For a typical
eCommerce site,
conversion rate
drops by up to 50%
when “browse”
pages increase
from 1 to 6
seconds

11.
Not All Pages are Created Equal
However, there is
much less impact
to conversion
when “checkout”
pages degrade

12.
Getting Started
• Organizational challenges
• Process challenges
• Visibility
• Expertise
• Focus & prioritization

13.
Steve Skroce
Staples
Sr. Manager, Site Operations
steven.skroce@staples.com

14.
How Did I Get Here
Developer SEO Performance

15.
SiteSpeed & Staples
• A top goal for Staples eCommerce
• It’s about customer experience
• Speed is essential to UX
• Because Staples is highly matrixed
• Visibility into the organization is key
• Approach as a discipline and institutionalize the process

16.
First Things First
• You can’t manage what you can’t measure
• Earliest work involved selecting the proper tools
• Business and Engineering led team
• Already had synthetic transaction monitoring tool
• Backend tools were also available
• RUM tool was missing from the set
• Needed RUM info if we were going to affect our customers’ experiences

17.
First Things First
• Assembling a cross-functional team
• Business
• Decides which features and functionality lives on the site
• Engineering
• How something gets built can make or break page load time
• Marketing
• Asset optimization is important
• Merchandising
• On-site content
• Analytics
• Analytics team ties everything together
• And others…

18.
Why Is Speed Important to Staples?
Bounce Rate increases as load time increases

19.
Why Is Speed Important to Staples?
Conversion rate decreases as load time increases

20.
The First Pass
• Governance meetings to evaluate functionality
• Cataloged all site features
• Weekly meeting to discuss one or two features in detail
• Feature owner comes prepared to review all aspects
• Full Page Load vs. Time to Interactive
• How Staples defines them
• FPL ensured wide exposure / awareness through the organization
• Everyone was involved from the beginning
• Set the expectation of accountability

21.
The First Pass
• Onsite promotional banners were too “heavy”
• Inconsistent image optimization
• Depended on the individual, not the process
• 25k and 250k
• Put a new process in place

22.
The First Pass
• Testing practices were inefficient
• Two vehicles for A/B testing
• Eliminate one
• Process improvement opportunity for ending campaigns
• Eliminate lingering artifacts

23.
The First Pass
• Needed “sanity” around third party calls
• Categorization of calls
• Ads
• Marketing
• Analytics
• Recommendations
• Social Buttons
• Etc.
• Removed some calls, optimized others
• Further work needs to be done

24.
The First Pass
• Database calls were inefficient
• Some pages went from 200 calls to 10

25.
Evolution
• Go bolder, sooner
• Sprint model allowed us to deliver something every release
• Something, not something substantial
• Not enough time for big, bold changes
• We needed to step-back and reevaluate

26.
How fast are we, really?
• Examples of pagespeed for Staples
• Median vs. percentiles
• Make it simple for a wide audience to understand the metrics
• Key page performance – target specific pages to focus on
• Home, Product, Search, Class, Login, Cart

27.
Make Faster Page Load Times Happen
• mPulse
• WebPagetest.org
• Dev Tools
• NetPanel
• “Motivational” emails from the top

28.
Make ‘Faster Page Load Times’ Happen
• Confirmation of what we already knew:
• Front-end vs. back-end time
X

29.
JS refactoring
Reduced k size by 48%
Reduced lines of code by 52%
CSS refactoring
Reduced k size by 83%
Reduced lines of code by 88%

30.
Early Results
• Shaved 1 full second off of Home page median
• Shaved 6 seconds off of the 98th%
• Conversion improved by roughly 10%