You know you need to do more with your digital channels - but what is it and how can you maximize your results? The answer: Test! Attend this session to learn how to maximize results from your testing efforts and discover new ways to make your tests statistically valid, design better tests, and even find value in failure. With real case studies from WWF and other organizations like Special Olympics, Human Rights Watch and the American Diabetes Association, among others, this session will leave you with numerous ideas to test for success. Digital Handout: http://inahat.co/testing-handout

1. Quick Tips for
Better Testing

2. Jen Boland
Analytics and
Optimization Lead
Sara Hoffman
Web Development
Manager
Brian Rogel
Associate Director of
SEO and SEM
About us
#Bridge16
Avon Breast Cancer
Crusade
Beaconfire REDBeaconfire RED

3. CONFIDENTIAL 3
Agenda
• Creating a plan
• Test strategy
• Test setup
• Things to do while your are testing
• How to evaluate results
• What to do after testing

4. Create a Plan

5. Just because someone
else did it doesn’t mean it
will work for you.
CONFIDENTIAL5

6. Define Website Pain Points
1) Determine goals
2) List problem areas
3) Rank list based on
costs and benefits
#Bridge16

7. Review Metrics of the Proposed Test
Key Metrics:
• Unique Visits
• Pageviews
• Conversion Rate
• Average Gift
#Bridge16

8. Form Analytics & Error Messaging
#Bridge16

9. #Bridge16
FPO

10. Usability Testing

11. User Testing Tools
#Bridge16

12. Test Strategy

13. Determine Metrics that can be Tested
#Bridge16
TOP-FUNNEL
(Landing Pages)
MID-FUNNEL
(Transition Pages)
BOTTOM-FUNNEL
(Conversion Forms)
User’sJourney
Decision Tips:
• Top-funnel tests run faster
than bottom-funnel tests
• Testing conversions is not
always possible
• Make sure the target
sample size can be reached
in the proposed test
duration

14. Determining Sample Size
Sample Size Tips:
• Use sample size
calculators
• Run tests until the
sample size is reached,
not until statistical
significance is reached
#Bridge16

15. A minimum detectable effect
(MDE) is the smallest true
effect that has a ‘good
chance’ of being found to be
statistically significant.

19. Test Across Thank You Pages & Emails
C: Honor GiftB: Sustainer
w/ Premium
A:Sustainer

20. Test Duration: Minimum
Minimum Test Duration Tips:
• Run tests for full week cycles
• One week minimum is possible for websites
with exceptionally high traffic
• A two week minimum is preferred for most
websites
#Bridge16

21. Test Duration: Maximum
Maximum Test Duration Tips:
• Clearing cookies can cause sample pollution
o Lifehacker Poll: 72.35% of people clear
cookies or use private browsing
o Microsoft Report: studies show 30% to
40% of people clear cookies monthly
• Avoid running tests longer than four weeks
#Bridge16

22. Test Duration and Skewed Results
#Bridge16
Avoid running tests during holidays or seasonal anomalies
(Ex: End-of-year and non-end-of-year traffic acts very differently)

23. Test Duration and Sample Size Recap
#Bridge16
• Determine target sample size
• Run tests for full week cycles
• 2 to 4 week test durations are best
• Run tests until the target sample size is reached,
not until statistical significance is reached
• Be aware of holidays and seasonal anomalies

24. #Bridge16
Where are you going?
Make a map
January February March April May June July
Homepage
User Testing
Homepage Hero Slider A/B Homepage Static Hero
Build Hero Ask String High-Low
Survey -
Legal
Survey
Rebuild
overlay
Pre Year-End
Overlay POC
 possible
extension
Action TY Story Title
Suggested
Amount
inahat.co/bridge-roadmap

25. CONFIDENTIAL 25
Test
Variation
Name
Test
Length Traffic % # of Visitors VWO Type Test Location
ETA for
Results
Hero Homepage Donation Ask No Test 7 60% 587 ABCD
CMS
Homepage 1/12/2015
Hero Homepage Donation Ask Control 7 13% 1401 ABCD
CMS
Homepage 1/12/2015
Hero Homepage Donation Ask Var1 7 13% 1458 ABCD
CMS
Homepage 1/12/2015
Hero Homepage Donation Ask Var2 7 13% 1464 ABCD
CMS
Homepage 1/12/2015
Grey Overlay Homepage Real Control 7 50% 883 Split test
CMS
Homepage 1/20/2015
Grey Overlay Homepage Grey Overlay 7 50% 857 Split test
CMS
Homepage 1/20/2015
Hamburger Control 14 50% 1828 ABC
Homepage
and Directory 6/8/2015
Hamburger Button 14 25% 920 ABC
Homepage
and Directory 6/8/2015
Hamburger Left Logo 14 25% 943 ABC
Homepage
and Directory 6/8/2015

26. Create a Hypothesis
CONFIDENTIAL26

27. Hypothesis Creation
#Bridge16
Sections of a Hypothesis:
1) Statement of Problem
2) Proposed Solution
3) Expected Results

28. Hypothesis Template
#Bridge16
Based on <data discovered> we believe <website issue> is
causing <undesirable result>. We believe <change to be tested>
will help resolve <website issue> and cause <desired result>.

29. Writing your Hypothesis
#Bridge16
Example Hypothesis:
Based on a 31% below average conversion
rate for paid search traffic we believe a
lack of information on donation forms is
causing a high drop-off rate. We believe
sending people to an information rich
splash page before the donation form will
help resolve the high drop-off rate and
cause an increase in conversion rate.

30. Paid Search Landing Page Test
#Bridge16
+47%
Conversion rate
Less form fields
More information

31. Test Setup
CONFIDENTIAL31

32. Testing Tools
#Bridge16

33. Test Setup
• Preserve Query
Parameters
• Link with Analytics
Tools
#Bridge16
https://testsite.org?Donation2?d
f_id=12362&12362.donation=for
m1&utm_source=newsletter&ut
m_medium=email&utm_content
=link1&utm_campaign=fall

34. Beware of Peeping Toms
• Browser Tools like WAPPALYZER, dataSlayer, Pessimezely or
WASP can expose your test and variant names!
#Bridge16

35. Segmentation
#Bridge16

36. Test Large to Small
#Bridge16

37. Test Broadly
Test VariationControl
#Bridge16

38. Don’t Work Late or on Weekends
#Bridge16

39. While You’re
Testing…

40. Monitor Results
#Bridge16

41. Monitor Results
#Bridge16

42. Micro Conversions
#Bridge16

43. Micro Conversions
#Bridge16
Control
Variant
+9.7%
Website sign-in clicks
+4.9%
Credit card sign-in clicks
+21%
Registration clicks

44. Avoid Other Website Changes
#Bridge16

45. Results

46. #Bridge16
Tools to Help with Testing – Handout!
Calculators!

47. Tools to Help with Testing – Handout!
Your Good Friend – Excel!
Percent Change
(Current Value – Previous Value) / Previous Value
#Bridge16

48. Unanticipated Results
#Bridge16

49. P-Value and Statistical Power
#Bridge16

50. “If you use p=0.05 to suggest
that you have made a
discovery, you will be wrong
at least 30% of the time.”
- David Colquhoun

51. Analyzing Data: What’s a Win?
#Bridge16
Checklist:
 Low P-Value
 High Power
 Test Reaches Target
Sample Size
 Test is Not an Industry
Outlier

52. What Tools Tell You
#Bridge16

53. Two-Tail T-Test

54. After Your Test

55. Validate – Iterate!
inahat.co/bridge-validate
Hypothesis
Create /
Iterate
Continuous
Testing
Measure
Results
#Bridge16

56. #Bridge16
Modify Your Map
January February March April May June July
Homepage
User Testing
Homepage Hero Slider A/B Homepage Static Hero
Build Hero Ask String High-Low Paid Search
Ask
DRTV Ask
Survey -
Legal
Survey
Rebuild
overlay
Pre Year-End
Overlay POC
 possible
extension
Paid Search
Overlay
Action TY Story Title
Suggested
Amount
inahat.co/bridge-roadmap

57. Sharing Lessons Across Channels
#Bridge16
Using Google AdWords to test how effective different
copy variations are at improving click-through-rates.
Control Variants
+25.2%
CTR Improvement
-6.9%
CTR Reduction

58. Sharing Lessons Across Channels
#Bridge16
The difference in “Don’t miss” and “Don’t miss out on”
made a 34.4% difference in click-through-rates. The test
was significant at 99% with 99.93% statistical power.
Variant
+34.4%
CTR Improvement
Control

59. Cross-Channel Implementations
#Bridge16
Copy tests help determine the best messaging for your audience.
Improvements discovered can be leveraged in many channels:
Meta Content (Organic)
Ad Copy (Paid)
• Email
• Direct Mail
• DRTV
• Social Media
• Organic Meta Content
• Display Ad Copy

60. I’ve failed over and
over and that is why I
succeed.
CONFIDENTIAL60
Michael Jordan

61. Questions?
Download our Free Testing Guide:
http://inahat.co/testing-handout
Don’t forget to visit the
Solutions Showcase!
Many of the ideas discussed today are on display at the
Solutions Showcase!