October 19, 2016 We hope you learn a little more about testing to improve user experience through different technology platforms

1. A.B.T. (Always Be Testing)
How UX testing can help you meet and exceed program goals

2. Why Test UX?
1

3. You might be thinking…
• It takes time and expertise to collect & analyze information…
• Testing might expose a need for more extensive work...
• This all impacts the cost and schedule of my project…
…So why should I do it?

4. 4
You’re losing money if you aren’t testing.
Testing helps identify problems before time and money
are spent creating something that doesn’t work well.

6. 6

7. 7

8. 8
Testing will help
you meet and
exceed your
goals.

9. User Experience User Engagement
User Satisfaction
Conversions / Meeting Goals

10. 10
Testing will validate
or invalidate your
assumptions.

11. Challenging Your Expectations
• Did you see what you expected to see?
• How are users defying your expectations?
• What is working well?
• What isn’t working well?

12. 12
Testing will help
you think like
your users.

13. Thinking Like Your Users
• See your content from the viewpoint of someone who isn’t you (your
organization, your BOD, etc.)
• Learn things you didn’t even know you should be thinking about

14. The Testing Blueprint
2

15. When?
The time is
always…

16. Testing Blueprint: When to Test – Now
Start testing right now, focusing on these areas:
1. High traffic pages
2. Conversion pages
3. Email campaigns

17. Testing Blueprint: When to Test – Optimal Times
Optimal testing times:
1. Before a redesign
2. Before EOY campaigns
3. Before large fundraising pushes (awareness or action months)
4. Don’t test during your largest and most critical campaigns, test before.

18. Testing Toolkit

19. Testing Toolkit: Tools for the Job
• Native Platform Analytics
• Google Analytics
• Crazy Egg
• Google Experiments
• Optimizely
• Moveable Ink (Email Campaigns)

20. Testing Toolkit: Native Platform Analytics
• Purpose: Data Collection
• Cost: Free/Included with Platform
• Learning Curve: Easy
• Examples:
– Jetpack for Wordpress
– Luminate Online Reporting
Jetpack for Wordpress

21. Testing Toolkit: Google Analytics
• Purpose: Data Collection
• Cost: Free
• Learning Curve: Medium
• Setup: Include JavaScript snippet
on every page of your website (or
any page you need tracking)
Google Analytics Dashboard

22. Testing Toolkit: Crazy Egg
• Purpose: Data Collection
• Cost: Low
• Learning Curve: Easy
• Use: View user clicks, hover and
scroll tendencies to determine effect
UX design with heat mapping.
Crazy Egg Heat
Map

23. Testing Toolkit: Google Experiments
• Purpose: Variation Testing (A/B/C)
• Cost: Free
• Learning Curve: Hard
• Setup: Use JavaScript to change
elements, Google
Analytics/Experiments handles
testing parameters and reporting
Google Analytics Dashboard

24. Testing Toolkit: Optimizely
• Purpose:
– Variation Testing (A/B/C)
– Data Collection
• Cost: Enterprise
• Learning Curve: Easy
• Setup: Insert JavaScript snippet
and run experiments using
Optimizely’s platform
Optimizely Experiment

25. Testing Toolkit: Movable Ink
• Purpose: Email Engagement
• Cost: Low (based on email sends)
• Learning Curve: Easy
• Use: Engage users with dynamic
images based on location,
constituent information or date/time.
Movable Ink Dashboard

26. Realistic Goals
To-Do List:
1. Wait for Tonight
2. Try and Take Over the World

27. Brainstorm Goals: The “How Can We?” Question
What are your goals? Start broad with a brainstorming session and generalized
language:
“How can we…”
– increase donations?
– get more email sign-ups?
– increase engagement with email campaigns?

28. Refine Goals: The “What Are We Measuring?” Question
Next, include a clear measurable you can monitor with data:
“How can we increase donation…”
– click throughs?
– conversions?
– average amounts?

29. Define Goals: The “Would This Work?” Question
Now refined, it’s time to define your goals with an action:
“Would this work?”
– A donation lightbox on high-traffic pages to increase donation click-throughs.
– A geo-targeted banner on the homepage to increase event registrations.
– Stronger language in our email campaigns to increase our CTR.

30. Measure Twice…
Test Once

31. Measuring Success: The First Measure
Before you test, answer these questions:
1. What is your current performance on these pages/campaigns?
2. What is the industry and internal organization average performance?
3. How long will we run the test to ensure statistically significant data?
4. What is a realistic expectation for success?

32. Measuring Success: The Test
While you test, here are your new mantras:
1. Be patient! Stick to the timeline and see it to the end.
2. Monitor consistently
1. Set up reports daily/weekly
2. Perform real-life tests periodically
3. Tweak if results show dramatic winner half-way through

33. Measuring Success: The Second Measure
After your test, debrief and decide:
1. Were the results statistically significant?
2. Did the test run long enough to ensure a large enough sample size for
scaling?
3. Were our expectations met?
4. What did we learn?

34. Testing for a Website
3

35. Google Analytics Content Experiment:
Donation Intercept Test
GOAL:
Determine the best CTA option
to increase traffic to the
donation form
35
TOOL:
Google Analytics (GA)
• No additional investment
required
• GA already set-up to track
the donation funnel

36. Control
36
Lightbox Test

37. Donation Intercept Test: Measures/Outcome
• Control:
– Donation button in header
– Donation array in footer
• Lightbox Version A:
– Lightbox appears after 15 seconds
• Lightbox Version B:
– Lightbox appears after 75% scroll
Measures
• Version A resulted in highest traffic
generation
• The control and Version A delivered
similar numbers of completed
donations
• Version B proved not to be a valid
option to drive either traffic or
donations
Outcome
37
If you have 1,000 donations per year, imagine what a 10% increase would do to your
bottom line!

38. Optimizely: Donation Array Test
38
GOAL:
• Determine the most valuable
offering for donation array
• Determine if there is any
variance based on entry
point to the donation form
TOOL:
Optimizely
• GA didn’t track donation funnel
• Easy to replicate and host multiple
forms
• Minimal disruption for their internal
web & IT teams
• Ability to segment audiences and
select traffic sources to determine
differences

39. Donation Array Test
39
Three Versions:
CONTROL VERSION A
(Lower)
VERSION B
(Higher)
$50 $35 $55
$100 $50 $100
$250 $100 $250
$500 $250 $500
$1,000 $500 $1,500

40. Donation Array Test: Measures/Outcome
40
Test Group
Home Page
Click-through Rate
High Traffic Internal Page
Click-through Rate
Donation Form
Completion Rate
Avg.
Donation
Variation 1 - Control 2.1% 0.014% 10.4% $99
Variation 2 - Lower 1.9% 0.020% 12.8% $106
Variation 3 - Higher 2.2% 0.022% 13.4% $144
Imagine if all of their donations from the first half of the year had come in at $144 vs. $99 avg.
gift ?

41. Optimizely : Geo-Targeted Event Recruitment Banners
41
GOAL:
• Determine if we could
convert Home Page visitors
to Walk participants
TOOL:
Optimizely
• Easy to modify pages without diving
deep into code
• Minimal disruption for their internal
web & IT teams
• Easy to serve specific creative to
different audiences (geo-targeted)
• Ability to set up timed experiments
based on event date

42. Geo-Targeted Event Registration Banners
42
Desktop Mobile

43. Geo-Targeted Recruitment Banners: Measures/Outcome
• Not a traditional test
• No versions to test against
• 6 pilot locations
• Each banner was coded to display to a
unique visitor within a specified geographic
catchment area
• Banners shut-off automatically the day
before the event
Measures
• Banners resulted in 2.5 – 6%
engagement rate, depending on locations
(vs. 0.08% for a traditional paid display
ad)
• No change in bounce rates or drop off
rates
• Registrations ranged from .5% – 1%
Outcome
43
What would a 1% increase in event registrations mean for your event?

44. Crazy Egg: Heat Mapping to Inform UX Design
44
GOAL:
• Re-design or Re-fresh
• Prove that the investment in
the current website design
still sound
TOOL:
Crazy Egg Heat-Mapping
• Site already used best practices for
successful event sites
• No obvious challenge areas
• Observe user behavior to identify
opportunities and gaps
• Connects to GA account via simple
code inserted into page
• Low monthly fee – very affordable
testing option

45. Heat Mapping to Inform UX Design
The Site Heatmap Overlay
Are site visitors doing what you want them to do?

46. Long Scroll Pages
Where are they
spending their
time?
Are they getting
to the most
valuable
information?
What
information are
they missing ?

47. Heat-mapping: Measures/Outcome
• Understanding how your site visitors are
interacting with your content to inform
possible updates or future testing
• Provides navigation cues to streamline
content to ensure focus is on your main
CTAs: register, donate, log in
• Identifies content placement priorities for
long-scrolling pages
Measures
• Proved that the overall structure of the
site is good
• Prioritize content on long-scrolling
pages
• Optimal placement for social sharing
tools
Outcome
47
Age old question: Website re-design or website re-fresh? Let your site visitors tell you what
they need.

48. Testing for an
Email Campaign4

49. Why email testing?
The channel where you are consistently reaching out to
communicate directly with your most dedicated supporters.
Really learn about your constituents and include those lessons in your
ongoing campaign strategies…
• What inspires them?
• What drives them?
• What might fall flat?
49

50. A/B Testing
What is it?
It’s split testing! Comparing two versions of a web page or email to see
which one performs better. You do this by splitting your audience and
serving up a different variant to find which creative results in a better
conversion rate.
Most email marketing tools have built-in, user-friendly
A/B testing features.
50

51. A/B Testing: General vs Hard-Hitting Language
51
GOAL:
• Prove theory to internal
review teams: hard-hitting
facts and statements would
result in better performance
• Statistically significant results
• Lift KPIs, including amount
raised
TOOL:
Built in A/B Testing Functionality
• Ability to easily split segments and
serve up A/B creative
• Manageable resource lift to create test
build from control email build
• Easy reporting to identify winners early
in campaign for later sends and,
ultimately, determine statistically
significant results

52. 52
General Language Version Hard-Hitting Language Version

53. A/B Testing: General vs Hard-Hitting Language
• Developed Marquee Testing strategy over
3 largest campaigns through first half of
year for 6 months of data on ONE test
• Followed Engagement KPIs: Open Rates,
CTR, Conversion Rate, # of Gifts, Amount
Raised, Avg Gift
Measures
• Hard-hitting copy had a direct impact not only on
the decision to give but the amount of which the
donor decided to give
– Amount raised and average gift KPIs for
launch up 100% or more compared to
control
• Confidence in a global brand refresh that heads in
bold direction
• All teams on the same page to focus on new
learnings
Outcome
53
If hard-hitting language was right for your organization and raised a
$30K campaign by 30-40%, you could make an additional $9,000 to
$12,000!

54. Movable Ink
What is it?
Contextual email marketing software that allows you to dynamically
change the content of an email upon the moment of open to tailor
messaging in real-time.
54

55. Movable Ink
What you can do with Movable Ink:
• Image Personalization
• Geo-Targeted Personalization
• Weather Personalization
• Device-Targeted Personalization
• Data Automation, Image Swaps
• Live Polling
• Live Social Feeds
• Real-Time Optimization
55

56. 56
Example: Humane Society CTA Button Testing

57. Example: Humane Society Average Gift Suggestion Test
57

58. ABT: Always Be Testing
58
• There is no wrong time to start testing
• Testing is way worth the investment
• Start by testing your assumptions
• Test in increments, one element at a
time

59. 59
WELCOME
TO THE FUTURE.
Coming soon…

60. Questions?