Step 1 - Find Your Top Optimization Opportunities LANDING PAGES WITH HIGH BOUNCE FUNNEL PAGES WITH HIGH EXIT Image source Start with your most valuable traffic sources ANALYTICS - TOP TRAFFIC SOURCE Landing pages with high bounce and high entries are optimization gold ANALYTICS - MOST VALUABLE ENTRY POINTS Funnel pages with high exit are the holes in the leaking bucket ANALYTICS - FUNNEL DROP OFF ANALYTICS - FUNNEL DROP OFF Segment Your Traffic For Accurate Analysis USEFUL SEGMENTS - NEW VISITORS WHO DIDN’T BOUNCE BEHAVIOR SIMILAR TO CONVERTERS Create a segment about your goal converters (in our case, Free Trial signups) Map their key metrics Create a segment to see those who behaved like goal converters but did not convert You can use this segment in multiple ways 1. Retarget them with a discount or promotional offer 2. Give them higher lead scores in your marketing automation system 3. Show them a right hand side pop-up 4. If you have their email, trigger a specific message if they don’t convert and leave the website 5. If they’ve opted in to browser push-notifications, trigger a message Step 2 - Ask Some Questions ON-SITE SURVEYS SAAS Page Question Insights drawn Pricing / Features / Trial- Signup What’s stopping you from signing up with us? ECOMMERCE Page Question Insights drawn Product page What’s stopping you from adding this product to cart? cart Step 3 - Easy Usability Testing EASY USABILITY TESTING Putting 5 to 10 users in front of your website or landing pages and observing them (or hearing them vocalize their thoughts) can teach you far more than most other forms of research. Caveat: these users have to be representative of your target customers. 1) User searches for ‘Returns Policy’ page and then proceeds to follow the steps listed ASKING THE RIGHT QUESTIONS Step 4 - Visual Analysis HEATMAPS Heatmaps can reveal some surprising insights More people clicked on the top-right “Free Trial” CTA instead of the primary, middle-of-page CTA What is it that we want our visitors to do, versus what are they doing instead? The goal should be to align our interests and theirs’ to a common outcome. Heatmaps quickly show you the difference. By now, problems on your pages should have ‘bubbled’ out. Step 5 - Hypothesis HOW TO CREATE GREAT HYPOTHESIS Research Hypothesis Image source: MarketingExperiments.com The Change: Effect model depends on 1. Presumed problem 2. Proposed solution 3. Anticipated result Image source: MarketingExperiments.com

1. A Step-By-Step Guide to
A/B Testing Research
SIDDHARTH DESWAL

2. Thue is the Kissmetrics Webinar Wizard and Marketing Ops Manager.
Before joining forces with Kissmetrics, he was a Lyft driver in SF,
which is also how he ended up as a KISSmetrics marketer. Whenever
Thue is not trying to automate everything around him, you can find him
hiking in the Sierras.
THUE MADSEN
Marketing Operations Manager, Kissmetrics
@ThueLMadsen
Siddharth runs demand generation at VWO. In addition to being
interested in all things CRO, he’s also passionate about B2B SaaS
marketing, Sales, growth hacking and human resource management.
SIDDHARTH DESWAL
Senior Marketer, Visual Website Optimizer
@SiddharthDeswal

3. @NEILPATEL #KissWebinar @Kissmetrics
#KissWebinar
@SiddharthDeswal

4. 1 Find your top optimization opportunities
2 Ask some questions
3 Easy usability testing
TABLE OF CONTENTS
4 Visual analysis
5 Build your hypotheses
5 Putting it all together

5. WATCH WEBINAR RECORDING NOW

6. Step 1 - Find Your Top
Optimization Opportunities

7. LANDING PAGES WITH HIGH BOUNCE

8. FUNNEL PAGES WITH HIGH EXIT
Image source

9. Start with your most valuable traffic sources
ANALYTICS - TOP TRAFFIC SOURCE

10. Landing pages with high bounce and high entries are optimization gold
ANALYTICS - MOST VALUABLE ENTRY POINTS

11. Landing pages with high bounce and high entries are optimization gold
ANALYTICS - MOST VALUABLE ENTRY POINTS

12. Funnel pages with high exit are the holes in the leaking bucket
ANALYTICS - FUNNEL DROP OFF

13. ANALYTICS - FUNNEL DROP OFF

14. LOG IN WITH
GOOGLE
Start Your Free Kissmetrics Trial

15. Segment Your Traffic For
Accurate Analysis

16. USEFUL SEGMENTS - NEW VISITORS WHO DIDN’T BOUNCE

17. USEFUL SEGMENTS - BEHAVIOR SIMILAR TO CONVERTERS
Create a segment about your goal converters (in our case, Free Trial
signups)

18. USEFUL SEGMENTS - BEHAVIOR SIMILAR TO CONVERTERS
Map their key metrics

19. USEFUL SEGMENTS - BEHAVIOR SIMILAR TO CONVERTERS
Create a segment to see those who behaved like goal converters but did not
convert

20. USEFUL SEGMENTS - BEHAVIOR SIMILAR TO CONVERTERS
You can use this segment in multiple ways
1. Retarget them with a discount or promotional offer
2. Give them higher lead scores in your marketing automation system
3. Show them a right hand side pop-up
4. If you have their email, trigger a specific message if they don’t convert
and leave the website
5. If they’ve opted in to browser push-notifications, trigger a message

21. Step 2 - Ask Some
Questions

22. ON-SITE SURVEYS

23. ON-SITE SURVEYS - SAAS
Page Question Insights drawn
Pricing / Features / Trial-
Signup
What’s stopping you from
signing up with us?
Common concerns,
objections, anxieties
Top exit pages Is there something you were
looking for that you couldn’t
find on the website?
Reasons that users leave,
Information that they
needed to make a decision
but couldn’t find
Top landing pages /
Homepage
How familiar are you with
[business offering]?
Understand how much
convincing/explaining do
your visitors need
Top landing pages /
Homepage
Where did you come to
know of us?
Top marketing channels

24. ON-SITE SURVEYS - SAAS

25. ON-SITE SURVEYS - ECOMMERCE
Page Question Insights drawn
Product page What’s stopping you from
adding this product to cart?
Common concerns,
objections, anxieties or ‘just
looking’
Top exit pages (product or
category)
Is there something you were
looking for that you couldn’t
find on the website?
Reasons that users leave,
Information that they needed
to make a decision but
couldn’t find
Top landing pages /
Homepage
Where did you come to
know of us?
Top marketing channels
Checkout / Cart What’s stopping you from
buying this product?
Why are visitors not
completing purchases in
spite of adding to cart

26. Step 3 - Easy Usability
Testing

27. EASY USABILITY TESTING
Putting 5 to 10 users in front of your website or landing pages and
observing them (or hearing them vocalize their thoughts) can teach you far
more than most other forms of research.
Caveat: these users have to be representative of your target customers.

28. USABILITY TESTING - ASKING THE RIGHT QUESTIONS
“You received a damaged product, try and return it”
vs.
“You received a damaged product, what do you do now?”

29. USABILITY TESTING - ASKING THE RIGHT QUESTIONS
“You received a damaged product, try and return it”
vs.
“You received a damaged product, what do you do now?”
1) User searches for ‘Returns Policy’ page and then proceeds to follow
the steps listed

30. USABILITY TESTING - ASKING THE RIGHT QUESTIONS
“You received a damaged product, try and return it”
vs.
“You received a damaged product, what do you do now?”
1) User searches for ‘Returns Policy’ page and then proceeds to follow
the steps listed
1) User opens ‘Live Chat’ and asks rep what to do, or
2) Takes to Twitter to complain to the brand, or
3) Looks for the ‘Returns’ page, or
4) Something that none of us were able to anticipate

31. EASY USABILITY TESTING
Choose your usability tasks/problems based on these parameters:
1. Do you want to understand problems in a specific, goal driven process,
or
2. Do you want to understand how users approach a problem they are
looking to solve?

32. Step 4 - Visual Analysis

33. HEATMAPS
Heatmaps can reveal some surprising insights

34. HEATMAPS
More people clicked on the top-right “Free Trial” CTA instead of the primary,
middle-of-page CTA

35. HEATMAPS
What is it that we want our visitors to do, versus what are they doing
instead?
The goal should be to align our interests and theirs’ to a common outcome.
Heatmaps quickly show you the difference.

36. By now, problems on your pages should have
‘bubbled’ out.

37. Step 5 - Hypothesis

38. HOW TO CREATE GREAT HYPOTHESIS
Research
Hypothesis

39. HOW TO CREATE GREAT HYPOTHESIS
Image source: MarketingExperiments.com

40. HOW TO CREATE GREAT HYPOTHESIS
The Change: Effect model depends on
1. Presumed problem
2. Proposed solution
3. Anticipated result
Image source: MarketingExperiments.com

41. Putting It Together

42. PUTTING IT ALL TOGETHER
1. Start by analyzing your top traffic source
2. Look at the landing pages that have high traffic and relatively higher
bounce
3. Look at pages that have the highest exit rates in a funnel
4. Segment your traffic to get more accurate insights (goal converters
vs. non converters, new visitors who didn’t bounce)
5. Run on-site surveys on your important landing or funnel pages
6. Ask live users to complete tasks or solve problems that involve these
pages
7. Run visual analysis on your most important pages
8. At the end of this process, issues with the pages should bubble
out, i.e. they will become obvious
9. Create hypotheses and start testing!

43. SIDDHARTH DESWAL
Senior Marketer, Visual Website Optimizer
@SiddharthDeswal
deswal@vwo.com
THUE MADSEN
Marketing Operations Manager, Kissmetrics
@ThueLMadsen
tmadsen@kissmetrics.com
Questions?