1.
How to Run Smarter A/B Tests, Faster
How to Run Smarter A/B Tests, Faster
@shanelle_mullin
We’ll cover…
1. How to conduct conversion research to find smarter A/B test ideas.
2. How to prioritize your A/B test ideas in a meaningful way.
3. What a high velocity testing program is and why it works.
4. Common high velocity testing program mistakes to avoid.
5. How to increase your testing velocity responsibly.

2.
How to Run Smarter A/B Tests, Faster
“You don’t have generic problems, you
have specific problems.”
~Peep Laja

3.
How to Run Smarter A/B Tests, Faster

4.
How to Run Smarter A/B Tests, Faster
1. Relevancy
Does the page’s content meet user expectation?
Does the page’s design meet user expectation?
How can it match what they expect / want even more?

5.
How to Run Smarter A/B Tests, Faster
2. Clarity
Is the content on this page as clear as possible?
Is the value prop on this page as clear as possible?
How can we make it clearer?

6.
How to Run Smarter A/B Tests, Faster
3. Motivation
What’s motivating the visitor to take action?
Is it clear, believable, timely and persuasive?
How can we increase motivation even more?

7.
How to Run Smarter A/B Tests, Faster
4. Friction
What is causing doubts, hesitations and uncertainties?
What makes the process difficult?
How can we simplify the process?

8.
How to Run Smarter A/B Tests, Faster
5. Distraction
What’s not helping the user take action?
What’s drawing attention?
Is everything that draws attention really necessary?

9.
How to Run Smarter A/B Tests, Faster
Cross-Browser Testing
Audience > Technology > Browser & OS
Switch from the default “Data” view to the “Comparison” view.

10.
How to Run Smarter A/B Tests, Faster
Cross-Browser Testing
Always compare within the same browser family.

11.
How to Run Smarter A/B Tests, Faster
Cross-Device Testing
Audience > Mobile > Devices
Switch from the default “Data” view to the “Comparison” view.

12.
How to Run Smarter A/B Tests, Faster
Speed Analysis
Page Load Time: Seconds until every element on the
page is done loading.
Page Interactive Time: Seconds until the site is usable.

13.
How to Run Smarter A/B Tests, Faster
Speed Analysis
Your site loads in 3 seconds or less.
Your site loads in 4 to 7 seconds.
Your site loads in 7 to 10 seconds (or more).

14.
How to Run Smarter A/B Tests, Faster
Speed Analysis
Behavior > Site Speed > Page Timings
Switch from the default “Data” view to the “Comparison” view.

15.
How to Run Smarter A/B Tests, Faster
Google Analytics Health Check
1. Are you collecting all of the data you need?
2. Can you trust the data you are collecting?
3. Is anything broken or tracking / reporting
incorrectly? Why?

16.
How to Run Smarter A/B Tests, Faster
Common Google Analytics Mistakes
1. Pages are missing.
2. Data is different in the shopping cart tool.
3. Cross-domain tracking isn’t working.

17.
How to Run Smarter A/B Tests, Faster
Full Health Check Guide
bit.ly/gahealthcheck

18.
How to Run Smarter A/B Tests, Faster
Start with the Highest Value
1. High volume, low value pages.
2. Low volume, high value pages.

19.
How to Run Smarter A/B Tests, Faster
Finding Broken Links

20.
How to Run Smarter A/B Tests, Faster
Finding Broken Links

21.
How to Run Smarter A/B Tests, Faster
Auditing Your Funnel
The further down the funnel, the smaller the
necessary impact.

22.
How to Run Smarter A/B Tests, Faster
Auditing Your Funnel

23.
How to Run Smarter A/B Tests, Faster
Auditing Your Funnel

24.
How to Run Smarter A/B Tests, Faster
Exploring Internal Search

25.
How to Run Smarter A/B Tests, Faster
Segmentation Is the Secret
bit.ly/gasegmentation

26.
How to Run Smarter A/B Tests, Faster
Click Maps
1. This info is largely available in Google Analytics via
“Navigation Summary”.
2. See when people are clicking on non-links.

27.
How to Run Smarter A/B Tests, Faster
Scroll Maps
1. Better prioritize your content and messaging.
2. Drastic colour changes mean a need for visual
cues… likely indicate false bottom.

28.
How to Run Smarter A/B Tests, Faster
Session Replays
1. Real users with real money.
2. Where do they pause? Where do they give up?
3. Use this to gather more insight on poor performers.

29.
How to Run Smarter A/B Tests, Faster
Form Analytics
1. Which form fields cause the most error messages?
2. Which do people hesitate to fill?
3. Which do people leave empty, even if required?

30.
How to Run Smarter A/B Tests, Faster
On-Site Surveys
Exit surveys.
On-page surveys.

31.
How to Run Smarter A/B Tests, Faster
On-Site Survey Questions
1. What was the purpose of your visit to our site?
2. Were you able to complete your task?
3. If not, why?
4. Is there anything holding you back?
5. Do you have unanswered questions?
Focus on actionable data, on the friction.

32.
How to Run Smarter A/B Tests, Faster
On-Site Survey Options
1 open-ended question.
Y/N and then an explanation post-answer.

33.
How to Run Smarter A/B Tests, Faster
On-Site Survey Guide
bit.ly/onsitesurveys

34.
How to Run Smarter A/B Tests, Faster
Customer Surveys
1. First time buyers.
2. No previous relationship with you.
3. Try for ~200 responses.

35.
How to Run Smarter A/B Tests, Faster
Customer Survey Questions
1. Who are you?
2. What are you using this product for?
3. What problem does it solve for you?
4. What do you like about the product most?
5. Did you consider any alternatives?
6. Why did you choose us over the competition?
7. What’s the one thing that nearly stopped you from buying?
8. What was your biggest concern or fear about us?

36.
How to Run Smarter A/B Tests, Faster
User Testing Tasks
1. A specific task.
2. A broad task.
3. Funnel completion.

37.
How to Run Smarter A/B Tests, Faster
Constant Ideation
1. Continue conducting conversion research.
2. Involve the entire team.
3. Focus on the entire funnel (acquisition to referral).

38.
How to Run Smarter A/B Tests, Faster
“If you come to me with an idea and it’s not
live in 2 weeks, it’s not because it’s a bad
idea. It’s because I have better things to test.”
~Pauline Marol

39.
How to Run Smarter A/B Tests, Faster
The Prioritization Problem
1. Not objective enough.
2. Not data-driven enough.
3. Not easily customized.

40.
How to Run Smarter A/B Tests, Faster

41.
How to Run Smarter A/B Tests, Faster
Benefits of PXL
1. Makes “potential” more objective.
2. Makes “ease” more objective.
3. Helps create a data-driven culture.
4. Easy to customize.

42.
How to Run Smarter A/B Tests, Faster
Prioritization Spreadsheet
bit.ly/pxlmodel

43.
How to Run Smarter A/B Tests, Faster
Better Management
1. Review KPIs.
2. Review tests launched (and not launched).
3. Review insights gained.
4. Choose tests for the upcoming week.
5. Discuss resources (or lack thereof).

44.
How to Run Smarter A/B Tests, Faster
Archive (Or Live to Regret It)
1. Avoid repeating tests by mistake.
2. Easily communicate wins and insights.
3. Improve the quality of future tests.
4. Create an insights > wins culture.

45.
How to Run Smarter A/B Tests, Faster
High Velocity Mistakes
1. Without buy-in and a testing culture, you’re
doomed.
2. Calling it too soon and non-representative samples.
3. Letting quality and effectiveness slip.

46.
How to Run Smarter A/B Tests, Faster
Testing Capacity
(52 / Avg. Req. Test Duration) x Number of Testable Pages
(52 / 2) x 10 = 260

47.
How to Run Smarter A/B Tests, Faster
Measuring Velocity
1. How well are you trending towards achieving your
capacity?
2. Trends are what matters… are you getting closer?
3. Chart it.

48.
How to Run Smarter A/B Tests, Faster
How to Run Smarter A/B Tests, Faster
@shanelle_mullin
1. Conduct proper conversion research.
2. Come up with ideas constantly.
3. Prioritize with data.
4. Manage the process.
5. Archive your learnings.
6. Achieve your testing capacity.
7. Stop listening to CRO BS you read online.

49.
How to Run Smarter A/B Tests, Faster
Resources
Optimization Training Program
bit.ly/cxlinstitute
Original UX Research
bit.ly/cxlresearch

50.
How to Run Smarter A/B Tests, Faster
Questions?
Ask any time via email: shanelle@conversionxl.com