This document outlines an approach for continuous experimentation and improvement. It recommends starting with data to identify high traffic, low conversion areas to improve. Form hypotheses about potential solutions and test them through small experiments. Document learnings to establish best practices and avoid regressions. The goal is to continuously run small tests to drive learning and improvements.
18. Simple 4-step experimentation process
Data
Figure out what to improve
Hypothesis
Make an educated guess
Experiment
Test your guess
Act
Decide what’s next
30. Increase Conversions
By A/B Testing
Do a Happy Dance
But Watch it Closely
Get More Traffic
So You Can Start A/B Testing
Get More Traffic
So You Can Do a Happy Dance
High Traffic
Low Traffic
High ConversionsLow Conversions
How To Figure Out What To Improve
31. What percentage of your
customers or users have
adopted each feature?
Intercom on Product Management
When you’ve already got usage, analyze it.
32. Where do you have the most
visitors, users, customers,
emails, or notifications?
More volume helps you test faster.
33. ๏ Website Traffic / App Installs
๏ Welcome Emails
๏ Web / Mobile Onboarding
๏ Email Digests
๏ Triggered Notifications
๏ Cart / Funnel Abandonment
๏ Dormant / Churned users
Common areas with large volume and high impact potential
47. Use simple equations to assess potential impact of each idea
DAN MCKINLEY HTTP://KISS.LY/DANETSY
visits
x
conversion
rate
x
order
value
x
li4
=
?
users
x
order
value
x
resurrec7on
rate
=
?
ESTIMATED
ESTIMATED
MODELED
MODELED
48. Assess the potential impact,
difficulty and priority of
each test idea you have.
Prioritize your tests
60. 1. Your starting hypothesis.
2. Dates for when you ran the test.
3. Screenshots of control and all variations used during the test.
4. Expected change in conversion.
5. Probability of change.
6. Screenshot of the raw data used in your analysis.
7. A screenshot to the A/B Test Report in KISSmetrics.
8. The decision you've made from the test.
9. What you learned from the test.
Document every A/B test and create a playbook
61. What learnings do we want to
carry over from the past?
Helps you avoid unknowingly
decreasing your conversions.
62. ✓ Keep the footer links
✓ Autofocus on the first form field
✓ Do not include social proof
✓ Link to the features page
✓ Use “Log in” copy on the bu on
Create experiment checklists based on learnings