Testing is an important part of a successful website as it allows for making changes on the fly that can lead to a much larger impact on your business. This section will outline the best practices for testing & personalization and our core methodology combining reach & impact to score the "best" test.
8. Pre-testing considerations
People
How much of an effect will
this test have on the users?
Process
Do you have the right
technology and skills in place
to develop and execute the
test?
Technology
Do you have the right
technology and skills in place
to develop and execute the
test?
11. Types of Tests
A/B test
Multivariate test
Multi-page tests
split up site traffic in a balanced
way, and then show one t of
users version A and the second
version B
setup to test many different
individual changes on a page
all at once
These tests involve changing
the experience across multiple
pages
CHANNELS
low high
C
O
M
P
L
E
X
I
T
Y
high
12. Client Example |
A/B Testing
Client was concerned that not having a
clearly defined 'guest' sign in option
for the checkout flow was driving away
customers from converting
13. The new design led to a higher
conversion level
Client Example |
A/B Testing
Tested different versions comparing
the default with a new design that had
guest sign-in front and center
17. Define what metrics to determine test
success
Translate a test idea into a formal hypothesis
Define what metrics to determine test success
Grade each hypothesis
18. Three dimensional grading
Reach
Impact
Technical difficulty
How many users will see this test?
How much of an effect will this test have on the users?
Do you have the right technology and skills in place to develop and
execute the test?
19. Metric selection
Metric selection is key to gauging test results, ensure
metrics that align with test goals & can detect
anomalies are selected.
Conversions | the primary metric of a test - measures
what you are trying to accomplish (e.g. orders, leads,
sign-ups)
Conversion
Secondary
Interactions
Tactical Engagement
Secondary Interactions | measure intermediate steps
towards conversion or post conversion (e.g. cart
additions, lead quality)
Tactical Engagement | these metrics monitor conversion
related values (e.g. units per order, average order value,
revenue, profit)
20. When not to test.
Your site design isn’t stable
Don’t know what they are trying to achieve
Don’t have the resources aligned in order to run a testing program
Aren’t mature enough to execute a complex test
22. Testing as a precursor to personalization
We don’t have to treat all users the same.
If the retailor could predict the price the customers were willing to pay, then lower friction and high
conversion value. An optimization program could have produced $75 in revenue versus $50
User Example: Coupons for everyone
Same shirt, but two customers are willing to pay different prices…
A retailer generally releases
coupons for 50% off retail
price
Customer B: $25
Customer A: $50 (retail price)
Customer B pays $25
Customer A pays $25
23. Personalization modeling
Rules based
personalization
Build out individual
personas
Personalization based
on visit history
Personalization based on
integration of online/offline data
Customized Personalization Predictive Personalization
Using 1st hit
attributes
Triggers can change
experience on the fly
Customers fit into
audiences
Customers fit into behavioral
segments
Advanced targeting
Look-a-like Models
Segments fit onto
customers
Machine learning
24. Client Example |
A/B Testing
Client needed to test the size of tiles when page
viewed on mobile phones
Saw a 36% increase in click-thru rate with
wider tile format
A test was run on both layouts