Appiterate held a workshop at SAIF Ignition (held at Bangalore on 20th Sep 2013) on A/B testing for mobile apps. It had a look at pre-existing practices in web A/B testing and also extensively covered mobile A/B testing.

1. TEXT
SAIF Ignition
Bangalore
20th Sep 2013
A/B testing for mobile apps

2. WHAT IS A/B TESTING?
Version ‘A’ vs Version ‘B’
That’s how the name A/B testing came about.
How does it work?

3. WEB A/B TESTING - Etsy
•
In-house A/B testing solution
•
Results are sometimes surprising!
INFINITE SCROLL
•
Infinite scroll is presumed to be a better
experience as you see more items
faster
•
Visitors clicked on
•
fewer results (-11%)
•
favourited less (-8%)
•
fewer items purchased (-22%)
Source – Design for continuous experimentation, http://www.slideshare.net/danmckinley/design-for-continuous-experimentation
Infinite
Scroll

4. WEB A/B TESTING
INFINITE SCROLL
•
What went wrong?
•
Changed multiple things at once
•
They should have broken-down the feature
change into two hypothesis testing validations:
•
More items is better
•
Faster is better

5. WEB A/B TESTING
SEARCH DROPDOWN
•
Broke-down the process into testing multiple short measurable tests
•
Decrease the number of items in dropdown
•
Default to ‘all items’
•
Search bars for in-shop and in-page search
•
Rich auto-suggest
•
.. and many more such tests!

6. WEB A/B TESTING
SEARCH DROPDOWN
•
Most people did not notice the major feature changes
•
Search dropdown was subsequently withdrawn
•
Iterative ‘Design – Develop – Measure’
Infinite Scroll
Design
Search Dropdown
Design
Develop Measure
Develop
Measure

7. MOBILE A/B TESTING – Why is it needed?
• Increase conversion, revenue, engagement
metrics
• User-segmented and targeted app feature
launches
• Most mobile apps struggle to make money -
faster iterations will genuinely help developers
in monetizing their apps

8. MOBILE A/B TESTING – Challenges
• Complex coding involved
• Delays in app store updates • slowing the pace of innovation
• Simplicity and ease of web A/B testing not
prevalent on mobiles • thus devoid of 'iterative improvements‘
• key growth drivers on the web

9. WHY A/B TEST MOBILE APPS?
•
Everyone A/B tests in real-life:
•
People try two or more pieces of clothing before buying
•
You have a test drive of two shortlisted cars
•
The HiPPO (the Highest Paid Person’s Opinion) method of
decision making
•
Decisions based on such opinions, hunches, guesses and
conjectures
•
Use data in decision making
•
A/B testing on mobile apps is a great way to increase conversions
and improving your key mobile metrics.

10. WHAT TO A/B TEST IN NATIVE MOBILE APPS?
If it is there in your app, it can be A/B tested!
Example use cases:
• Call to Action buttons (these include buttons
like ‘buy now’, ‘add to cart’, ‘sign up’)
• Set any value to the text, size, colors, fonts
or its position
• App features – Test engagement metrics by
having different number of free levels in
games
• User experience and app navigation sequence
tests
• Entire screen themes/layouts
• Different page opens on tap of a button
• Different ways of social media credentialbased sign-ins

11. CASE STUDY: PRICE COMPARISON APP
The app lets users compare prices of
multiple cross-category products from
online stores.
User
engagement metric
‘review’ button
•Price link – Increases
element (arrow) beside the
price
Variation
A review button with longer
text No visual element
(arrow) beside the price
A review button with shorter
text A small arrow beside the
price
DISTRIBUTION:
50%
50%
Price link clicks on 18.1% of
screen opens
Price link clicks on 22.4% of
screen opens
Hypotheses:
•Review button –
Control
DESCRIPTION:
Aim was to increase
conversion on the following:
VARIATIONS:
revenue in the form of
referral fees.
•A short or a long text on a
•Impact of a small visual
RESULTS:
Longer text button got
Shorter text button got clicked
clicked 1.4% times of screen
1.1% times of screen opens
opens
Price link with a visual
element resulted in an
increase of 23.4% in
revenues for the app!
The longer text on the
review button is better at
driving engagement by
23.1%

12. values
in-code test
(normal)
more
•Integrate SDK
•Pre-specify tests
•Pre-specify attribute
PRE SETUP TIME
less
COMPARING TEST TYPES (1/3)
slow
TIME FOR DEPLOYMENT & ITERATION
fast

13. COMPARING TEST TYPES (2/3)
•Integrate SDK
•Pre-specify tests
•Change attribute values
values
data driven test
in-code test
(normal)
more
•Integrate SDK
•Pre-specify tests
•Pre-specify attribute
PRE SETUP TIME
less
live
slow
TIME FOR DEPLOYMENT & ITERATION
fast

14. COMPARING TEST TYPES (3/3)
•Integrate SDK
•Pre-specify tests
•Change attribute values
PRE SETUP TIME
less
values
wysiwyg
data driven test
in-code test
(normal)
more
•Integrate SDK
•Pre-specify tests
•Pre-specify attribute
•Bypass app store updates
•Drag-n-drop visual interface
•Fast and easy deployment
live
slow
TIME FOR DEPLOYMENT & ITERATION
fast

15. NEW FEATURES
Live Tracking of User’s Interaction
with the App
Content Management System
Feature Launches and Segmented
Rollouts

16. www.appiterate.com
“If genius is, as Edison said, 1% inspiration and 99% perspiration, then perhaps
innovation is 1% invention and 99% iteration.”