A/B Testing is about using data to challenge assumptions and test new ideas, often leading to increased engagement and monetization. In addition to showing you the latest cross-platform testing services at your disposal, we'll discuss the "happy accident" that the Air Patriots team made. It was unexpected and led to an increase in both retention and monetization.

1. 1
@PaulCutsinger
Russ Carroll
A|B Testing

2. 2

3. 3

4. 4
Surface the Value
Image from Diner Dash by PlayFirst
Source: Amazon Appstore, January 2013
+75%
ARPPU

5. 5
Show Users How to “Consume”
Image from Cut the Rope by ZeptoLab
Source: Amazon Appstore, January 2013
+65%
repeat orders

6. 6
Experiment and react quickly
7 day retention
Time
Active
Players

7. 7
A|B Testing

8. 8
This is Air Patriots

9. Case Study: Store Optimization

10. 10
• Goal – Easier to access store content
• Platform – Android & iOS
• Test – Make it possible to unlock content on Main Menu
STORE OPTIMIZATION

11. 11
LOCKED ON MAIN
MENU
UNLOCKABLE >
<

12. 12
STORE OPTIMIZATION
Test Case
Conversion to
Customer
7-Day Retention
Locked on Main Menu
(Control)
– –
Unlockable on Main
Menu
20.93% 4.17%

13. 13
• Goal – Drive players to our other games
• Platform – Android
• Test – Which image is clicked on more
CROSS PROMOTION

14. CROSS PROMOTION

15. 15
CROSS PROMOTION

16. 16
Air Patriots: Cross Promotion
B
A 1.77%
0.93%
ClickThrough

17. 19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. Ads Clicked +1.89% +3.07%
43,120
The Producer’s Perspective
Ads on menu or in game

18. 19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In GameOn MenuNo Ads
The Producer’s Perspective

19. 19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In GameOn MenuNo AdsGoals
The Producer’s Perspective

20. 19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In GameOn MenuNo AdsGoals
The Producer’s Perspective

21. 19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In GameOn MenuNo AdsGoals
The Producer’s Perspective

22. 22
Retrieve a variation
Objective C

23. 23
Record a view and conversion
Objective C
bit.ly/ABTestXPlat
bit.ly/ABTestiOS
bit.ly/ABTestJava
More details at

24. Case Study: Junk Race by GameNug

25. Junk Race Overview

26. 26
Junk Race Overview

27. 27
Tips:
Producers in control: “With two-way services you can request a value from the
server that you control. Two-way services let you make changes to your live game
instantly.
Free: Amazon is the only free two-way service that we have found.”
Multi-variable: “We don't test variables, we test user experiences.”
Watch sample size: “If a couple users can change your conclusion, you need a
larger sample”
Deploy Instantly Cross Platform: “A/B testing on iOS has the advantage of
skipping the week long review period ”
-John Goforth

28. 28
Case Study: Difficulty Level

29. 29Amazon Services Confidential
GAME DIFFICULTY TEST
7 day retention
Time
Active
Players

30. 30Amazon Services Confidential
GAME DIFFICULTY TEST
• Goal – A Dramatic Increase!
• Platform – Android & iOS
• Test – A/B/n Test – 5 Difficulty levels!
OR?

31. 31Amazon Services Confidential
GAME DIFFICULTY TEST

32. 32Amazon Services Confidential
Test 1:
A
Control
B
2% easier
C
4% easier
D
6% easier
E
8% easier
7-Day Retention – tbd tbd tbd tbd
Conversion to customer – -1.71% 16.11% 19.23% 4.84%
GAME DIFFICULTY TEST

33. 33Amazon Services Confidential
GAME DIFFICULTY TEST
• How much would you pay to increase retention by 20%?
• How much time would you invest?
Our Take-Aways!
• Never launch a game in future w/o AB difficulty test!
• Low-hanging fruit!
• Opportunities abound!

34. 34Amazon Services Confidential
A|B Testing
• Monetization – minimize luck and maximize
skill
• Delight your customers
• test customer experiences
• AB testing tools are easy to use
• Cross Platform
• Deploy changes instantly
• Free
• Low-hanging fruit is waiting to be harvested
Paul Cutsinger
Russell Carroll
Find me at:
• @PaulCutsinger
• @AmazonAppDev
• developer.amazon.com/blog
• bit.ly/ABTestXPlat