We extensively experiment with all aspects of product at Viki. These experiments take the form of AB testing, cohort analysis and various customisations. We collect and analyse the data on how these experiments affect user behaviour and other metrics. We are always in a continuous product improvement cycle heavily influenced by these experiments and corresponding data. We have developed our own custom Experimentation Framework called Turing for this purpose. In this talk I will explain how we carry out these experiments at Viki. I will also talk about Turing, why and how we built it.

1. How Experiments
drive product at Viki
Ishan Agrawal
Software Engineer, Data
Viki

2. Why Experiment?

3. What is AB testing?

4. How to AB test?

5. What can you test?
Anything that affects user behavior

7. To build or not to build…
More control
Our own Data

8. … in walks Turing
Our own Experimentation framework built in Golang
and Rails!

9. The Challenges
Flexibility
Speed
Scalability
User Partitioning
Stickiness

10. Architecture

11. How to make it fast and
furious!
• Api is in Golang
~ 300 micro seconds
• Store everything in-memory hashes (timely syncs)
• No external calls (not even a redis cache)

12. Advantages of using Go lang
• Run as a binary anywhere!
• Static Type checking
• A good trade-off between performance and
development speed

13. Stickiness and User partitions
without a cache?

14. Points to note
• Server side determination
• Clients log the experiments tied to the current
session.
• A user is under only 1 experiment at a time

15. So how do you create
experiments?
Features and Variations on them

16. Feature and Variations
Example
Feature: Homepage
Variations:
Default : { version : 0 }!
Setting1 : { version : 1 }!
conditions: { country : ‘us’}!
Experiment1 : {version : 2}!
!!
conditions: { country : ‘sg’, start_time : today, end_time : tomorrow}!
!
Experiment2 : {version : 0}!
!
conditions: { country : ‘sg’, start_time : today, end_time : tomorrow}!
!

17. The magic!
• Settings
o Power of customizations
o Different result for everyone
• Country
• Regions
• User type
• Time Ranges
• Application versions

20. Data Collection and
Analysis

21. Making it faster
Each release version is tagged with the enabled
features.

22. Sometimes users define the
product…
• Feature toggling
• Feature tweaking / improvement (with data
feedback)
• Feature customization
• Slow feature roll-outs
• In app messages

23. Some Experiments we
ran…

24. Viki Pass Landing Page(1)

25. Viki Pass Landing Page(2)

26. Viki Pass Landing Page(3)

27. And the winner is…

28. Home page experiments

29. Home page experiments
20%
Conversion Rate

30. Ads optimization

31. Ads optimization

32. Ads
optimization

33. We are still figuring it
out..

34. Thanks!
@ishanagr
ishan@viki.com
Any questions?

35. Appendix

36. In-­‐‑app
messaging

37. AB testing Favorite CTA

38. AB testing like buYon CTA