When we started with analytics department, we had a couple of AB tests a month and each took us about a week of work. With more apps and increased interest in data, we started running more and more tests, coming to a couple a week. A lot of metrics were the same in all reports (retention, engagement, revenue data) and so we decided it would make sense to automatise this and have hours for digging deeper into other metrics or even do other analyses, such as churn prediction and user behaviour. With this came additional positive thing which is we can now daily see what is happening with tests we are currently running and if there are any bugs, we can spot them sooner than before.

1. Automate
everyday
work
Sara Krk
Belgrade, 19th September 2018
© Outfit7 Limited 2010 – 2018

2. What is the problem?
• AB test reports consisting of same metrics
• tasks performed on a daily basis
• any other manual work that needs to be
done multiple times

3. How to solve the problem?
1. choose what to automate
2. everyone on the team can update the code
3. create daily job to execute the code
4. create nice UI for results
5. create easy setup environment

4. choose what to automate
everyone on the team can update the code
create daily job to execute the code
create nice UI for results
create easy setup environment
• which metrics from AB test reports:
• retention
• engagement
• revenue metrics
• add super-useful features – for us: filters
• create user aggregates to decrease cost of querying

5. choose what to automate
everyone on the team can update the code
create daily job to execute the code
create nice UI for results
create easy setup environment
• filters:
• select for which users you want results – “android only”
• applies to every plot
• see effect for different clusters

6. choose what to automate
everyone on the team can update the code
create daily job to execute the code
create nice UI for results
create easy setup environment
• User aggregates
• everyday
• one line for one user
• all info that you need on a daily basis
• add info for AB tests if needed

7. choose what to automate
everyone on the team can update the code
create daily job to execute the code
create nice UI for results
create easy setup environment
• same code for everybody – trouble?
• data manipulation in bigQuery – faster execution
• plotting in ggplot in R – hassle free and very neat

8. choose what to automate
everyone on the team can update the code
create daily job to execute the code
create nice UI for results
create easy setup environment
• we are currently using jupyter notebooks
• for running scripts: nbconvert library in Python
cron job
selecting AB
tests
running AB
test scripts
saving plots

9. choose what to automate
everyone on the team can update the code
create daily job to execute the code
create nice UI for results
create easy setup environment
• we chose shiny – we use it for all kinds of
dashboards
• select app, test and sub-test
• split plots into different categories

10. choose what to automate
everyone on the team can update the code
create daily job to execute the code
create nice UI for results
create easy setup environment
• Google sheet with columns
• bigQuery links Google sheet with a BQ table – you can query it
• use special notebook to run selected test

11. Problem solved!
TOP
COMPANIES
TOP MOBILE
GAMES

12. Thank you for your attention!

13. TOP
COMPANIES
TOP MOBILE
GAMESThis content and all attachments to this content (“Content”) are copyright of Outfit7 Limited - ©
2010–2018 Outfit7 Limited. All Rights Reserved. Unauthorized use is prohibited. Any
redistribution or reproduction of part or all of the Content in any form is prohibited. You may
not, except with our express written permission, distribute or commercially exploit the Content,
nor may you transmit or store it in any other website or other form of electronic retrieval
system.
DISCLAIMER