This document discusses methods for evaluating the long-term impact of changes on audience reach through digital analytics. It proposes looking at the daily share of returning visitors to test options against a baseline. While traditional A/B testing looks at immediate metrics like page views and clicks, this approach combines audience metrics to evaluate the mid-term effect on audience reach. The document outlines using a statistical significance test to determine if results are due to the product change or just noise. Tracking returning visitor share over time can provide insights about how changes impact audience development that typical A/B tests may miss.

1. evaluating long-term
audience effect
how to know the impact of changes
on audience reach

2. The speakers
Dmitry Kharitonov
Digital Director, RBC
Andrey Sverdlov
Digital Analytics
Consultant, AT Internet

3. what’s the problem?

4. The problem
• You’re a digital publisher having earnings
from advertising.
• You’re addicted to audience volume in terms
of reach and page views.
• One day you’re going to update the site
significantly.

5. How can you be sure?
• if you change your site will it affect the
audience metrics and how?

6. How can you trust the data?
• if the difference is based on product change
or just noise?

7. Common approach – A/B test
There’s number of ways to test direct response
as page views per visit, CTR, clicks and other
actions.
We have a lack of tools to measure impact on
audience reach.

8. towards the solution
with Andrey

9. Audience-centric approach in testing
Key idea: look at development of daily share of
returning visitors.
Spoiler: share of visitors of tested option B as a
metric shows this development.

10. Idea background
Day
Audience A total
Audience B total
new
new
returning
9 500
returning
Share of B
new
500
returning
5.00%
1
1 200
8 300
63
9 300
437
4.98%
400
5.00%
4.12%
2
1 100
8 200
58
9 700
342
5.01%
700
4.00%
6.73%
3
1 500
8 200
79
621
5.00%
7.04%
New visitors always have base proportion, returning shares play a key role.

11. Idea background
Day
Audience A total
Audience B total
new
new
returning
9 500
returning
Share of B
new
500
returning
5.00%
1
1 200
8 300
63
9 300
437
4.98%
400
5.00%
4.12%
2
1 100
8 200
58
9 700
342
5.01%
700
4.00%
6.73%
3
1 500
8 200
79
621
5.00%
7.04%
New visitors always have base proportion, returning shares play key role.

12. mathematics behind
you can’t trust the numbers without it

13. Fundamentals
Share of B is a random value.
Share_of_B = B / (B + A) / Base_proportion – 100%
Example:
Day 1
B = 500
B+A = 10 000 (total)
Base proportion = 5%
Share_of_B = 500 / 10 000 / 5% – 100%
= 0%
Day 2
B = 400
B+A = 9 700 (total)
Base proportion = 5%
Share_of_B = 400/ 9 700 / 5% – 100%
= -17,6%

14. Fundamentals
Share of B is a random number.
and We
don’t know it’s distribution function.
Still we can apply methods of statistical
analysis.

15. Hypothesis
Zero hypothesis: tested option B is the same as
A = no effect on reach.
Right-hand alternative: option B attracts more
returning visitors than A = B is better than A.

16. Metric to test hypothesis
Daily share of visitors that have seen option B
10,0%
8,0%
6,0%
4,0%
2,0%
0,0%
1
3
5
7
9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75
-2,0%
-4,0%
Audience
7 Moy. mobile sur pér. (Audience)

17. Hypothesis in terms of metric
Zero hypothesis: proportion of A and B has not
changed = no effect on reach.
Right-hand hypothesis: proportion has skewed
to option B = B is better than A.

18. What we are testing now?
We’re testing A and B against 1 number –
share of B.
We can make decision based on this number
combined with range of other metrics: page
views number, clicks on links, revenue, CTR,
etc.

19. we’ve got data, so what?

20. Can we trust the numbers?
Ok, now we know B is X% better than A.
But can we trust this value?
We need to prove statistical significance of
result.

21. Statistical significance proof
Build sample distribution.
It’s very much like normal distribution.
If result is out of 3*sigma interval
it is significant.
Otherwise it’s just noise.
We can also use Student’s T-test.

22. Statistical significance proof: bonus
Empirical Rambler values:
significant results start from 2-3% in audience
reach.
Disclaimer: values may vary for your sites.
It is not possible to calculate these border
values with only AT Internet tools.

23. Technical means

24. What we used
A/B test – nginx w/testing module
new visitors have constant proportion
returning visitors see what they have
seen first time
“murmurhash32” algorithm
Measurement – ATI Multivariate Testing tag
Proof – own server logs and processing

25. Conclusion

26. Pros&Cons
+ can understand an impact on an audience reach
+ can set up GO / NO GO constraints for changes
+ can mix audience and direct response metrics in tests
- no out-of-the-box tool
- complex computations
- mathematics involved

27. Outcome




Evaluation of immediate effect is not enough for
publishers.
Traditional test metrics (page views, clicks, CTR, etc.)
could be combined with audience metrics.
It makes possible to evaluate test effect on audience
reach in mid-term.
Not the easiest approach, still very useful.

28. Example of metrics mix
Audience reach and Impressions
10,0%
8,0%
6,0%
4,0%
2,0%
0,0%
1
3
5
7
9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75
-2,0%
-4,0%
-6,0%
-8,0%
-10,0%
Audience
Impressions
7 Moy. mobile sur pér. (Audience)
7 Moy. mobile sur pér. (Impressions)

29. Q&A
Dmitry, Andrey