how to know the impact of changes on audience reach - user and partner conference 2013
DESCRIPTION
Evaluating long-term audience effect - how to know the impact of changes on audience reachTRANSCRIPT
![Page 1: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/1.jpg)
evaluating long-term
audience effect
how to know the impact of changes
on audience reach
![Page 2: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/2.jpg)
The speakers
Dmitry Kharitonov Digital Director, RBC
Andrey Sverdlov Digital Analytics
Consultant, AT Internet
![Page 3: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/3.jpg)
what’s the problem?
![Page 4: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/4.jpg)
The problem
• You’re a digital publisher having earnings
from advertising.
• You’re addicted to audience volume in terms
of reach and page views.
• One day you’re going to update the site
significantly.
![Page 5: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/5.jpg)
• if you change your site will it affect the
audience metrics and how?
How can you be sure?
![Page 6: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/6.jpg)
• if the difference is based on product change
or just noise?
How can you trust the data?
![Page 7: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/7.jpg)
Common approach – A/B test
There’s number of ways to test direct response
as page views per visit, CTR, clicks and other
actions.
We have a lack of tools to measure impact on
audience reach.
![Page 8: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/8.jpg)
towards the solution
with Andrey
![Page 9: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/9.jpg)
Audience-centric approach in testing
Key idea: look at development of daily share of
returning visitors.
Spoiler: share of visitors of tested option B as a
metric shows this development.
![Page 10: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/10.jpg)
Idea background
New visitors always have base proportion, returning shares play a key role.
Day Audience A total Audience B total Share of B
new returning new returning new returning
1 9 500 500 5.00%
1 200 8 300 63 437 4.98% 5.00%
2 9 300 400 4.12%
1 100 8 200 58 342 5.01% 4.00%
3 9 700 700 6.73%
1 500 8 200 79 621 5.00% 7.04%
![Page 11: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/11.jpg)
Idea background
New visitors always have base proportion, returning shares play key role.
Day Audience A total Audience B total Share of B
new returning new returning new returning
1 9 500 500 5.00%
1 200 8 300 63 437 4.98% 5.00%
2 9 300 400 4.12%
1 100 8 200 58 342 5.01% 4.00%
3 9 700 700 6.73%
1 500 8 200 79 621 5.00% 7.04%
![Page 12: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/12.jpg)
mathematics behind
you can’t trust the numbers without it
![Page 13: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/13.jpg)
Fundamentals
Share of B is a random value.
Share_of_B = B / (B + A) / Base_proportion – 100%
Example:
Day 1
B = 500
B+A = 10 000 (total)
Base proportion = 5%
Share_of_B = 500 / 10 000 / 5% – 100%
= 0%
Day 2
B = 400
B+A = 9 700 (total)
Base proportion = 5%
Share_of_B = 400/ 9 700 / 5% – 100%
= -17,6%
![Page 14: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/14.jpg)
Fundamentals
Share of B is a random number.
and We don’t know it’s distribution function.
Still we can apply methods of statistical
analysis.
![Page 15: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/15.jpg)
Hypothesis
Zero hypothesis: tested option B is the same as
A = no effect on reach.
Right-hand alternative: option B attracts more
returning visitors than A = B is better than A.
![Page 16: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/16.jpg)
Metric to test hypothesis
Daily share of visitors that have seen option B
-4,0%
-2,0%
0,0%
2,0%
4,0%
6,0%
8,0%
10,0%
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75
Audience 7 Moy. mobile sur pér. (Audience)
![Page 17: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/17.jpg)
Zero hypothesis: proportion of A and B has not
changed = no effect on reach.
Right-hand hypothesis: proportion has skewed
to option B = B is better than A.
Hypothesis in terms of metric
![Page 18: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/18.jpg)
What we are testing now?
We’re testing A and B against 1 number –
share of B.
We can make decision based on this number
combined with range of other metrics: page
views number, clicks on links, revenue, CTR,
etc.
![Page 19: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/19.jpg)
we’ve got data, so what?
![Page 20: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/20.jpg)
Can we trust the numbers?
Ok, now we know B is X% better than A.
But can we trust this value?
We need to prove statistical significance of
result.
![Page 21: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/21.jpg)
Statistical significance proof
Build sample distribution.
It’s very much like normal distribution.
If result is out of 3*sigma interval
it is significant.
Otherwise it’s just noise.
We can also use Student’s T-test.
![Page 22: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/22.jpg)
Statistical significance proof: bonus
Empirical Rambler values:
significant results start from 2-3% in audience
reach. Disclaimer: values may vary for your sites.
It is not possible to calculate these border
values with only AT Internet tools.
![Page 23: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/23.jpg)
Technical means
![Page 24: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/24.jpg)
What we used
A/B test – nginx w/testing module
new visitors have constant proportion
returning visitors see what they have seen first time
“murmurhash32” algorithm
Measurement – ATI Multivariate Testing tag
Proof – own server logs and processing
![Page 25: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/25.jpg)
Conclusion
![Page 26: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/26.jpg)
Pros&Cons
+ can understand an impact on an audience reach
+ can set up GO / NO GO constraints for changes
+ can mix audience and direct response metrics in tests
- no out-of-the-box tool
- complex computations
- mathematics involved
![Page 27: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/27.jpg)
Outcome
Evaluation of immediate effect is not enough for
publishers.
Traditional test metrics (page views, clicks, CTR, etc.)
could be combined with audience metrics.
It makes possible to evaluate test effect on audience
reach in mid-term.
Not the easiest approach, still very useful.
![Page 28: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/28.jpg)
Example of metrics mix
Audience reach and Impressions
-10,0%
-8,0%
-6,0%
-4,0%
-2,0%
0,0%
2,0%
4,0%
6,0%
8,0%
10,0%
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75
Audience Impressions 7 Moy. mobile sur pér. (Audience) 7 Moy. mobile sur pér. (Impressions)
![Page 29: How to know the impact of changes on audience reach - User and partner conference 2013](https://reader031.vdocument.in/reader031/viewer/2022020207/558198a5d8b42a417f8b5106/html5/thumbnails/29.jpg)
Q&A
Dmitry, Andrey