measuring community health dario taraborelli • aaron ......dario taraborelli • aaron halfaker...
TRANSCRIPT
![Page 1: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/1.jpg)
Measuring Community HealthVital Signs for Wikimedia Projects
Dario Taraborelli • Aaron Halfaker • Dan AndreescuWikimania 2014, London
![Page 2: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/2.jpg)
![Page 3: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/3.jpg)
metrics != science
![Page 4: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/4.jpg)
metrics as filters
![Page 5: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/5.jpg)
summer 2013
![Page 6: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/6.jpg)
cohort-level metrics
![Page 7: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/7.jpg)
cohort-level metrics project-level metrics
![Page 8: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/8.jpg)
project-level metrics
![Page 9: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/9.jpg)
project-level metrics
![Page 10: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/10.jpg)
Existing data sources
wikistats report card
http://stats.wikimedia.org http://reportcard.wmflabs.org/
![Page 11: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/11.jpg)
ENWIKI New Editors / day 1D: 21% 30D: 18% YTD: 20%
Vital Signs
• granular measurements of new user engagement, community size, and content growth • aggregated daily / weekly / monthly• for every single Wikimedia project• visualizations + raw datahttps://www.mediawiki.org/wiki/Analytics/Epics/Editor_Engagement_Vital_Signs
02/01: 1240•
summer 2014
![Page 12: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/12.jpg)
4 categories of metrics
New users Community Content Curation
Newly registered users
New editors
New active editors
Productive new editors
Surviving new editors
Surviving new active editors
Active editors
Recurring old active editors
Re-activated editors
Unique editors
Unique anonymous editors
Unique editing bots
Unique page creators
Unique media creators
Edits
Anonymous edits
Bot edits
Pages created
Media uploaded
Pages deleted
Pages protected
Pages moved
Reverts
https://meta.wikimedia.org/wiki/Research:Metrics_standardization
![Page 13: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/13.jpg)
metric definitions
![Page 14: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/14.jpg)
RelevantMeasure quantities that describe important phenomena
ReplicableMake research easily replicable and verifiable
Transparent Provide formal specifications, remove ambiguity
ConsistentReplace proprietary, ad-hoc metric definitions; compare apples to apples
RobustMake metrics replicable via multiple data sources at any point in time
GranularComputable at different time scales
Principles
![Page 15: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/15.jpg)
Anatomy of a metric 1. specification
![Page 16: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/16.jpg)
Anatomy of a metric 2. visualizations
registration
time
Activation Trial Survival
![Page 17: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/17.jpg)
Anatomy of a metric 2. visualizations
New editor
Productive new editor
Surviving new editor
![Page 18: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/18.jpg)
Anatomy of a metric 3. discussion
![Page 19: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/19.jpg)
Anatomy of a metric 4. sensitivity analysis
![Page 20: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/20.jpg)
Sensitivity analysis
https://meta.wikimedia.org/wiki/Research:Productive_new_editor
Does new editor productivity vary when we measure it over the first day or the first week?
![Page 21: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/21.jpg)
Sensitivity analysis
https://meta.wikimedia.org/wiki/Research:New_editor
Should we define new editors based on activity in the article namespace only?
![Page 22: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/22.jpg)
https://meta.wikimedia.org/wiki/Research:Rolling_monthly_active_editor
What segments of the population of a project drive total active editor numbers
Anatomy of a metric 5. segmentation
![Page 23: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/23.jpg)
Use cases
![Page 24: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/24.jpg)
Use cases
1. Data exploration“Newly registered users on German and Dutch Wikipedia have a higher activation rate than newbies who join the English Wikipedia”
“Spanish Wikipedia adds every day twice as many new editors than German Wikipedia, despite having only half its new user activation rate”
2. Natural experiments
“A change in abuse filter rules on the Italian Wikipedia significantly increased new editor survival”
![Page 25: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/25.jpg)
Use cases
3. Projections and target setting“To stop the active editor decline in the English Wikipedia, we should increase the retention of existing users by 87% or increase the activation of new editors by 23%”
25.8%
37.3%
33.7%
3.1%
New active
Surviving new active
Recurring old active
Re-activated
![Page 26: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/26.jpg)
What we’re building
![Page 27: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/27.jpg)
Data generation
DEMO
https://metrics-staging.wmflabs.org/static/public/dash/
![Page 28: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/28.jpg)
Information architecture
http://pauginer.github.io/prototypes/analytics-dashboard/
![Page 29: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/29.jpg)
Beyond basic measurements
Beyond signals based on simple edit counts: quality, value added
Better segmenting of the editor population (classifying edit types)
Readership metrics (unique visitors, pageviews)
Feedback and evaluation
![Page 30: Measuring Community Health Dario Taraborelli • Aaron ......Dario Taraborelli • Aaron Halfaker • Dan Andreescu Wikimania 2014, London. metrics != science. metrics as filters](https://reader034.vdocument.in/reader034/viewer/2022043008/5f96d6f4d9339176797026a3/html5/thumbnails/30.jpg)
Questions?
[email protected] [[User:DarTar]] @[email protected] [[User:Halfak (WMF)]] @[email protected] [[User:DAndreescu]] @DanAndreescu
Read morehttps://meta.wikimedia.org/wiki/Research:Metrics_standardization
Image credits
Tim Sheerman-Chase. Fair, Barometer Detail (CC BY) https://www.flickr.com/photos/68932647@N00/7615682432/ Mark Dumont. Mad Science (CC BY) https://www.flickr.com/photos/23661161@N02/5826590955/ Future scientist experiments on sister http://www.gifbay.com/gif/future_scientist_experiments_on_sister-117763/ Joaquim Alves Gaspar. Vernier Caliper (CC BY SA) https://commons.wikimedia.org/wiki/File:Vernier_caliper.png