project snowball – sharing data for cross-institutional benchmarking lisa colledge, anna clements,...
TRANSCRIPT
Project Snowball – sharing data for cross-institutional benchmarking
Lisa Colledge, Anna Clements, M’hamed el Aisati, Scott Rutherford
euroCRIS 2012 with modifications for JISC RIM Meeting Bristol
Jun 28th 2012
The aims of Snowball
• Higher education institutions – Agree a common set of metrics to support institutional
decision making– Reach consensus on standard methodologies for
calculating these metrics– Publish the “recipe book” as open standard definitions
• These metrics will cover the entire landscape of research activity
• These metrics will become global sector standards
The origins of these aims...• Growing recognition of value of metrics to support strategies• Dissatisfaction with the tools available• Frustration over availability of metrics to make sensible measurements
• Institutions and funders should work more collaboratively, and develop stronger relationships with suppliers
• An agreed national framework for data and metric standards is needed• Suppliers should participate in the development of data and metric
standards
BACKGROUND
Joint Imperial-Elsevier JISC-funded study of research information management, available via http://www.projectsnowball.info/
RECOMMENDATIONS
Snowball has evolved from these recommendations
• Agree methodologies for a standard set of metrics to support strategic decision making
• Driven by higher education institutions - with recognised common challenges and goal - working with a supplier (Elsevier), with everyone contributing voluntarily
The goal? To enable cross-institutional benchmarking
Comprehensive metrics landscape
Metrics require institutional, proprietary and third party data
Test 1 to calculate the metrics landscapeApproach: institution and Elsevier contribute data on 10
chemistry researchers as proxy for the whole university
Definitions of metrics
Data availability across landscape
Sensitivity of some data types (next slide)
Researcher-level data
Manual labour in data collection
Data types with high sensitivity
Test 2 of metric calculation feasibility
Definitions of metrics
Data availability across landscape
Sensitivity of some data types
Researcher-level data
Manual labour in data collection
Experts group formed to select and define phase 1 metrics – impactful, do-
able, require data from 3 sources
Data agreement prepared by partnersMost sensitive data types not phase 1
Used minimallyMetric granularity
Institution and Elsevier supply data as close to native as possible
Approach: institution and Elsevier test scalability by contributing data on whole university for a smaller set of metrics
Test 2 of metric calculation feasibility
Metrics require institutional, proprietary and third party data
Test 2 of metric calculation feasibility
HESA cost centreHESA cost centre
HESA FTE research, reserch
& teaching
HESA FTE research, reserch
& teaching
Project Snowball recap
• Driven by sector • Facilitated and supported by Elsevier • Public service The project has demonstrated feasibility of scalably
inputting data from 3 sources to generate metrics and benchmarks
• Institutional • Proprietary • Third party
Next steps
• Publish the phase 1 metrics “recipe book” as open standards – Sep 2012
• Refine phase 1 metrics as global standards, and extend same approach to more metrics
• CERIFy metrics – meeting scheduled Sep 2012• Spread the word – Russell Group, 94 Group,
Vendors, Funders
