dr steven hill, head of policy research, hefce
TRANSCRIPT
The Research Excellence Framework: learning from 2014, planning for the futureSteven HillHead of Research Policy
OpenForum Events, Manchester14 July 2015
@stevenhill
• Evaluation of REF2014
• Future planning
Summary
• Evaluation of REF2014
• Future planning
Summary
REF 2014: Evaluation programme
Evaluation activity
•Two-phase evaluation of impact•Feedback from participating institutions•REF panel feedback•Review of costs, benefit and burden•Multi- and inter-disciplinary research in the UK•Equality and diversity analysis
Wider evidence and information base
•Analysis of impact case studies and database•Independent Review of Metrics•Open access•REF reports (manager’s, panels, EDAP)•Perspectives in media•International Research Systems
REF 2014: Evaluation programme
Evaluation activity
•Two-phase evaluation of impact•Feedback from participating institutions•REF panel feedback•Review of costs, benefit and burden•Multi- and inter-disciplinary research in the UK•Equality and diversity analysis
Wider evidence and information base
•Analysis of impact case studies and database•Independent Review of Metrics•Open access•REF reports (manager’s, panels, EDAP)•Perspectives in media•International Research Systems
REF 2014: Evaluation programme
Evaluation activity
•Two-phase evaluation of impact•Feedback from participating institutions•REF panel feedback•Review of costs, benefit and burden•Multi- and inter-disciplinary research in the UK•Equality and diversity analysis
Wider evidence and information base
•Analysis of impact case studies and database•Independent Review of Metrics•Open access•REF reports (manager’s, panels, EDAP)•Perspectives in media•International Research Systems
• What volume of UK research is interdisciplinary?
• Was interdisciplinary research submitted to REF 2014 in proportion to national volume?
Interdisciplinary research: two questions
• Evaluation of REF2014
• Future planning
Summary
Evidence and evaluation phase
Spring 2015
• Evidence launched to sector at conference on 25 March 2015
Informal dialogue
Spring/summer 2015
• Funding bodies in ‘listening mode’ as consultation proposals developed
Formal consultation
Autumn 2015
• Consultation with sector on proposals for a future exercise
Developing a future exercise
Informal dialogue
• The evidence we have collected through the evaluation of REF 2014 and wider projects has indicated some key issues requiring careful consideration
• Prior to developing proposals for a future exercise, the funding bodies wish to explore these issues in more depth
• REF reports have highlighted that while efforts were made to increase the representativeness of the REF panels, this met with some but limited success
• Recommendation for further work in future to improve the representativeness of panels
Panel recruitment
Key questions:
How could the process better mainstream equality and diversity throughout the stages of recruitment?
What role should nominating bodies play in this process?
Staff selection
• Evaluation work and REF reports have identified issues relating to burden and potential for divisiveness
• Any alternative approaches need to be balanced against their unintended consequences and the differential impact across institutions
• Further workshop to be held to look in detail at options
Key questions:
What are the key advantages of staff selection for institutions? And for individual researchers?
In addition to burden and potential for divisiveness, what are the disadvantages of staff selection for institutions and researchers?
UOA structure
• Revised structure found to be broadly acceptable and increased consistency welcomed
• Need to reflect further on a small number of UOAs, e.g. the best configuration for engineering
Key questions:
In which UOAs did any key challenges arise in terms of preparing submissions?
What was the nature of the challenges, where identified?
Interdisciplinary research• Improved procedures for assessing
interdisciplinary research in REF 2014 – analysis revealed this research found to be as of equally high quality
• Case study analysis showed majority of impacts underpinned by multiple disciplines
• Some perceptions remain that the REF disincentivises or does not take fair account of interdisciplinarity
• Further evidence on interdisciplinary research in the REF will be available in June
Key question:
Should further improvements or incentives be considered in developing future proposals?
Some of the key issues identified through evaluation activity:
• Rules and the representativeness of impacts
• Increased burden and cost
• Impact template
• Ratio of case studies to staff submitted
• Challenges of and panel access to evidence
• Lack of granularity in scoring
• Desire for information ASAP!
ImpactA detailed evaluation was conducted on this new element of the exercise, delivering a range of detailed insights and recommendations. We intend to run a further workshop to consider future proposals.
• Evaluation indicating need for some fundamental questions to be explored around aims and value
• Consideration should be given to the balance of narrative and metric elements in the environment
Environment and metrics
Key questions: What should an environment element
seek to assess? How effectively would the current
process meet this aim?
