REF2014

HODOMS Birmingham 8th April

Ann Dowling: Chairman of REF Main Panel B

John Toland: Chairman of REF Sub-Panel B10: Mathematical Sciences

The REF assessment framework

First meeting of Sub-panels

Graeme Rosenberg, REF Manager

The assessment framework

The REF is a single assessment framework, within which panels have flexibility to develop aspects of the criteria and working methods

Key documents

During 2011, our task is to produce two key documents:

• Guidance on submissions

• Panel criteria and working methods

These will set in stone what HEIs need to include in their submissions, and how the panels will assess them

Timetable

The overall assessment framework

65% 15% 20%

Outputs Environment Impact

4* 3* 2* 1* u/c

0 40 40 20 0

4* 3* 2* 1* u/c

20 45 35 0 0

65%

Overall Quality Profile

12

4*

0104236

u/c1*2*3*

4* 3* 2* 1* u/c

12.8 32.5 43.3 11.4 0

15% 20%

Quality Level

% of the submission

Submissions

Each submission to a UOA will include staff selected by the HEI, their research outputs and associated environment and impact:

• Normally one submission by an HEI to a UOA

• Multiple submissions in exceptional cases

• Joint submissions

• Sub-panels to produce UOA descriptors and boundaries

Research staff

Submissions to include:

• Details of eligible staff selected by the HEI

• Up to 4 outputs per selected researcher

• The number of outputs may be reduced without penalty for staff with circumstances that constrained their productivity:

– Early career researchers

– Clearly defined periods of absence

– Complex circumstances

Research outputs

• All research outputs placed in the public domain during the publication period (1 Jan 2008 to 31 Dec 2013) will be eligible

• Outputs from all types of research to be treated equally

• Main panels to expand on:

– The diversity of expected output types

– How co-authorship will be handled

– Requirements for additional information

– Criteria for ‘double-weighting’ outputs of exceptional scale and scope

Citation data

We expect some panels will use citation information to support their review of outputs:

• We will identify a standard approach to making data available to panels and provide guidance:

– Judgements based on expert review; citations only inform ‘academic influence’

– All types of research to be treated equally

– Clear limitations for recently published outputs

• Main panels to:

– Decide whether or not citation data will be used

– Be explicit about any use of citation data

Research environment• A structured template to describe, for example:

– Overview

– Strategy

– People

– Income, infrastructure and facilities

– Collaboration and contribution

• Standard data on research income and research students

• Main panels to explain:

– What kinds of evidence or information are required

– The relative importance of each section

– The approach to forming an environment ‘sub-profile’

Research impact• A broad generic definition of (non academic) impact

• Sub-panels will make judgements based on:

– Case studies detailing specific impacts

– Information about how the unit has supported and enabled impact

• Generic templates and scale for the number of cases

• Main panels to expand on:

– How the criteria (reach and significance) will be applied

– Types of impacts and evidence appropriate to their disciplines

– Suitable evidence of underpinning research quality

– How the ‘impact template’ will influence the sub-profile

Equalities and diversity

We have built on the measures taken in the 2008 RAE:

• A consistent approach to enabling staff with constraining circumstances to submit fewer outputs

• All institutions to adhere to a code of practice on the fair and transparent selection of staff

• Evidence of support for equalities and diversity (within the research environment)

• Analysis of selection rates at sector level

• Panels will be briefed on equalities

• Ongoing advice from an equalities expert panel

Panel working methods

Working methods to explain:

• How the main panels will in practice carry out their roles in providing oversight and governance

• How the sub-panels will in practice carry out the assessment, including:

– The approach to making use of assessors

– The approach to allocating and reviewing

submissions

– Procedural matters

A note on assessors

• For the criteria phase, the funding bodies have appointed panels with broad expertise to develop the criteria; and with numbers in proportion to each panel’s remit

• Additional assessors will be appointed in 2013 where the breadth or depth of a panel’s expertise needs to be extended, to undertake the assessment:

– Assessors (either academic or user) will play a full and equal role to panel members in developing the sub-profiles (either outputs or impact)

– They will be trained and briefed and attend meetings as necessary during the assessment phase