The Research Excellence Framework

Data and Audit

May 2012

Presentation outline• Citation data

• The submission system

• Contextual data

• REF 4 data

• Research doctoral degrees awarded

• Research income data

• Returning data

• Audit and data verification procedures

Citation data

“Those panels using citation data will do sowithin the framework set out in ‘guidance onsubmissions’ (paragraphs 131-136). In particular,they will consider the number of times an output hasbeen cited as additional information about theacademic significance of submitted outputs. Panelswill continue to rely on expert review as the primarymeans of assessing outputs, in order to reachrounded judgements about the full range ofassessment criteria (‘originality, significance andrigour’).”

(Panel Criteria and Working Methods, Part 1 paragraph 51)

Citation data

Sub-panel 1: Clinical MedicineSub-panel 2: Public Health, Health Services and Primary CareSub-panel 3: Allied Health Professions, Dentistry, Nursing and PharmacySub-panel 4: Psychology, Psychiatry and NeuroscienceSub-panel 5: Biological SciencesSub-panel 6: Agriculture, Veterinary, and Food ScienceSub panel 7: Earth Systems and Environmental SciencesSub-panel 8: ChemistrySub-panel 9: PhysicsSub-panel 11: Computer Science and InformaticsSub-panel 18: Economics and Econometrics

• Will only be used by the following sub-panels:

Obtaining Citation data

• Items matched to Scopus via the submission system

• Via the web interface

• When entering individual outputs

• For whole submissions

• On bulk import

• Support from Elsevier available

Obtaining Citation data

Obtaining Citation data

Obtaining Citation data

Support from Scopus

• Elsevier will provide technical support on the matching

process during the submissions phase

• Initial means of support will be by email

• We will publish further details of support arrangements

before the submissions period

Contextual citation data

• Sub-panels using citation data will also receive contextual

data about citation rates to help inform their interpretation of

the citation counts.

Contextual citation data

• Scopus have assigned journals to ~350 subject categories

• Contextual data will be provided based on these subject

categories, for each year

• The average number of times items were cited

• The number of citations required to be in the top 1%,

5%, 10% and 25% of papers in that category

• The data will also be provided at the level of UOA by

combining subject categories, after the submission system

closes.

• Data will be provided for the years 2008 to 2012

Contextual citation data (2)

• HEIs will be given the same data when the submissions

system opens, except:

• Data will be provided for the years 2007 to 2011

• UOA level data cannot be provided

• Unable to assign Scopus subject categories to UOAs

until all REF data has been collected.

• Data will be available as an Excel workbook to download

via the submission system.

• Data will be available shortly after the submission system

opens

• HEIs will have access to the same data as the panels after

the close of the submissions system

Questions?

Environment data • All submissions to include data on:

- Research doctoral degrees awarded (REF4a)

- Research income (REF4b)

- Research income in-kind (REF4c)

• Definitions are aligned with HESA returns

• Data relates to whole unit of assessment - not just submitted staff

• Sub-panels 8, 9, 19, 25 and 26 request specific additional data, to be included within the environment template (REF5)

Environment data (2)

HESA data will be presented to institutions at three stages:

- Data for 2008-09, 2009-10 and 2010-11 in May 2012

- Excel workbook containing data

- Guidance note explaining how the data were extracted

- Data for 2011-12 in April 2013

- Doctoral degree awarded data for 2012-13 will be available from HESA when the 2012-13 HESA student record is submitted

Research doctoral degrees awarded

• Taken from the HESA student record

• All instances who successfully complete with:

• D00: “Doctorate degree obtained primarily through

advanced supervised research written up as a

thesis/dissertation”

• D01: “New Route PhD”

• An award is only counted the first year it is given to an

instance

• An award is not counted if the instance is not closed.

Subject mapping

• Awards data are mapped to:

• RAE 2008 UOAs

• Cost Centres

• Data linked between years back to 2007-08

• To map awards to cost centres and RAE UOAs

• To detect instances awarded multiple times

Research income

• Taken from the HESA Finance Statistics Return (FSR)

• FSR Table 5b: External research grants and contracts

• Academic departments (Head 1)

• Academic services (Head 2)

• Less any income passed on to other institutions or

organisations as part of a collaborative project or

subcontracted work (Head 5)

Research income-in-kind• Data on research income in kind will be provided to

institutions from:

• The UK Research Councils

• Health research funding bodies:

• The National Institute of Health Research (NIHR)

• The Chief Scientist Office, Scottish Government

Health and Social Care Directorates

• The Welsh Government’s National Institute for

Social Care and Health Research (NISCHR)

• The Health and Social Care Research &

Development (HSC R&D), Northern Ireland.

Income-in-kind data template

Institution Department Researcher title

Researcher surname

Researcher initials Facility

Income-in-kind (£ by academic year)

2008-09 2009-10 2010-11 2011-12 2012-13

                     

                     

                     

                     

                     

                     

Returning data

• Data should be mapped to REF UOAs

• Research degrees awarded for all students in those UOAs

should be returned, not just those with supervisors returned

to the REF

• Income associated with all staff in those UOAs should be

returned, not just those who are submitted

• REF4 data should align with corresponding HESA, RCUK,

health research funding bodies data as closely as possible

• Validation checks will be run within the submission system

software before this data can be submitted

Returning data (2)• The total number of research degrees awarded submitted by the

HEI as a whole should not exceed the total number of degrees awarded provided in the HESA data by:• More than 20 and 5% over the period 2008-09 to 2011-12• More 10 and 10% in any AY2008-09 to 2011-12

• The total amount of research income submitted by the HEI as a whole should not exceed the total amount of income provided in the HESA data by• More than £200,000 and 5% over the period 2008-09 to 2011-12• More than £100,000 and 10% in any AY2008-09 to 2011-12.• More than £100,000 and 10% in any category of income over the

period 2008-09 to 2011 12• The total amount of research income in kind (REF4c) submitted by the

HEI as a whole should not exceed the total amount of income in kind provided by the research councils and health agencies by:• More than £100,000 and 5% over the period 2008-09 to 2011‑12• More than £50,000 and 10% in any AY2008-09 to 2011-12

Data adjustments

• Institutions should contact the REF team as soon as

possible where they propose to return environment data

that falls beyond the set tolerance limits.

• Requests should be directed to info@ref.ac.uk

• All requests must be submitted by 1 June 2013 at the

very latest.

Questions?

Audit & data verification procedures

• Audit and data verification of REF submissions is necessary to:

• support principles of fairness and to adhere to good regulation

• provide sufficient confidence in the quality and reliability of data and assessment outcomes

• Any part of a REF submission can be audited and institutions should maintain accurate records in order to verify any information provided as part of their submission.

• Further guidance on audit and data verification will be published in June 2012

• Audit procedures will involve the following processes:• Routine checks• Structured and targeted audits• Systematic data comparisons• Panel instigated audits

Audit – Staff eligibility

• A sample of randomly selected Category A and C staff will

be audited in 1 - 4 units of assessment for every institution

• Institutions will be contacted promptly following the

submissions deadline to request evidence of staff eligibility.

• Evidence may include employment contracts or other HR

personnel records

Audit – Individual staff circumstances• Submitted details for every individual with a clearly defined

reduction in outputs will be examined to check that the

reductions has been calculated correctly. Further information

may be requested.

• All complex circumstances will be reviewed by EDAP and

further information may be requested to inform judgements.

• A sample of staff with individual circumstances will be audited

from each institution to verify the circumstances declared.

• Evidence will be largely based on:• Staff contracts• HR confirmation of leave• Confirmation that appropriate evidence has been

consulted

Audit – Research outputs

• Audit of research outputs will include a systematic data

comparison of:• Outputs that may have been returned to RAE2008• Outputs that are listed more than twice within a

submission

• Panels may also raise audit queries on research outputs,

these will be followed up, where necessary, by the REF

audit team

Audit - Impact

• Routine audit will be carried out on a sample of impacts

from each HEI.

• Queries will be informed by advice from the panels and

are expected to peak in March – June 2014

• Queries are expected to relate to:

• Claims made that can be corroborated through independent sources

• Eligibility of underpinning research

Audit – Environment

• Targeted audit will be undertaken where:

• The environment data (REF4a/b/c) is close to the

system limits

OR

• Where a comparison at subject level or cost centre

suggests discrepancies in the allocation of data to

REF UOAs

• If audited, HEIs will need to demonstrate how any additional

data is eligible; and/or justify how this has been allocated to

REF UOAs.

Audit – next steps

• Guidance on audit and data verification is due to be

finalised and published on the REF web-site in June

Questions?