ref 2021 briefing event follow us on twitter june 2019 · judgements about the full range of...
TRANSCRIPT
Consultation
• 294 respondents
• Views sought on clarity – broad agreement (70% +) across most areas
• Some areas showing well over 80% agreement
• Not always good indicator of feedback!
• Mixed views on key issues
Key changesGuidance on submissions• Approach to taking account of staff circumstances• Additional clarification on research independence and
significant responsibility• Eligibility of staff in non-UK based units• Eligibility of outputs by former staff made redundant• Version of output returned (former staff)• Returning co-authored outputs more than once in same
submission• Tolerance band for open access• Word limit for institutional level environment statement
Key changes
Panel criteria• Research independence• Co-authorship statements• Double weighting• Continued impact case studies• Increased focus on equality and diversity in environment
Staff circumstances
• Ensure recognition of the effect circumstances can have upon an individual• researcher’s productivity.• Create the right incentives for HEIs to support staff with circumstances (and
avoid introducing negative incentives, for example around recruitment).• Recognise the potential disparity in the available output pool for units in
particular contexts, for example where there are high proportions of staff with circumstances.
• Maintain the integrity of exercise – both in supporting equality and diversity and ensuring the credibility of assessment process.
• Focus on HEIs’ responsibility to support staff with circumstances• Empower individuals to declare their circumstances
Staff circumstances – reductions
Reduction in unit’s required no. of outputs
•Optional reduction in output requirement where staff have not been able to research productively due to individual circumstances. Defined reductions for:
• Early career researchers
• Secondments/career breaks
• Family-related leave
• Clinical lectureships
•Plus circumstances equivalent to absence that require a judgement
Removal of minimum of one output
•Exceptional circumstances where the individual has not been able to produce an output
•Individual may be returned without min. of one output
• Unit’s output requirement will be further reduced by one
• Up to 1.5 output reduction per staff member• More simplified approach to submitting information – requests at unit, not individual, level
Independent researchers• ‘An individual who undertakes self-directed research, rather than carrying out another
individual’s research programme’
• Research assistants / associates not normally eligible
• GOS includes generic indicators, including: • Being named as principal investigator
• Holding an independently won, competitively awarded fellowship where research independence is a requirement. (List at www.ref.ac.uk/guidance)
• Leading a research group or a substantial or specialised work package. leading a research group or a substantial or specialised work package.
Main Panels C and D supplementary criteria – independent researchers
1. In addition to the generic criteria specified in the ‘Guidance on submissions’, Main Panels
C and D also consider that the following attributes may generally indicate research independence
in their disciplines:
• Being named as a Co-I on an externally funded research grant/award.
• Having significant input into the design, conduct and interpretation of the research.
Significant responsibility
• proportion of time allocated for research, as determined in the context of the institution’s practices and applied in a consistent way
• research allocation in a workload model or equivalent.
Explicit time and resources are made
available…
• eligibility to apply for research funding as the lead or co-applicant
• access to research leave or sabbaticals
• membership of research centres or institutes within the HEI.
…to engage actively in independent
research…
• current research responsibilities as indicated in, for example, career pathways or stated objectives
• expectations of research by role as indicated in, for example, job descriptions and appraisals.
…and that is an expectation of their
job role.
Staff for whom:
Staff in non-UK based units
• Staff employed by the UK HEI and based outside the UK will be eligible if the primary focus of their research activity on the census date is clearly and directly connected to the submitting unit based in the UK.
• HEIs should use guidance on demonstrating a substantive connection to help determine whether they are eligible
• Eligible staff should be returned to HESA. REF team is working with HESA to update their guidance.
Outputs – former staff
Redundancy• Funding bodies consulted on whether HEIs should be able to submit the outputs
of former staff who have since been made redundant
• Very mixed responses – opinion split almost 50/50
• Concerns raised by those against the proposal:• Would affect those on fixed-term contracts, incl. ECR research fellowships. Creates
disincentive for HEIs to invest in these fellowships if unable to return outputs.
• Could have a negative effect on future career prospects of those made redundant if they couldn’t point to outputs submitted to REF
• Would require HEIs to share sensitive confidential data on who has been made redundant with academic departments
• Require HEIs to set out approach to submitting outputs of former staff, incl. those made redundant, in the code of practice and the UOA environment statement
Outputs – version
• Proposed in draft guidance that HEIs submitting outputs of former staff must submit the version that was made publicly available when they were employed by that institution
• Feedback from sector and consultation with panels revealed preference for submitting final version
• Concerns that it is not always possible to identify the final version (e.g. for practice outputs)
• HEIs can submit either version made available during employment or final version
Co-authored outputsSubmitting more than once:
• Consulted on whether HEIs should be able to submit an output more than once in a submission to a UOA
• Mixed response from sector – suggested disciplinary differences might be justified
• Main Panels A-C will not permit this
• Main Panel D will permit submission up to two times. Such outputs may make up max. 5% of submission. Cannot be combined with double-weighting
Contribution information:
• Appetite expressed in consultation for alignment between MPs A and B
• Agreed to follow same process and reached compromise position of 15 authors (increase from 6 in MP A and decrease from 25 in MP B)
• Further reflection in MP B revealed that this was only necessary in SP 9 (Physics) –other SPs will not require a contribution statement
Double-weighting outputs• Alignment of criteria in Main Panels C and D
• Removal of reference to the ‘disciplinary norm’
• Expectation that most books will warrant double-weighting BUT this is not automatic
• Suggestion in consultation that HEIs should submit a ranked list of reserve outputs, rather than linking them to specific outputs
• Panels agreed that this was unnecessarily complicated
Outputs – open access• Outputs deposited as soon after
the point of acceptance as possible, and no later than three months after this date from 1 April 2018.
• Deposit exception from 1 April 2018 – outputs remain compliant if they are deposited up to three months after the date of publication.
• Additional flexibility – 5% tolerance band (or one output) per submission to a UOA
Impact Consistency with REF 2014
• Criteria: reach and significance
• Impact remains non-portable
• 2* quality threshold
• Timeframe:
• 1 January 2000 - 31 December 2020 for underpinning research
• 1 August 2013 - 31 July 2020 for impacts
Refinements
• Impact template integrated into Environment statement
• Impact on teaching within (and beyond) own HEI is eligible
• Enhanced clarity on scope of underpinning research – bodies of work
• Guidance on submitting continued impact case studies
• Enhanced guidance on public engagement
Environment template
Sections
a. Unit context, research and impact strategy.
b. People, including:
a. staffing strategy and staff development
b. research students
c. equality and diversity.
c. Income, infrastructure and facilities.
d. Collaboration and contribution to the research base, economy and society.
Information about the unit’s support for impact to be included across the four sections
Main Panel A, B and C will attach equal weighting to each of the four sections. Main Panel D will attach differential weight to sections, to reflect the primary role that people play as the key resource in the arts and humanities.
Institutional level assessment of environment• Institutional-level information will be appended to the UOA-level
environment template and will be taken into account by the sub-panel when assessing the unit-level statement.
• Pilot of the standalone assessment of the discrete institutional-level environment will draw on this submitted information.
• Outcomes from the separate pilot exercise will not be included in REF 2021 but will inform future research assessment.
• Increase in word limit to min. 4,000 words.
• Further guidance and criteria to be published in summer 2019 following consultation.
Timetable
Spring/summer 2019
Submission deadline for codes of practice: 7 June 2019Invitation to request multiple submissions, case studies requiring security clearance, and exceptions to submission for small units Launch beta version of submission system
Autumn 2019Pilot of the REF submission system; Survey of submissions intentions opens; Invitation to submit reduction requests for staff circumstances
December 2019
Survey of submissions intentions completeFinal deadline for requests for multiple submissions, case studies requiring security clearance, and exceptions to submission for small unitsPublication of approved codes of practice
Early/mid 2020
Formal release of the submission systems and technical guidanceInvitation to HEIs to make submissionsAppointment of panel members & assessors for assessment phase Deadline for staff circumstances requests
31 July 2020Census date for staffEnd of assessment period (for impact, environment, and data about research income and research doctoral degrees awarded)
27 November 2020 Closing date for submissions
Further information
• www.ref.ac.uk (includes all relevant documents and FAQs)
• Webinars on codes of practice, EIAs and staff circumstances: https://www.ref.ac.uk/events/codes-of-practice-and-staff-circumstances-webinars/.
• Enquiries from staff at HEIs should be directed to nominated institutional contact (available at www.ref.ac.uk/contact)
• Other enquiries to [email protected]
Citation data
• “Those panels using citation data will do so within the framework set out in ‘Guidance on submissions’ (paragraphs 288 to 292). Panels will continue to rely on expert review as the primary means of assessing outputs, in order to reach rounded judgements about the full range of assessment criteria (‘originality, significance and rigour’). They will also recognise the significance of outputs beyond academia wherever appropriate, and will assess all outputs on an equal basis, regardless of whether or not citation data is available for them.”
• Panel Criteria and Working Methods, paragraph 275
Citation data
• Will only be used by the following sub-panels:• Sub-panel 1: Clinical Medicine
• Sub-panel 2: Public Health, Health Services and Primary Care
• Sub-panel 3: Allied Health Professions, Dentistry, Nursing and Pharmacy
• Sub-panel 4: Psychology, Psychiatry and Neuroscience
• Sub-panel 5: Biological Sciences
• Sub-panel 6: Agriculture, Food and Veterinary Sciences
• Sub panel 7: Earth Systems and Environmental Sciences
• Sub-panel 8: Chemistry
• Sub-panel 9: Physics
• Sub-panel 11: Computer Science and Informatics
• Sub-panel 16: Economics and Econometrics
Obtaining citation data
• Items matched to Web of Science (core) via the submission system
• Via the web interface• When entering individual outputs
• For whole submissions
• On bulk import
• Support from Clarivate available
Obtaining citation dataMatchedNo further action required
Multiple matches foundSelect a result from the list
Too many matchesThe number of search results is too large. Please provide further metadata to narrow the search.
No match foundNo further action required
Journal not indexedThe journal that contains the sought article is not in the database
No match was found that was published between 2014 and 2020If you believe a match should have been found then please contact Clarivate
Support from Clarivate
• Clarivate will provide technical support on the matching process during the submissions phase
• Initial means of support will be by email
• We will publish further details of support arrangements before the submissions period
Contextual citation data
•Sub-panels using citation data will also receive contextual data about citation rates to help inform their interpretation of the citation counts
Contextual citation data
• Clarivate have assigned journals to ~250 subject categories
• Contextual data will be provided based on these subject categories, for each year
• The average number of times items were cited
• The number of citations required to be in the top 1%, 5%, 10%, 25% and 50% of papers in that category
• The data will also be provided at the level of UOA by combining subject categories, after the submission system closes
• Data will be provided for the years 2014 to 2019
Contextual citation data (2)
• HEIs will be given the same data when the submissions system opens, except:
• Data will be provided for the years 2013 to 2018
• UOA level data cannot be provided • Unable to assign Clarivate subject categories to UOAs until all REF data has been
collected.
• Data will be available as an Excel workbook to download via the submission system
• Data will be available shortly after the submission system opens
• HEIs will have access to the same data as the panels after the close of the submissions system
• More guidance to come…
• Questions?
Environment data (1)
• Submissions need to include 7 years of data (2013-14 to 2019-20) on:• REF4a: Number of research doctoral degrees awarded by year
• REF4b: Research income by year and source of income
• REF4c: Research income-in-kind by year
• To help institutions with submissions: • REF team will extract data from HESA records and provide data for REF4a and REF4b up to 2018-19
• UKRI and UK health research funding bodies will provide data for REF4c to both institutions and the REF team
• 2019-20 data will be independently processed by institutions and verified by the REF team following submission
Environment data (2)
• Institutions will get three workbooks:• One from REF team with data for REF4a and REF4b (+ guidance)
• One from UKRI with data for REF4c
• One from the devolved health research funding body with data for REF4c
• The REF team will do two despatches of data and is encouraging UKRI and UK health research funding bodies to do the same :
• September 2019: 2013-14 to 2017-18 data + guidance
• March 2020: 2018-19 data
• HEIs will still have to map data to REF2021 UOAs
• Data will take mergers into account
Research doctoral degrees awarded(1)
• We extract the following from HESA student records :• Number of research doctoral degrees awarded by Cost Centre
• Number of research doctoral degrees awarded by UOAs as for REF2014
• Individualised data
• We identify research doctoral degrees awarded as the instances who:• Successfully completed a course with qualification that is a doctorate degree that meets the
criteria for a research-based higher degree
• Are not incoming or visiting exchange students
• If award is returned twice earliest award counts
• We consider cases of formal concurrent and sequential supervisions for students who started on or after 2015-16 to the extent that this is properly recorded in HESA returns by providers
Research doctoral degrees awarded (2)
• What does this mean in terms of HESA fields?• Instance is closed (ENDDATE is not missing)
• Highest qualification (XQOBTN01) is either D00 “Doctorate degree obtained primarily through advanced supervised research written up as a thesis/dissertation” or D01 “New Route PhD”
• Instance excluded if:• Location of study (LOCSDY) is ‘S’: students studying abroad but spending more than 8 weeks in the UK
and hence returned to HESA
• EXCHANGE is ‘4’ or ‘G’: incoming ERASMUS or other exchange/visiting students
Research doctoral degrees awarded(3)
Unit of Assessment or Cost Centre 2013-14 2014-15 2015-16 2016-17 2017-18
1
2
10
12
15
19
21
…
First despatch from REF team - data sample for research doctoral degrees awarded
Research income (1)• Externally sponsored research carried out by the HE provider or its subsidiary split by
sources of income
• Source: HESA Finance record – Table 5 “Research grants and contracts” (or 5b for 2013-14 and 2014-15)
• To take into account changes across years:• We use template with relevant old heads
• Assessment panels will receive the average over 5 years from 2015-16 onwards
• We will provide:• Academic cost centres’ subtotals (Head 1)
• Total academic services (Head 2)
• Total administration and central services (Head 3) Not in REF2014
• For 2013-14 and 2014-15 also
• Income from restricted held by provider (Head 5) Not in REF2014
• Less any income passed onto as part of a collaborative project or subcontracted work(Head 7)
• For 2015-16 Balance of research related deferred capital and revenue grants (Head 6) Not in REF2014
Cost centreSource of income
2013-14 2014-15 2015-16 2016-17 2017-18
101 1 £ £ £ £ £
130 1 £ £ £ £ £
160 1 £ £ £ £ £
… 1 £ £ £ £ £
201 Total academic services 1 £ £ £ £ £
Total admin & central services 1 £ £ £ £ £
Income from restricted endowments held by the provider
1 £ £ X X X
Balance of research related deferred capital and revenue grants as at 31 July 2015 taken to reserves
1 X X £ X X
Less any income passed on to other providers or organizations as part of a collaborative project or subcontracted work
1 -£ -£ X X X
First despatch from REF team - data sample for one source of research income
Research income (2)
Research income-in-kind (1)
• Estimated value of Research Council facility time allocated through peer review and used by researchers should be returned as research income-in-kind (non monetary income) over the period 2013-14 to 2019-20
• No double counting If the institution is already returning an element of this funding as REF4b it should not return it in REF4c.
• Projects or allocations across multiple institutions funders will provide data only to the lead institution responsibility of the lead institutions to:
• calculate the proportions of the income-in-kind that apply to partner institutions
• inform partner institutions of the proportions
• not include the proportions attributed to other institutions in their own submission
These calculations may be subject to audit to ensure there is no double counting
Research income-in-kind (2)
Institution Dept. PIORCID
(optional)Facility
Grant ref. number
(if available)
RC or health
research funding
body 20
13
-14
20
14
-15
20
15
-16
20
16
-17
20
17
-18
A 2 £ £ £ £ £
A 3 £ £ £ £ £
A 3 £ £ £ £ £
A 3 £ £ £ £ £
…
First despatch from UKRI and UK health research funding bodies - data sample for research income-in-kind
Submitting data
• Data should align with corresponding HESA, UKRI and UK health research funding bodies data as closely as possible
• Validation checks will be run within the submission system software
• Data that at the institutional level falls beyond the tolerance limits will result in a validation failure. Validation errors will need to be cleared before submissions can be finalised.
• Targeted audit may be undertaken where:• Environment data returned exhibits significant variances
• Significant differences at subject level between HESA and submissions indicate discrepancies in way HEI allocated data to UOAs
Tolerance levelsREF4a:
Total number of degrees awarded submitted should not exceed the number of degrees awarded provided in the HESA Student Record by:
• More than 20 awards AND 5% over the period 2013-14 to 2018-19
• More than 10 awards AND 10% in any academic year 2013-14 to 2018-19
REF4b:
Total amount of research income submitted should not exceed HESA equivalent by:
• More than £200,000 AND 5% over the period 2013-14 to 2018-19
• More than £100,000 AND 10% in any academic year 2013-14 to 2018-19
• More than £100,000 AND 10% in any source of income over the period 2013-14 to 2018-19
REF4c:
Amount of RIIK should not exceed that provided by UKRI and UK health research funding bodies by:
• More than £200,000 AND 5% over the period 2013-14 to 2018-19
• More than £100,000 AND 10% in any year over the period 2013-14 to 2018-19
Audit and verification of REF 2021 submission data Follow us on Twitter
@REF_2021
Email us: [email protected]
Audit & data verification procedures
• Audit and data verification of REF submissions is necessary to:
• support principles of fairness and to adhere to good regulation
• provide sufficient confidence in the quality and reliability of data and assessment outcomes
• Any part of a REF submission can be audited and institutions should maintain accurate records in order to verify any information provided as part of their submission.
• Guidance on audit and data verification was published on 11th June
• Audit procedures will involve the following processes:
• Routine checks
• Structured and targeted audits
• Systematic data comparisons
• Panel instigated audits
Staff eligibility/inclusion (REF1)
• A sample of randomly selected Category A staff will be audited in 1 - 4 units of assessment for every institution
• Will included submitted and non-submitted staff
• Institutions will be contacted promptly following the submissions deadline to request evidence of staff eligibility
• … and evidence that non-submitted staff do not have Significant Responsibility for Research
• Evidence may include employment contracts or other HR personnel records
• Can refer to complaints process for codes of practice, if there are serious concerns
Research outputs (REF2)
• Sample audit of outputs to verify the dates the outputs became publicly available
• … and, for outputs attributed to former staff, to verify that the staff member was employed in the submitting unit as category A eligible when the output was published
• Audit of research outputs will include a systematic data comparison of outputs that may have been returned to REF 2014
• Panels may also raise audit queries on research outputs, these will be followed up, where necessary, by the REF audit team
• Open Access… (later)
Impact (REF3)
• Corroborating evidence to be submitted by 29 January 2021• Change from REF 2014
• Routine audit will be carried out on a sample of impact case studies from each HEI
• Queries will be informed by advice from the panels and are expected to peak in March – June 2021
• Queries are expected to relate to:• Claims made that can be corroborated through independent sources
• Eligibility of underpinning research
Environment data (REF4)
• Targeted audit will be undertaken where:• The environment data (REF4a/b/c) is close to the system
limits
OR• Where a comparison at subject level or cost centre suggests
discrepancies in the allocation of data to REF UOAs
• If audited, HEIs will need to demonstrate how any additional data is eligible; and/or justify how this has been allocated to REF UOAs
Environment (REF5)
• We will audit a sample of unit-level environment templates• Based on advice from sub-panels
• We will seek to verify key claims made within the environment template. HEIs will be invited to provide further corroborating evidence for those that are selected for audit.
• EDAP may also refer to an HEI’s code of practice when providing advice to the sub-panels on its assessment of the ‘People’ section in the unit-level environment templates (REF5b) and may use this as a basis for audit, where necessary
Individual staff circumstances (REF6)
• Staff circumstances can reduce the number of outputs required via:• Removal of the minimum-of-one (REF6a)
• Unit reductions (REF6b)
• Either (or both) can be audited to verify that they are based on self-declarations by the staff member(s)
• Some circumstances will involve ‘special category data’ (as defined in the GDPR)• For these, we will ONLY require evidence of the self-declaration to verify the circumstance
• Where the circumstance does not involve ‘special category data’ we may request further evidence:
• ECRs
• Secondments and career breaks
• Junior Clinical Academics
• May also require evidence of the effect (as distinct from the circumstance) when considering removal of the minimum-of-one
Open access• Big change for REF 2021
• 5% tolerance of in-scope outputs that are not compliant and do not have an exception
• Risk-based approach to open access compliance• Rank based on: number of ‘other’ exceptions; whether an OA copy is accessible; whether the
available copy is searchable text; whether deposited within 92 days of publication
• We will ask higher-risk HEIs to send us their OA process, with evidence that they are managing the REF 2021 policy within their process
• Process should address ‘Gold’ open access
• Where controls appear insufficient, we will conduct a targeted sample audit of outputs that are marked as compliant or with an exception
• Sampled outputs will be changed to non-compliant where the HEI cannot demonstrate policy compliance or a valid exception
REF 2021 submission system
Follow us on Twitter @REF_2021
Email us: [email protected]
Submission systemBriefing eventJune 2019
REF 2021 submission system
Agile approach
• Requirements a broken into user stories
• Refined on a fortnightly basis, policy and development team together
• Placed into a backlog ready to go into development
• Fortnightly sprints are run back to back
• Each sprint will have a sprint goal; what we endeavour to deliver within the sprint
• Developed changes are deployed frequently to the test environment and are tested during
the sprint
• Plan to deploy frequently to the test environment which is available to everyone and there
less frequently to live
REF 2021 Submission System
Architecture and platform
• Developed using a JavaScript client framework to allow for a more responsive UI.
• An API to allow for automation of the submission system.
• Cloud hosted services which can be scaled up and down as required.
• Cloud hosting enables us to provide high availability.
REF 2021 submission system
Security
• Design with security in mind
• However, there is one weak link in the chain of any software system
• Unfortunately that is the users
Malicious attacks
• The most vulnerable items of data are your username and password
• Spear-phishing attacks are the most common attacks
• Be alert, report any unusual emails to [email protected]
• Never give your password details to anyone, especially in an email/over the phone
• Penetration testing
REF 2021 submission system
User support and guidance
• User help is embedded in every page in the system; cross refers to GOS and
Panel Criteria
• User guide is available on the REF website
• User support desk is open by phone and email during office hours