intentions and strategies for evaluating the societal impact of research: insights from ref2014...

Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators

Derrick, G.E.

Health Economics Research Group (HERG)

Brunel University London


13 April 2023

2

Introduction

• UK REF2014 – national evaluation process of universities to distribute over £1,952 billion of government funding for research.

• Criteria: 65% Outputs (Peer review of publications), 15% Environment (Esteem) and 20% Impact.

• Impact is defined as research that has had “…an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.” (HEFCE, 2011)

• Peer review only accepted as legitimate IF the results of outcomes are perceived to have been

• Evaluation of criteria conducted by 4 Main Panels divided into “Units of assessment” or disciplines.

• For “Impact” evaluate 4 page case studies “in session” – ex-post impact evaluation

• 3 groups of REF evaluators

1. Output only evaluators (n (interview) =8)

2. Impact only evaluators (n (interview) =9); AND

3. Output and Impact evaluators (n (interview) =47)



13 April 2023

3

The UK REF2014 Evaluation process


MA

IN P

AN

EL

ASub-panel 1 – Clinical Medicine

Sub-panel 2 – Public Health, Health services and Primary care

Sub-panel 3 – Allied Health Professions, Dentistry, Nursing and Pharmacy

Sub-panel 4 – Psychology, Psychiatry and Neuroscience

Sub-panel 5 – Biological Sciences

Sub-panel 6 – Agriculture, Veterinary and Food Science

Evaluation items

Outputs – 50,317

Impact case studies – 1,621


13 April 2023

4Intentions and strategies for evaluating the societal impact of research: Insights from REF2014 evaluators

Sig

nifi

can

ce

Reach

1High significance

Low Reach(¿2* or 3*?)

4Low significance

High Reach(¿2* or 3*?)

2High significance

High Reach(4*)

3Low significance

Low Reach(¿0-1*?)


13 April 2023

5

Methodology – The Treatment• Semi structured pre-evaluation interviews with Main Panel A and its sub panel

evaluators.• Analysed using cognitive-based grounded theory • Questions asked about the evaluators background (most proud), opinion of

impact (definition, what is important), and strategies for overcoming barriers in evaluation (panel roles, attribution, definition differences etc)

EVALUATION PROCESSPre-evaluation

InterviewsPost-evaluation

Interviews

Jan-Mar 2014Dec 2014 – Feb 2015

(1) Definitions(2) Opinions(3) Strategies(4) Intentions(5) Biases

(1) Re-test (1)-(5)(2) Process(3) Conflicts(4) Power roles



13 April 2023

6

Main Panel A and its subpanels

Sub-panel nameNo. of sub-panel members

No. of academic evaluators (AEs) (% AE’s / sub-panel)

No. of research user evaluators (UEs)(% UEs/ sub-panel)

No. of participants

Main Panel 19 14 (73.7%) 5 (26.3%) 8 (42.1%)

Sub-panel 1 – Clinical Medicine

39 32 (82.0%) 7 (18.0%) 10 (25.6%)

Sub-panel 2 – Public Health, Health services & Primary care

27 23 (85.1%) 4 (14.9%) 13 (48.1%)

Sub-panel 3 – Allied Health Professions, Dentistry, Nursing & Pharmacy

51 42 (82.3%) 9 (17.7%) 14 (27.5%)

Sub-panel 4 – Psychology, Psychiatry & Neuroscience

35 28 (80.0%) 7 (20.0%) 9 (25.7%)

Sub-panel 5 – Biological Sciences

35 30 (85.7%) 5 (14.3%) 6 (17.1%)

Sub-panel 6 – Agriculture, Veterinary and Food Science

29 16 (55.1%) 13 (4.9%) 4 (13.8%)

TOTAL 235 185 (78.7%) 50 (23.2%) 62 (28.8%)


Brunel University London 7

ResultsPre-treatment – evaluation without precedence

• Uncertainty of Impact evaluation

LACK OF DEFINITION

• “I´m still not convinced everybody shares exactly the same definition of what constitutes impact or where they place the weight or if it’s impact or isn’t” P3Imp1

LACK OF EXPERIENCE

• “I’m very happy to describe the quality of the research [but] the valuing of impact is something I have no idea about” P0P2 Out-Imp1

• Resort to evaluation “comfort zone” (“what we cut our teeth on”)

USE TRADITIONAL TOOLS FOR RESEARCH EVALUATION RATHER THAN “impact stuff”

“And I don’t believe that we know how to do it- you have to contrast this with the assessment of outputs which is really just reviewing, which is bread and butter stuff for an academic. That’s what we cut our teeth on, that’s what we do every day and so there may be an awful lot of it…but it is just what we do. Whereas this impact stuff we just don’t know. So I feel a little bit nervous about it.” (P0 P2 outimp1)



13 April 2023

8

ResultsExperience with ex-ante impact evaluation

• Previous experience with RCUK “pathways to impact statements” – potential future impact

NOT FORMAL, A “tick box criteria”

“The research council introduced this criteria, it's just a tick box. But it's changing, it's a slow process, you can't instantly get scientists to change their view. So they got this box, you may just tick it. We will tell them why this will have amazing impact on humanity for the rest of eternity, and everybody ticks that and then the REF comes along…. “ P2 OutImp 5

REGARDED AS A “DEAD WEIGHT”

“But that sometimes becomes such a dead weight around the necks of the people making the decisions that it outweighs everything else, including those other sciences that could help.” P2Imp1

EX-POST EVALUATION FOR REF2014 ONE “big experiment”

“I think it's all new territory for all of us, and none of us know – we are going to learn on the job I think.” P4OutImp6



13 April 2023

9

The balance scale of impact evaluation


Tendencies / Decisions

Quality focused Impact focused


13 April 2023

10

The “societal impact” focused evaluator

• Impact was regarded as independent of the research underpinning it.

“The quality of the research has zero role at all in ensuring the impact of the research’” P6 OutImp 2

“The research that is at application is not necessarily getting in these journals but it could be very important to get something into the marketplace.” P0 OutImp3

• Not pre-occupied with the underpinning research

“What maybe a product or an end result of research has different criteria associated with it because what you’re looking for here is a societal change…..…whereas the research….…it’s quite different……[it]…..is all around rigor and methodology and the quality of the idea and making sure that the methods and the quality of the idea match” P2 OutImp7



13 April 2023

Presentation Title 11

Tendencies of the societal-impact focused evaluator• DECISION 1 – Impact is serendipidous / uncontrollable / Messy

• “‘it can often be that coffee you have with somebody at the right moment and the information passing that way’” P1Imp1

• “That kind of linear approach is very, very rare indeed. The way it is instead is that findings accumulate over a period of time and either the weight of evidence in the end wins the day or a moment arrives when the politicians have made up their minds that they want to go in a particular direction, they are looking around for the evidence to support their decision “ P2 Imp1

• DECISION 2 – Value of push factors in achieving impact

• “You can’t assume that it will happen through happenstance, there needs to be some mechanism in place.” P1Imp1

• “Getting from the university stage of research out to the end impact has so many steps in it, not all of which are easy. They require a little effort and somebody championing them from one end or the other” P1 OutImp 4


13 April 2023

Presentation Title 12

• DECISION 3 – Hard vs soft impacts (Impact outcome vs impact journey)

• “If you think of impact as a verb rather than a noun, I think it’s a lot easier to analyse. Impact is the relationships you build. It is the dialog that you have that makes you ask research questions that are subtly different from the ones you would have asked if you hadn’t linked with - whether it's policymakers, whether it's citizens, whether it's industry at the beginning. So impact is not something that you have right at the end. Impact is a relationship and that attitude of mind that you have throughout the research process” P0 OutImp4

• ‘levering it to the next stage’: [whatever] gets the research being taken up and moving it forward, that has to be considered valuable. Maybe the question we should be asking is whether enough effort has gone into that in the past and levering research into its next stage” P0 OutImp6.

• DECISION 4 – Can it be measured?

• “And it’s important not to ignore them because you can’t measure them because sometimes you throw out the most important things because you can’t measure them properly.” P1 OutImp2

• “These [soft] are unquantifiable, so therefore it will be difficult to assess them as impact. And again, it’s subject to imagination, you don’t know how you’ve affected anything until you see the results. So the only time that you know there is an impact is when there is a result. So therefore just talking to people is not an impact." P6 Out2

Tendencies of the societal-impact focused evaluator cont.


13 April 2023

13

The “quality” focused evaluator

• Preoccupation with considering the underpinning research as a proxy for impact

• “I think research will only have an impact if it’s of high quality. I think that quality is the sine qua non of impact” P0 OutImp2

• “So you can get big impacts from very, very bad quality of research, and so that's -- if that's the way you are going to measure impact then you're going to go completely the wrong way.” P0 OutImp2

• Belief in the mantra of excellent impact being dependent on excellent research

• “I think that certainly the quality of the research is an important part. It’s a critical part. You have to have the highest quality research in order for it to be believable and repeatable.” P0 OutImp5

• Bias towards “applied research” as “easy impact”

• “It's [her applied work] is not the research I've done that's the most prestigious in terms of, if you like, the judgment of academia for I've done other research that have been associated with bigger grants, research council grants and if you like higher impact academic products. But [this applied work] is the easiest to demonstrate a real impact.” P3 OutImp2

• “And that the impact case studies might be based on, you know, research is good enough, the equivalent of a two-star, but it won't be four-star research, and but it has an impact because it was applied research.” P2 OutImp 2



Tendencies of the quality-focused evaluator

• DECISION 1 – Research excellence and impact

DANGER OF IMPACT OF BAD RESEARCH

• “It can’t be actually used or believed until its repeated and proven.” P0 OutImp 5

• “I think that poor quality research can only have negative impacts.” P4 OutImp8

HOPE THAT ASSUMPTION OF VALUE OF EXCELLENT RESEARCH IS CORRECT

• “ ….excellent impact should depend on excellent research” P1 OutImp6

• “…you would hope they were synonymous, wouldn’t you?” P3 OutImp 5

• DECISION 2 – Linearity of impact realisation

RATIONALISATION OF IMPACT REALISATION

• “Research was done, showed the benefits of [the evidence], got into the clinical guidelines, and over time you can track the proportion of the relevant professionals who are implementing the better evidence. It’s quite straightforward in fact” P3 OutImp8

• “Impact requires that you generate the evidence and then that you, in turn you get into guidelines and the people start using that information to change their practice’” P4 Out1.



Tendencies of the quality-focused evaluator cont.• DECISION 3 – Pull / Picked up factors

SOMEONE ELSE’s JOB – RESEARCHERS JUST DISSEMINATE

“I mean, I presented this work at the International Embryo Transfer Society meeting, and then afterwards, some people talked to me and what should we do. And I said, well, you’ve just got to purify these hormones better and you have to do this and that and they did it. Then it turned out to be very important for a company called Bioniche because they now have something like 80, 90 percent of the market for the hormones used in superovulation.” P0 OutImp1

• DECISION 4 – Eventually all excellent research has impact

LINK BETWEEN RESEARCH QUALITY AND IMPACT AS WELL AS “ALL IN GOOD TIME”

“….my feeling is that eventually it will [have impact] but it may take a long time.”

• DECISION 5 – Assessing the “right impact” / Lesson from MMR

DANGER OF EVALUATING IMPACT TOO SOON

“The example…is the MMR story, that was poor science. It’s had huge impact, negative impact. Its resulted in lots of morbidity amongst children plus women -- people didn’t get their children vaccinated. The quality of the science was poor, but it had a huge impact in a negative way.” P2 OutImp9



13 April 2023

16

Discussion – what this means for Impact evaluation

• Is there a bias? Depends on definition of bias? Perhaps a conservative bias.

• REF2014 showed a variety of values about what constitutes impact, what will be assessed as part of impact.

• Some of these went beyond the HEFCE or REF2014 definition of impact eg. Impact journey valued, as was the role of public engagement.

• Concern that evaluator´s lack of experience in impact evaluation might do one of 2 things

1. Force them to use traditional measures as proxies for impact (Quality focused evaluator) but also any type of quantitative indicators eg. QALY, deaths/lives saved, % GDP, $$$$

2. Make haphazard judgements about impact being absolute (eg. Give everything 4* because Impact is impact) – more Societal impact evaluator.

• What will tip the scales will be worked out during the evaluation process “…learn on the job.”

• Post evaluation interviews will serve to investigate this further.


intentions and strategies for evaluating the societal impact of research: insights from ref2014...

Data & Analytics

societal impact of research

valuing of impact

impact stuffand

amazing impact

impact orwhere

research evaluation

main panel asubpanel

impact evaluators n