Writing a REF Impact Case Study

Impact of Social Sciences Project

LSE Public Policy Group

The current state of play

• 360 days to REF deadline

• Survey of submission intentions due in tomorrow (only 300 words in total)

• Universities have a first draft of prospective cases

• But these still need quite a lot of work

• How your institutions are handling things will vary

• There is still time for action – but much uncertainty, and we have no magic or official answer

Today is a discursive session on key uncertainties:

1. Uncertainties around reach and significance

2. Envisioning the overall narrative

3. Proving 2* quality for underpinning research

4. What is ‘impact’ and what is not?

5. Maximizing intensity of impact evidence

Overview of the session

1. Uncertainties around reach and significance

Definitions of reach and significance

• Reach is understood as extent and diversity of the communities, environments, individuals, organizations or any other beneficiaries that have benefitted or have been affected.

• Significance is understood as the degree to which impact has enriched, influenced, informed or changed policies, opportunities, perspectives, or practices of communities, individuals or organizations.

Mapping HEFCE’s impact criteria

Definitions of reach and significance • What is the relationship between the two? Can

increased reach compensate for reduced significance?

• Emphasis on showing change – this implies need to demonstrate linear effects

• Uncertainty around things like generating debate in a profession, engagement over proposition not taken up

• Is this biased towards public policy impacts?

• Panels reassure us that iterative, dynamic, and narrative accounts will be expected and encouraged

2. Envisioning the overall narrative

Think creatively about the narrative

• It’s all about the narrative – and the ‘so what?’ question

• Start with the user side and work backwards

• Not necessarily your most cited academic work – where have you engaged?

• Research outputs must be ‘credibly linked’ to impact

• Corporate versus individual submissions

• Integrate different streams of work or expertise in the department

The case study narrative

Short summary of the case (100 words)

Underpinning research (500 words)

References to the research (max. 6)

Details of the impact (750 words)

Corroborating sources Max. 10 (5 individuals)

Strong opening headlining the research and impacts

Narrative account of how the research evolved

Majority of 2* references to research outputs

Building to a strong, factoid-intensive narrative on impacts

Good mix of references and testimonies,

Imbalance towards research rather than impacts

Over-emphasis on research outputs and weak quality indicators

Excessive focus on dissemination and advice

Lack of factoid evidence and weak signs of self-research, over-reliance on weak testimonials

IDEALLY… IN REALITY…

Process trace your impact back to your underlying research

Effects

Changes

Benefits

Impact of outputs or activities of

external body/actors

Intermediate impacts – usage or cites, by external actors

Partnership funding /

Direct contacts = auditable impacts

Under- pinning

research

Evidence of (strong)

dissemination

Public engagement contributions

Public engagement contributions

3. Proving 2* quality of underpinning research

Deciding what research to submit

• All work must cross the 2* quality threshold (i.e. it must be ‘internationally recognized’)

• Easier for traditional academic outputs (i.e. already in previous RAE submissions)

• However impact research outputs may be more applied: e.g. grey literature

• Panel will take a look, if not obviously 2*, then it is read.

• Therefore need to reassure the Panel that this work is of the required quality via another means

Critically assessing your own research

• Has any of your impact output been assessed e.g. through an internal or organisational review process?

• For research reports, was an evaluation completed on outputs from a programme?

• Panels C and D accept outputs that can be shown to be highly cited or form a ‘reference point for further research beyond the original institution’

• If none of these apply, then you or your Department will need to organise your own review or evaluation

4. What is impact and what is not?

What is feasible in assessing impact?

Writing about your impacts

– Dissemination activities on their own without any follow-on interaction or take up

– Tricky in the area of public dissemination – a radio broadcast is not enough

– Something that is generally part of academic service

– Just providing expertise and advice to however important a person/organization

Net impacts

Gross impacts

Non-impacts

– Maximizing the level of detail and specifics at this level – always be above this threshold

– Danger of over-claiming or making infeasible estimates

– Making informed guesstimates at net effects

5. Maximizing intensity of impact evidence

Research your impact proactively

• 750 words max to demonstrate intensive impacts

• Narrative should not detract from specifics / data

• Being your own detective and research yourself

• Start with Google and see where your research has been picked up by external users

• Then carefully document official and unofficial dissemination that you undertook

• Outline as clearly as you can the ‘effect, change or benefit’ that your research made.

Research your impact proactively II

• Can you be precise about who exactly used your work in business, government, or third sector?

• Can you offer at least some good evidence of how they used it and the effects it had?

• Did your contacts or involvement continue over a period of time? And did it had knock-on effects?

• Did people pay for your work or advice? Can you track all the flows of money?

For more details see:

Impact of Social Sciences blog:

http://blogs.lse.ac.uk/impactofsocialsciences/

Email: impactofsocialsciences@lse.ac.uk

Twitter: @lseimpactblog

Facebook: Impact of Social Sciences

Supplementary Slides

Definition of impact

• ‘an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia’

• INCLUDES raising the level of public debate and public scrutiny, public engagement, changing policy and practices, use by professional bodies.

• EXCLUDES (a) impact on research or the advancement of academic knowledge within the HE sector (b) acting as an expert or advisor to a public body

The REF Main Panels

Main Panel A covers: Clinical Medicine Public Health, Health Services and Primary Care Allied Health Professions Psychology, Psychiatry and Neuroscience Biological, Agricultural and Food Sciences

Main Panel B covers: Earth Systems and Environmental Sciences Chemistry and Physics Mathematical Sciences Computer Science and Informatics All areas of Engineering

Main Panel C covers the social sciences including: Anthropology; Business and Management; Education; Economics; Geography, Law; Politics and IR; Social Policy; Sociology

Main Panel D covers the humanities including: Art and Design; Dance; Drama; Communication and Media Studies, English; History; Modern Languages; Music; Philosophy; Theology;

More on the social science REF panel

• One user appointed per sub-panel

• These appointees will coordinate recruiting additional

research users (by Spring 2013)

• Two readers of case studies – 1 user, 1 academic

• Not ‘user-led’ but users will play a key role in

assessment

HEFCE guidance on underpinning research

HEFCE guidance on recording impacts

HEFCE guidance on impact evidence

For public engagement cases

You need to be able to answer these questions:

• Can you show evidence of your research being strongly disseminated?

• Did you undertake outreach activities?

• Can you show evidence of your research being widely received?

• Were there secondary or follow-on effects? i.e. reaching additional audiences, building new interactions, creating additional outputs.