project output versus influence in practice: impact as a dimension of research quality

Project output versus influence in practice

Impact as a dimension of research quality

6th International Conference on Evidence based Librarianship and Information PracticeKeynote presentation by Professor Hazel Hall, Director, Centre for Social Informatics, 30th June 2011

http://lisresearch.orghttp://lisresearch.org

If you are tweeting this session, please refer to me as @hazelh. Thank you!

If you are tweeting this session, please refer to me as @hazelh. Thank you!

LIS impact takes a number of forms Research impact measurement is tricky Research impact measurement matters Where impact is measured as a dimension of

research quality, a number of strategies can be deployed to extend project outcome reach

Key points

LIS impact takes a number of forms Research impact measurement is tricky Research impact measurement matters Where impact is measured as a dimension of

research quality, a number of strategies can be deployed to extend project outcome reach

Key points

Impact themes in LIS literature1. Evaluation of services – general use (debated)Impact themes in LIS literature1. Evaluation of services – general use (debated)

Impact themes in LIS literature2. Evaluation of service impact on end user behavioursImpact themes in LIS literature2. Evaluation of service impact on end user behaviours

Impact themes in LIS literature3. Evaluation of services as related to specific outcomes

Impact themes in LIS literature3. Evaluation of services as related to specific outcomes

Impact themes in LIS literature4. How to evaluate servicesImpact themes in LIS literature4. How to evaluate services

Impact themes in LIS literature5. BibliometricsImpact themes in LIS literature5. Bibliometrics

Impact themes in LIS literature5. Bibliometrics – publication impactsImpact themes in LIS literature5. Bibliometrics – publication impacts

Impact themes in LIS literature6. Bibliometrics research evaluationWe can stand on the shoulders of giants because librarians and information scientists have maintained a curated record

Impact themes in LIS literature6. Bibliometrics research evaluationWe can stand on the shoulders of giants because librarians and information scientists have maintained a curated record

Impact themes in LIS literature7. LIS research output LIS practiceImpact themes in LIS literature7. LIS research output LIS practice

Impact themes in LIS literature7. LIS research output LIS practiceImpact themes in LIS literature7. LIS research output LIS practice

Impact themes in LIS literature8. LIS practice LIS research (Brookes, 1976)

Impact themes in LIS literature8. LIS practice LIS research (Brookes, 1976)

Impact routes vary

Cumulative and indirect effect of exposure to research output

Direct policy mandate that derives from a particular study

Impact routes vary

Cumulative and indirect effect of exposure to research output

Direct policy mandate that derives from a particular study

LIS impact takes a number of forms Research impact measurement is tricky Research impact matters Where impact is measured as a dimension of

research quality, a number of strategies can be deployed to extend project outcome reach

Key points

Measurement challenge 1Attribution, and cause and effect, are not always accurate nor obvious

Measurement challenge 1Attribution, and cause and effect, are not always accurate nor obvious

Measurement challenge 1Attribution, and cause and effect, are not always accurate nor obvious

Measurement challenge 1Attribution, and cause and effect, are not always accurate nor obvious

Greer GarsonBest actress 1942 for role in Mrs Miniver Acceptance speech 5 minutes 30 seconds

Greer GarsonBest actress 1942 for role in Mrs Miniver Acceptance speech 5 minutes 30 seconds

Measurement challenge 2Sources of mediated training material are not always appreciated by practitioners

Measurement challenge 2Sources of mediated training material are not always appreciated by practitioners

Measurement challenge 3End-users are not normally aware of research effort to support services design and delivery

Measurement challenge 3End-users are not normally aware of research effort to support services design and delivery

Measurement challenge 4Certain types of research output attract more attention than others

Measurement challenge 4Certain types of research output attract more attention than others

Measurement challenge 5The impact of some is overwhelming (and disproportionate)

Measurement challenge 5The impact of some is overwhelming (and disproportionate)

LIS impact takes a number of forms Research impact measurement is tricky Research impact matters Where impact is measured as a dimension of

research quality, a number of strategies can be deployed to extend project outcome reach

Key points

Practitioner interest in research outputIncreases value of research undertakenNarrows research-practice gap benefits health of the profession

Practitioner interest in research outputIncreases value of research undertakenNarrows research-practice gap benefits health of the profession

Local contexts influence attitudes towards impact measurementREF2014 in the UK

Local contexts influence attitudes towards impact measurementREF2014 in the UK

LIS impact takes a number of forms Research impact measurement is tricky Research impact matters Where impact is measured as a dimension of

research quality, a number of strategies can be deployed to extend project outcome reach

Key points

NB presented here are strong messages that have emerged from RiLIES project empirical data analysed to date

http://lisresearch.org/rilies-project/

NB presented here are strong messages that have emerged from RiLIES project empirical data analysed to date

http://lisresearch.org/rilies-project/

Literature review; poll; focus groups; case studies; surveyLiterature review; poll; focus groups; case studies; survey

Research ideas for impact in practice

Focus on research areas of direct relevance to practitioners, both in terms of subject and context - time, place, sector

(Perth and London focus groups)

Research ideas for impact in practice

Focus on research areas of direct relevance to practitioners, both in terms of subject and context - time, place, sector

(Perth and London focus groups)

Research approach for impact in practice

Deploy action research approach

Mix in development and engagement, e.g. capacity building as hook, workshops throughout project

Encourage community creation around project for sustainability

Involve high profile, prestigious partners

(Evalued, Evaluating the impact of clinical librarian services, Open to all case studies)

Research approach for impact in practice

Deploy action research approach

Mix in development and engagement, e.g. capacity building as hook, workshops throughout project

Encourage community creation around project for sustainability

Involve high profile, prestigious partners

(Evalued, Evaluating the impact of clinical librarian services, Open to all case studies)

Project scale and diversity for impact in practice

Data collection needs to be on a scale for project recommendations to be applicable elsewhere

Ensure that projects are outward-looking to safeguard against accusations of bias

(Perth focus group)

Project scale and diversity for impact in practice

Data collection needs to be on a scale for project recommendations to be applicable elsewhere

Ensure that projects are outward-looking to safeguard against accusations of bias

(Perth focus group)

Face-to-face dissemination for impact in practice

Exploit face-to-face dissemination as the practitioners’ favoured route

Target dissemination according to career stage preferences

Invest project budget in training activities for project legacy

Launch events effective for policy maker target audiences

(Poll, focus groups, case studies)

Face-to-face dissemination for impact in practice

Exploit face-to-face dissemination as the practitioners’ favoured route

Target dissemination according to career stage preferences

Invest project budget in training activities for project legacy

Launch events effective for policy maker target audiences

(Poll, focus groups, case studies)

Accessible textual sources for impact in practice

Tailor a variety of reports according to intended audiences: tools they can access, career stage, sector…

Make reports accessible: easy to locate e.g. open access; understandable, e.g. possible to act on project outcomes

Work with profile dissemination partners (e.g. HEFCE, JISC, OCLC, Pew, RIN, SCONUL) to enhance authority of work and increase its reach

Ensure work is picked up by digest/alert services and bloggers

Accessible textual sources for impact in practice

Tailor a variety of reports according to intended audiences: tools they can access, career stage, sector…

Make reports accessible: easy to locate e.g. open access; understandable, e.g. possible to act on project outcomes

Work with profile dissemination partners (e.g. HEFCE, JISC, OCLC, Pew, RIN, SCONUL) to enhance authority of work and increase its reach

Ensure work is picked up by digest/alert services and bloggers

Conference registration openConference registration open

Training day announcementTraining day announcement

Publication of research reportPublication of research report

Study day announcementStudy day announcement

Study day announcementStudy day announcement

Conference registration to closeConference registration to close

Conference registration openConference registration open

Publication of blog postPublication of blog post

http://www.twitter.com/LISResearch provides an alert serviceshttp://www.twitter.com/LISResearch provides an alert services

Exploit network hubs5 key individuals in UK emerge from RiLIES project…

Exploit network hubs5 key individuals in UK emerge from RiLIES project…

5 UK research connectors 1. Rhona Arthur @RhonaEA5 UK research connectors 1. Rhona Arthur @RhonaEA

5 UK research connectors 2. Andrew Booth @scharrlock5 UK research connectors 2. Andrew Booth @scharrlock

5 UK research connectors 3. Phil Bradley @Philbradley5 UK research connectors 3. Phil Bradley @Philbradley

5 UK research connectors 4. Alison Brettle @BrettleAli5 UK research connectors 4. Alison Brettle @BrettleAli

5 UK research connectors 5. Hazel Hall @hazelh5 UK research connectors 5. Hazel Hall @hazelh

Face-to-face dissemination for impact in practice

What happens when formal opportunities for face-to-face dissemination are restricted:

training/travel budgets cut?

reduced professional memberships?

… and when research infrastructures are fractured? This has been addressed in UK in the formation of the Library and Information Science Research Coalition

Face-to-face dissemination for impact in practice

What happens when formal opportunities for face-to-face dissemination are restricted:

training/travel budgets cut?

reduced professional memberships?

… and when research infrastructures are fractured? This has been addressed in UK in the formation of the Library and Information Science Research Coalition

Researcher priorityPresent research output in an accessible way

Service manager priorityFacilitate research output consumption, e.g. reverse mentoring, job descriptions, reporting back from conferences

Course/accreditation priorityApplication of research in practice

Researcher priorityPresent research output in an accessible way

Service manager priorityFacilitate research output consumption, e.g. reverse mentoring, job descriptions, reporting back from conferences

Course/accreditation priorityApplication of research in practice

A greater role for social media?

Currently divides the practitioner community, e.g. blogs very popular with academic librarians but inaccessible to some others; Twitter essential to some, irrelevant to others

A greater role for social media?

Currently divides the practitioner community, e.g. blogs very popular with academic librarians but inaccessible to some others; Twitter essential to some, irrelevant to others

http://www.bite.ca/bitedaily/2010/07/social-media-monopoly-board-game/

A greater role for self-organising networks as communities of practice?

(with technical infrastructure provided by social media)

A greater role for self-organising networks as communities of practice?

(with technical infrastructure provided by social media)

Research funding to support researcher-practitioner links

http://www.lisresearch.org/dream-project

Research funding to support researcher-practitioner links

http://www.lisresearch.org/dream-project

DREaM project launch conferenceTuesday 19th July, London

Places still available

DREaM project launch conferenceTuesday 19th July, London

Places still available

LIS impact takes a number of forms Research impact measurement is tricky Research impact measurement matters Where impact is measured as a dimension of

research quality, a number of strategies can be deployed to extend project outcome reach Preliminary findings of RiLIES project dominated

by influence of dissemination strategy

Key points

Bringing it back to Salford: anyone recognise these lads?Bringing it back to Salford: anyone recognise these lads?

Bringing it back to Salford: anyone recognise these lads?Bringing it back to Salford: anyone recognise these lads?

What difference does it make? (1984)

What difference does it make? (1984)

Playlist 20 March 1984

Hand in glove; Still ill; Heaven knows I’m miserable now; This charming man; Girl afraid; Pretty girls make graves; This night has opened my eyes; Barbarism begins at home; What difference does it make? I don’t owe you anything; Handsome devil; You’ve got everything now; These things take time

Playlist 20 March 1984

Hand in glove; Still ill; Heaven knows I’m miserable now; This charming man; Girl afraid; Pretty girls make graves; This night has opened my eyes; Barbarism begins at home; What difference does it make? I don’t owe you anything; Handsome devil; You’ve got everything now; These things take time

Playlist 20 March 1984

Hand in glove; Still ill; Heaven knows I’m miserable now; This charming man; Girl afraid; Pretty girls make graves; This night has opened my eyes; Barbarism begins at home; What difference does it make? I don’t owe you anything; Handsome devil; You’ve got everything now; These things take time

Playlist 20 March 1984

Hand in glove; Still ill; Heaven knows I’m miserable now; This charming man; Girl afraid; Pretty girls make graves; This night has opened my eyes; Barbarism begins at home; What difference does it make? I don’t owe you anything; Handsome devil; You’ve got everything now; These things take time

Playlist 20 March 1984

Hand in glove; Still ill; Heaven knows I’m miserable now; This charming man; Girl afraid; Pretty girls make graves; This night has opened my eyes; Barbarism begins at home; What difference does it make? I don’t owe you anything; Handsome devil; You’ve got everything now; These things take time

Playlist 20 March 1984

Hand in glove; Still ill; Heaven knows I’m miserable now; This charming man; Girl afraid; Pretty girls make graves; This night has opened my eyes; Barbarism begins at home; What difference does it make? I don’t owe you anything; Handsome devil; You’ve got everything now; These things take time

Photo credits

Image: 'I Can Gather All the News I+Need+on+the+Weather+Report' http://www.flickr.com/photos/51035555243@N01/2671536366

Image: 'These Books are Sweet' http://www.flickr.com/photos/47823583@N03/4474869956

Image: ‘Radar Graph – Library Usage v Final Grades' http://www.flickr.com/photos/davepattern/4172110833/in/set-72157622847182901/

Image: '50lb weights' http://www.flickr.com/photos/62327186@N00/2185667691

Image: 'Article-level metrics at PLoS' http://www.flickr.com/photos/14829735@N00/3929622080

Image: '365: day 119' http://www.flickr.com/photos/33563858@N00/2451800342

Image: 'The locker misery' http://www.flickr.com/photos/33685308@N00/513100970

Image: '90" engine valves' http://www.flickr.com/photos/61132483@N00/2975215

Image: 'Oscar' http://www.flickr.com/photos/45756179@N03/5068766020

Image: 'ed vaizey' http://www.flickr.com/photos/25414047@N00/4065621328

Image: 'Greer Garson & Robert Young' http://www.flickr.com/photos/44186100@N00/2490143154

Image: ‘Waiting for the library to open’http://www.flickr.com/photos/davepattern/1424246518/in/set-72157602122007409

Image: 'BC007' http://www.flickr.com/photos/51548378@N05/4745027706

Image: 'Money to burn' http://www.flickr.com/photos/8363329@N05/811315055

Image: 'Warhol's Light Bulbs' http://www.flickr.com/photos/66814335@N00/3036254720

Image: 'Cheers' http://www.flickr.com/photos/71477195@N00/204879104

Image: 'P1010121' http://www.flickr.com/photos/67614763@N00/4304374413

Image: 'My social Network on Flickr, Facebook, Twitter+and+MyblogLog' http://www.flickr.com/photos/49503019876@N01/1824234195

Smiths at Salford Lads Club (inside cover of The Queen is dead)by Stephen Wright

Brookes, B.C. (1976). A new paradigm for information science. The Information Scientist, 10(3), 103-12.

Peter Cruickshank, Ella Taylor-Smith, Jenny Gebel (RiLIES project team)

Stephanie Kenna (LIS Research Coalition) All those who have contributed data to date

Reference & acknowledgements

Professor Hazel Hallh.hall@napier.ac.uk @hazelh