methodologies for measuring research impact for a new interdisciplinary institute
DESCRIPTION
Methodologies for measuring research impact for a new interdisciplinary institute. Janet Baker and Dr Nina Prasolova Queensland University of Technology Library. Complicated subject and so best told as a story. The request The task – impact report for a “virtual” institute The challenge - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/1.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
L I B R A R Y S E R V I C E S
www.library.qut.edu.au
D I V I S I O N O F T E C H N O L O G Y , I N F O R M A T I O N A N D L E A R N I N G S U P P O R TD I V I S I O N O F T E C H N O L O G Y , I N F O R M A T I O N A N D L E A R N I N G S U P P O R T
Methodologies for measuring research impact for a new interdisciplinary institute
Janet Baker and Dr Nina PrasolovaQueensland University of Technology Library
![Page 2: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/2.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Complicated subject and so best told as a story
The request The task – impact report for a “virtual” institute The challenge The negotiations The methodologies The workflow The lessons we learned
![Page 3: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/3.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Characteristics of Institutes or Research Groups
Established institutes
Clear purpose and identity Set agenda, coherent workflow Membership list Physical space with known teams High quality resources already available to measure
research impact Benchmarking progressive and peer group known
![Page 4: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/4.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Institute for Future Environments
“Virtual” institute
Based around broad concepts, a Big Idea or evolving research themes with loosely affiliated individual researchers
“Fluid” relationships Disparate output types and citing cultures to interpret
![Page 5: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/5.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E SInterdisciplinarity: a field of study that crosses traditional boundaries between academic disciplines, which integrates these different approaches or methods (Center for the Study of Interdisciplinarity).
Boston Interdisciplinary Instituteshttp://www.bu.edu/cgs/citl/interdisciplinary-institutes/
Literature review reveals increasing number of reports and interest in a growing body of literature around issues, policies, challenges regarding IDR
![Page 6: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/6.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Challenges
No clear boundaries or disciplinary structure Loosely affiliated clusters Defining Interdisciplinarity - subject categories or
research codes Multiple variables – type of publication, where indexed,
differing culture of citation, collaborative arrangements Expectations – make clear what is possible
![Page 7: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/7.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Expectations/ Negotiations: What is the goal?To evaluate performance – factors to consider Look at the traditional metrics – what are you trying to illustrate, what
can you use - Which indicators are consistent for group What tools will give greatest coverage/accuracy Is it possible to analyse cited references - interdisciplinarity Journal subject categories? – WoS and Scopus Percentiles and Average Citation Metrics Percentage share of total publications
![Page 8: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/8.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Expectations - What is the goal?To predict and for strategic planning Co-authorship graphs – analysis of effective collaborations Co-citation analysis (SciVal) identifying emerging trends Identify Networks outside the discipline boundaries
To improve communications and reporting efficiency Find other research support staff also collecting and
evaluating impact? Tacit knowledge (people evaluating are not the researchers) Knitting together support around the researcher
![Page 9: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/9.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
What is the reality?
Each subgroup is different – refine the theme and vocabulary
The type of research output dictates the assessment tools
Which metrics are available across the theme Benchmarking strategy is specific for the
subgroup/cluster
![Page 10: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/10.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Identifying clusters – establish borders
Field of Research Codes MeSH or other Thesauri Journal CategoriesKeyword analysis
![Page 11: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/11.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Street computing : towards an integrated open data Application Programming Interface (API) for cities
http://eprints.qut.edu.au/59160/
![Page 12: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/12.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Analysis of productivity and impact
Percentage of the outputs in the theme Distribution of citations (average and percentile) Collaborations Interdisciplinarity Downloads Attention Metrics Media releases
![Page 13: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/13.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Metrics of Research Productivity and Impact
![Page 14: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/14.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
ESI Table
![Page 15: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/15.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Search Scopus – using identified research themes then work out citation percentiles
Map QUT output to the total output and citation percentiles
Export and analyse patterns
Assess Collaborations
Assess interdisciplinarity
Workflow
![Page 16: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/16.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Scopus benchmarking (share and percentiles of ‘cited by’)“Tropical Crop*” (All fields) (limit to Country: Australia)
QUT share in the total pubs: 33 (36%) out of 91
3 QUT papers in 10th percentile and9 QUT papers in 25th percentile
![Page 17: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/17.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
![Page 18: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/18.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Author collaborations
![Page 19: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/19.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Analysing collaborations in SciVal
SciVal (Elsevier) January, 2014
![Page 20: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/20.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Summary: what we have learnt
For “virtual” institutes it is important to identify clusters Baselines – sometimes need to change, cultures differ Google Scholar Citations Profile – appears most useful for
some clusters, but data needs to be checked Possible to manually granulate by retrieving those
publications that are in the theme or cluster and then benchmark your institute
Final report needs detailed explanation of metrics and tools chosen and their applicability and replicability
![Page 21: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/21.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
What we learned
There is a place for librarians in this exercise
For emerging new institutes librarians could be instrumental providing input into shaping the research clusters and showing future opportunities in interdisciplinarity and collaborations
![Page 22: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/22.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
References
Huutoniemi, K., Klein, J. T., Bruun, H., & Hukkinen, J. (2010). Analyzing interdisciplinarity: Typology and indicators. Research Policy, 39(1), 79-88. doi: http://dx.doi.org/10.1016/j.respol.2009.09.011
Roessner, D., Porter, A. L., Nersessian, N. J., & Carley, S. (2013). Validating indicators of interdisciplinarity: linking bibliometric measures to studies of engineering research labs. Scientometrics, 94(2), 439-468.
Wagner, C. S., Roessner, J. D., Bobb, K., Klein, J. T., Boyack, K. W., Keyton, J., . . . Börner, K. (2011). Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature. Journal of Informetrics, 5(1), 14-26.
Rymer, L. (2011). Go8 Backgrounder 23 | Measuring the impact of research – The context for metric development. Turner, ACT: The Group of Eight. Retrieved from
Center for the Study of Interdisciplinarity. (2013). Retrieved from http://csid.unt.edu/research/ria_overview.html
LSE Public Policy Group. (2011). Maximizing the impacts of your research: A handbook for social scientists (Consultation Draft 3). Retrieved from http://www.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_April_2011.pdf
![Page 23: Methodologies for measuring research impact for a new interdisciplinary institute](https://reader036.vdocument.in/reader036/viewer/2022062310/568162af550346895dd336e0/html5/thumbnails/23.jpg)
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Creative Solutions
Valerie Ucumari(CC BY-NC-ND 2.0)