Bibliometrics in higher Bibliometrics in higher education: the impact on education: the impact on

librarieslibraries

Keith WebsterKeith WebsterUniversity of Queensland, University of Queensland,

AustraliaAustraliaDr Berenika M. WebsterDr Berenika M. Webster

CIBER, Univ. College LondonCIBER, Univ. College London

OutlineOutline• Bibliometrics and university librariesBibliometrics and university libraries

– Structure of disciplinesStructure of disciplines– Collection managementCollection management– Research evaluationResearch evaluation

• Bibliometric techniquesBibliometric techniques• Sample projectsSample projects

Structure of disciplinesStructure of disciplines

• Domain analysisDomain analysis

• Patterns of scholarshipPatterns of scholarship

Collection managementCollection management

• Patterns of usePatterns of use

• Core journals in disciplineCore journals in discipline

• International comparisonInternational comparison

Research evaluationResearch evaluation

• Approaches to research assessment: experiences Approaches to research assessment: experiences from UK, NZ and Australiafrom UK, NZ and Australia

• Peer assessment and metricsPeer assessment and metrics

• BibliometricsBibliometrics

– Assessment toolsAssessment tools

– Challenges of bibliometric approachChallenges of bibliometric approach

– How to increase the quality and impact of published outputsHow to increase the quality and impact of published outputs

What do governments want What do governments want for their money?for their money?

• Economic outcomesEconomic outcomes– increase wealth creation & prosperityincrease wealth creation & prosperity– improve nation’s health, environment & improve nation’s health, environment &

quality of lifequality of life

• InnovationInnovation• R&D from private sectorR&D from private sector• Improved competitivenessImproved competitiveness• Less “curiosity-driven” activityLess “curiosity-driven” activity

Research Assessment Exercise Research Assessment Exercise (UK)(UK)

• Started in mid-1980s to determine size of research funding from Started in mid-1980s to determine size of research funding from HEFCEs to universities. Early 1990s brought shift from quantity to HEFCEs to universities. Early 1990s brought shift from quantity to quality quality

• 60 subject panels examine, for quality, selected outputs of UK 60 subject panels examine, for quality, selected outputs of UK academics. Also, previous funding, evidence of esteem, PhD academics. Also, previous funding, evidence of esteem, PhD completions and res. environment are taken into considerationcompletions and res. environment are taken into consideration

• Grades for submission units; correlates with bibliometrics measures Grades for submission units; correlates with bibliometrics measures (Charles Oppenheim’s research)(Charles Oppenheim’s research)

• It will be scraped in 2008. New systems will be a hybrid of a “basket It will be scraped in 2008. New systems will be a hybrid of a “basket of metrics” and peer-review (???)of metrics” and peer-review (???)

• Changed behaviour of scholarsChanged behaviour of scholars– Increased volume of publications (salami slicing?)Increased volume of publications (salami slicing?)– Increased quality of outputs (numbers of 5 and 5* departments increase)Increased quality of outputs (numbers of 5 and 5* departments increase)

Performance-Based Research Performance-Based Research Fund (PBRF) New Zealand Fund (PBRF) New Zealand • Introduced in early 2003Introduced in early 2003• 12 panels grades individuals in 41 subject categories 12 panels grades individuals in 41 subject categories • Portfolio has three componentsPortfolio has three components

– Publications (4 best and the rest)Publications (4 best and the rest)– Peer esteemPeer esteem– Contribution to research environmentContribution to research environment

• 2003 round produced lower scores than expected and great 2003 round produced lower scores than expected and great variability between subjects (from 4.7 in philosophy to 0.3 variability between subjects (from 4.7 in philosophy to 0.3 in nursing). 40% of submitted staff were judged research in nursing). 40% of submitted staff were judged research inactive (grade R)inactive (grade R)

• Scores for institutions are calculated from these and used Scores for institutions are calculated from these and used together with PhD completion and external funding metrics together with PhD completion and external funding metrics to calculate awards to institutionsto calculate awards to institutions

• 2006 round incorporates “breaks” for new researchers2006 round incorporates “breaks” for new researchers

Research Quality Framework Research Quality Framework (RQF) - Australia(RQF) - Australia• To be introduced in 2007 (?) To be introduced in 2007 (?)

• QualityQuality includes the intrinsic merit of original research and includes the intrinsic merit of original research and academic impact. This relates to recognition of the originality of academic impact. This relates to recognition of the originality of research by peers and its impact on the development of the same research by peers and its impact on the development of the same or related discipline areas within community of peers.or related discipline areas within community of peers.

Assessment: peer review and metricsAssessment: peer review and metrics

• ImpactImpact or use of the original research outside of the peer or use of the original research outside of the peer community that will typically not be reported in traditional peer community that will typically not be reported in traditional peer review literature. Relates to the recognition by qualified end-users review literature. Relates to the recognition by qualified end-users that quality research has been successfully applied to achieve that quality research has been successfully applied to achieve social, cultural, economic and/or environmental outcomes.social, cultural, economic and/or environmental outcomes.

Assessment: expert opinion and narratives or metrics?Assessment: expert opinion and narratives or metrics?

RQF evidence portfolioRQF evidence portfolio

• Four “best” outputs (with rationale for selection)Four “best” outputs (with rationale for selection)

• List of all published outputs in the last six yearsList of all published outputs in the last six years

• Statement of “early impact” as assessed by Statement of “early impact” as assessed by qualified end-users (description of outcome; qualified end-users (description of outcome; identification of beneficiaries; metrics illustrating identification of beneficiaries; metrics illustrating benefits; linkage between claimant and benefits; linkage between claimant and beneficiary)beneficiary)

• Context statement for the research groupContext statement for the research group

Evaluation: peer vs. metricsEvaluation: peer vs. metrics

• Evaluate/reward past performanceEvaluate/reward past performance

• Measures short to medium term impactsMeasures short to medium term impacts

• Established use for the evaluation of Established use for the evaluation of academic impact (within the group of peers)academic impact (within the group of peers)

• No established/accepted procedures for No established/accepted procedures for evaluation of impact outside academeevaluation of impact outside academe

Peer reviewPeer review

• ““Traditional” assessment is an outputs-based peer-reviewTraditional” assessment is an outputs-based peer-review

– a panel of experts “reads” submissions or outputsa panel of experts “reads” submissions or outputs– not always transparent (selection; evaluation criteria)not always transparent (selection; evaluation criteria)– subjective (concept of quality is very difficult to objectivise)subjective (concept of quality is very difficult to objectivise)– relies on proxies (e.g. ranking/prestige of publishing journal; relies on proxies (e.g. ranking/prestige of publishing journal;

inst. affiliation of authors, etc.) inst. affiliation of authors, etc.) – composition of panels will affect their judgements composition of panels will affect their judgements – lacks comparability between panelslacks comparability between panels– ““punishes” innovative or multidisciplinary researchpunishes” innovative or multidisciplinary research– ““old boys” club disadvantaging young researchers old boys” club disadvantaging young researchers – assesses what already has been assessed by journals’ peer-assesses what already has been assessed by journals’ peer-

review processes review processes – time and resource intensivetime and resource intensive

MetricsMetrics• Range of unobtrusive and Range of unobtrusive and

transparent measures can be:transparent measures can be:– Input-basedInput-based

• IncomeIncome

•Success ratesSuccess rates

– Output-basedOutput-based•Long-term impact (patents, IP, etc.)Long-term impact (patents, IP, etc.)

•Bibliometric measures (volume and Bibliometric measures (volume and impact/quality)impact/quality)

Problems with metrics (1)Problems with metrics (1)

• Input-basedInput-based– RetrospectiveRetrospective– Creates “Matthew effect”Creates “Matthew effect”– Disadvantages younger researchers Disadvantages younger researchers

(without previous track record)(without previous track record)– Undervalues research done “on the Undervalues research done “on the

string” string”

Problems with metrics (2)Problems with metrics (2)

• Output basedOutput based

– Citations as a measure of intellectual influence?Citations as a measure of intellectual influence?– Comparing like with like (benchmarking)Comparing like with like (benchmarking)– Inadequate toolsInadequate tools– Social science and humanities (in search for new Social science and humanities (in search for new

metrics)metrics)– Research on the periphery (local citation indices)Research on the periphery (local citation indices)– New models of communication (capturing it)New models of communication (capturing it)– Impacts on wider community (“tangible” outcomes) Impacts on wider community (“tangible” outcomes) – Lack of expertise in analysis and interpretation Lack of expertise in analysis and interpretation

(education)(education)

Indicators of quality as Indicators of quality as measured using published measured using published outputsoutputs• Number of publicationsNumber of publications

• Citation counts to these publications (adjusted for self-citations)Citation counts to these publications (adjusted for self-citations)-what “window” should be used? 4, 5, 10 years?-what “window” should be used? 4, 5, 10 years?

• Citations per publicationCitations per publication

• Percentage of uncited papersPercentage of uncited papers

• Impact factors (of publishing journals)Impact factors (of publishing journals)

• Diffusion factor (of citing journals) – profile of users of research (who, where, when and Diffusion factor (of citing journals) – profile of users of research (who, where, when and what)what)

• ““Impact factor” of a scholar - Hirsh index (h index) Impact factor” of a scholar - Hirsh index (h index) – (numbers of papers with this number of citations). (numbers of papers with this number of citations). – Your h index =75 if you wrote at least 75 papers with 75 citations each.Your h index =75 if you wrote at least 75 papers with 75 citations each.

Note: These should not be seen as “absolute” numbers but always seen in the context of Note: These should not be seen as “absolute” numbers but always seen in the context of the discipline, research type, institution profile, seniority of a researcher, etc.the discipline, research type, institution profile, seniority of a researcher, etc.

Compare like with like!Compare like with like!• Applied research attracts fewer citations than basic research.Applied research attracts fewer citations than basic research.

• Differences in citation behaviour between disciplines (e.g. papers in Differences in citation behaviour between disciplines (e.g. papers in organisational behaviour attract 5 times as many citations as papers in organisational behaviour attract 5 times as many citations as papers in accounting). accounting).

• Highest IF journal in immunology is Highest IF journal in immunology is Ann Rev ImmunAnn Rev Immun (IF 47.3) Mean for cat. 4.02; (IF 47.3) Mean for cat. 4.02; and in health care and services category is and in health care and services category is Milbank QMilbank Q. (IF of 3.8). Mean for cat. . (IF of 3.8). Mean for cat. 1.09.1.09.

• Matthew effect.Matthew effect.

Benchmarking must be done using comparable variables!Benchmarking must be done using comparable variables!

Tools availableTools available• Publication and citation dataPublication and citation data

– Web of ScienceWeb of Science– SCOPUSSCOPUS– Google ScholarGoogle Scholar– Local/national databases Local/national databases

• Other bibliometric indicatorsOther bibliometric indicators

– Journal Citation Reports (JCR)Journal Citation Reports (JCR)– Other indicators databases (national, essential, Other indicators databases (national, essential,

university, institutional)university, institutional)– ISIHighlyCited.com ISIHighlyCited.com

WoS and Scopus: subject WoS and Scopus: subject coverage (% of total records)coverage (% of total records)

Science, 77

Social Sciences, 14

Arts & Humanities,

9

Biological & Environmental Sciences, 13

Social Sciences, 2

Physical Sciences, 25

Health & Life Sciences, 60

WoS SCOPUS

Google Scholar ?

Jacso, 2005

Web of ScienceWeb of Science• Covers around 9,000 journal titles and 200 Covers around 9,000 journal titles and 200

book series divided between SCI, SSCI and book series divided between SCI, SSCI and A&HCI.A&HCI.

• Electronic back files available to 1900 for SCI Electronic back files available to 1900 for SCI and mid- 50s for SSCI and mid-70s for A&HCI.and mid- 50s for SSCI and mid-70s for A&HCI.

• Very good coverage of sciences; patchy on Very good coverage of sciences; patchy on “softer” sciences, social sciences and arts and “softer” sciences, social sciences and arts and humanities.humanities.

• US and English-language biased.US and English-language biased.• Full coverage of citations.Full coverage of citations.• Name disambiguation tool. Name disambiguation tool. • Limited downloading options.Limited downloading options.

ScopusScopus• Positioning itself as an alternative to ISIPositioning itself as an alternative to ISI• More journals from smaller publishers and open access More journals from smaller publishers and open access

(13,000 journal titles; 600 open access journals; 750 conf (13,000 journal titles; 600 open access journals; 750 conf proceedings; 600 trade publications, 2.5 mil “quality” web proceedings; 600 trade publications, 2.5 mil “quality” web pages)pages)

• Source data back to 1960.Source data back to 1960.• Excellent for physical and biological sciences; poor for social Excellent for physical and biological sciences; poor for social

sciences; does not cover humanities or arts.sciences; does not cover humanities or arts.• Better international coverage (60% of titles are non-US)Better international coverage (60% of titles are non-US)• Not much of a back file (e.g. citation data for the last decade Not much of a back file (e.g. citation data for the last decade

only)only)• Not “cover to cover” and not up to dateNot “cover to cover” and not up to date• Easy to use in searching for source publications; clumsy in Easy to use in searching for source publications; clumsy in

searching cited publications.searching cited publications.• Citation tracker works up to 1000 records only.Citation tracker works up to 1000 records only.• Limited downloading options.Limited downloading options.

Google ScholarGoogle Scholar

• Coverage and scope?Coverage and scope?

• Inclusion criteria?Inclusion criteria?

• Very limited search optionsVery limited search options

• No separate cited author searchNo separate cited author search

• Free!Free!

Taiwan in the three sources,1996-Taiwan in the three sources,1996-20052005

0

10000

20000

30000

40000

50000

60000

70000

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Scopus (1.23%)

WoS (0.98%)

Google Scholar

Social sciencesSocial sciences• Communication in social sciencesCommunication in social sciences

– BooksBooks– National journalsNational journals– International journalsInternational journals– Non-scholarly outputs (culture-creating role) Non-scholarly outputs (culture-creating role)

• ““International” tools inadequateInternational” tools inadequate• Creation of “national” and “regional” toolsCreation of “national” and “regional” tools

– Chinese Social Sciences Citation IndexChinese Social Sciences Citation Index– Polish Sociology Citation IndexPolish Sociology Citation Index– Serbian Social Sciences IndexSerbian Social Sciences Index– Polish Citation Index for HumanitiesPolish Citation Index for Humanities– European Humanities Citation index European Humanities Citation index

Science on the peripheryScience on the periphery

• Poor coverage in international Poor coverage in international databasesdatabases

• Intl. benchmarking difficultIntl. benchmarking difficult

• Impossible for local evaluationsImpossible for local evaluations

• Development of national sources of Development of national sources of datadata

• While volume is growing, citation rates While volume is growing, citation rates lag behindlag behind

Science on the peripheryScience on the periphery

Figure 1: Polish scientific papers in SCI (in Polish and foreign journals), three-year running means and Polish SCI journals, 1980-

1999

0

1000

2000

3000

4000

5000

6000

7000

8000

1981 1983 1985 1987 1989 1991 1993 1995 1997

Pap

ers

per

yea

r

0

5

10

15

20

25

30

Nu

mb

er o

f P

oli

sh j

ou

rnal

s in

SC

I

Papers in foreign journals

Papers in Polish journals journals

Polish SCI journals (right scale)

Capturing new forms of Capturing new forms of communicationcommunication

• Use of online journals (deep log Use of online journals (deep log analysis)analysis)

• Traffic on websites and downloadsTraffic on websites and downloads

• In-links to websitesIn-links to websites

Impact of research: example of Impact of research: example of biomedicinebiomedicine

• Informing policies (citations on guidelines, Informing policies (citations on guidelines, govt. policy, development of medicines)govt. policy, development of medicines)

• Building capacity (training; development)Building capacity (training; development)

• Relationship between research and health Relationship between research and health outcomes and cost savingsoutcomes and cost savings

• Healthier workforceHealthier workforce

Citations on clinical Citations on clinical guidelinesguidelines• 47 Clinical Guidelines and 64 Health 47 Clinical Guidelines and 64 Health

Technology Appraisals prepared by the Technology Appraisals prepared by the UK’s National Institute for Clinical UK’s National Institute for Clinical Excellence (NICE) were analysed for Excellence (NICE) were analysed for citationscitations– cited 25% of UK papers (2.5 more than cited 25% of UK papers (2.5 more than

expected)expected)– Majority were clinical papersMajority were clinical papers– Median lag time between publication of res. Median lag time between publication of res.

paper and its citation was 3.3 yearspaper and its citation was 3.3 years

Relative commitment to research Relative commitment to research and burden of diseaseand burden of disease

Prostate cancer

IT

JP

CA

ESDE

AU

NL

FR

US

SE

CH

UK

y = 0.1112x + 0.1645

R 2 = 0.087

0.0

0.2

0.4

0.6

0.8

0.0 0.5 1.0 1.5 2.0 2.5 3.0

Percent of all deaths

Per

cent

of

biom

ed r

esea

rch

How to increase “quality” of How to increase “quality” of your publications?your publications?• Publish in the right journals (prestige; importance to the discipline; impact factor vs. diffusion Publish in the right journals (prestige; importance to the discipline; impact factor vs. diffusion

factor)factor)

• Publish in English Publish in English

• Write review articlesWrite review articles

• Engage in basic researchEngage in basic research

• Become a journal editor (Lange, 1997)Become a journal editor (Lange, 1997)

• Acquire a co-author (preferably from US or UK)Acquire a co-author (preferably from US or UK)

• Get external funding (from different sources) Get external funding (from different sources)

• Make your outputs available in open access (own website, institutional and subject Make your outputs available in open access (own website, institutional and subject repositories) (Antelman, 2004; Harnard various)repositories) (Antelman, 2004; Harnard various)

• ““Advertise” your publications on listservs and discussion groupsAdvertise” your publications on listservs and discussion groups

• Make and maintain professional/social contacts with others in your research area (Rowlands, Make and maintain professional/social contacts with others in your research area (Rowlands, 2000) 2000)