bibliometrics – use and limitations wolfgang glänzel ku leuven, belgium ispr, has, hungary

BIBLIOMETRICS BIBLIOMETRICS ––

USE AND LIMITATIONSUSE AND LIMITATIONS

Wolfgang GlänzelKU Leuven, BelgiumISPR, HAS, Hungary

Wolfgang Glänzel Bibliometrics. Use and limitations 2/37

STRUCTURE OF THE PRESENTATIONSTRUCTURE OF THE PRESENTATION

1. Introduction

2. Scope and structure of bibliometrics

3. Data sources of bibliometric research

4. “Perspective shift” in use of bibliometrics

5. Bibliometrics as a tool in research evaluation

6. Bibliometric research and services at the Steunpunt O&O Statistieken


1. 1. INTRODUCTIONINTRODUCTION


1. Introduction

What is bibliometrics?What is bibliometrics?

The terms bibliometrics and scientometrics were introduced by Pritchard and Nalimov & Mulchenko in 1969, and are nowadays used almost synonymously.

Pritchard

“bibliometrics is the application of mathematical and statistical methods to books and other media of communication”.

Nalimov & Mulchenko

Scientometrics is “the application of those quantitative methods which are dealing with the analysis of science viewed as an information process”.


1. Introduction

Gorkova (1988) Informetrics stands for a more general subfield of information science dealing with mathematical-statistical analysis of communication processes in science.

A quite broad definition of the scope of bibliometrics given by Glänzel & Schoepflin (1994) includes “all quantitative aspects and models of science communication, storage, dissemination and retrieval of scientific information”.


1. Introduction

These notions also integrate many presently existing orientations such as applications to science policy, library science, technometrics, Web analyses and information retrieval. Prominent methodological issues are related to

• subject delineation • subject evolution and structure • publication activity/scientific productivity • scientific collaboration, co-authorship networks • citation impact and citation networks • bibliographic coupling, co-citation analysis • computational linguistics; text mining, co-word analysis • visualisation techniques• data matching • science-technology linkage


Common misbeliefs on bibliometricsCommon misbeliefs on bibliometrics

• Main task of bibliometrics should be the expeditious issuing of “prompt” and “comprehensible” indicators for science policy and research management.

• Bibliometrics is only an auxiliary tool, research on methodology is unnecessary. Instead, bibliometricians should elaborate guidelines explaining the use of their indicators.

• Bibliometrics might be reduced to simple counting activities in order to replace/supplement qualitative assessment by quantitative indicators and to set publication output off against funding.

1. Introduction


1. Introduction

Facts about bibliometricsFacts about bibliometrics

Bibliometrics is a powerful, multifaceted endeavour encompassing sub-areas such as

- Structural scientometrics came up with results like the re-mapping of the epistemological structure of science.

- Dynamic scientometrics constructed sophisticated models of scientific growth, obsolescence, citation processes, etc.

- Evaluative scientometrics developed arrays of indicators to be used to characterise research performance at different levels of aggregation and come up with benchmarking solutions.


2.2. SCOPE AND STRUCTURE OF SCOPE AND STRUCTURE OF

BIBLIOMETRICSBIBLIOMETRICS


2. Structure of BibliometricsStructure of Bibliometrics

Three “components” of present-day bibliometrics according to its three main target-groups

Bibliometrics for bibliometricians

The domain of bibliometric “basic research”.

Bibliometrics for scientific disciplinesA large but also the most diverse interest-group. Due to the scientists’ primary scientific orientation, their interests are strongly related to their speciality.

Bibliometrics for science policy and research managementThe most important field of application. Here the assessment and comparative analysis of research performance are in the foreground.


2. Structure of BibliometricsStructure of Bibliometrics

Webometrics

Informetrics

Mathematics/Physics

Library andInformation Science

Sociology of science

History of science

Economics

Scientific information

Services for

Research in

Librarianship

Science policyResearch management

Scientometricsapplied

basicLife sciences

Links of bibliometrics with related research fields and application services


3.3. DATA SOURCES OF BIBLIOMETRIC DATA SOURCES OF BIBLIOMETRIC

RESEARCHRESEARCH


Data sources of bibliometrics are bibliographies and bibliographic databases.

Large scale analyses can only be based on bibliographic databases.

Despite objections to journal coverage and data processing policy of the ISI, the databases of the Institute for Scientific Information (Thomson - ISI), above all, the Science Citation Index (Expanded) have become the most generally accepted data source.

Recently several countries have developed their own national citation indexes which are also used (as supplementary) data sources.

3. Sources of BibliometricsSources of Bibliometrics


3. Sources of BibliometricsSources of Bibliometrics

Bibliographic databases are generally designed for the retrieval of information. However, their versatility is often limited in bibliometric use.

Originally, bibliometrics was developed for studying communication in basic research. Present-day tasks:

• How to extend bibliometrics towards application to engineering, social sciences and humanities?

• How to measure and evaluate aspects of technology linkage?

Since non-periodicals are important conveyors of information in engineering, social sciences and humanities, journal based data-sources are accepted by scientists only with certain reservations.


4.4. “PERSPECTIVE SHIFT” IN USE OF “PERSPECTIVE SHIFT” IN USE OF

BIBLIOMETRICSBIBLIOMETRICS


4. Perspective ShiftPerspective Shift

Bibliometrics deals with the quantitative analysis of the (social) network of scientific communication.

From the historical perspective, information scientists and bibliometricians proceeded from the same approach. First citation analyses were designed to aid the decision which periodicals should best be purchased by small libraries. Many tools and indicators developed by the ISI were designed for use in scientific information, information retrieval and libraries.

The application to science policy has brought a new perspective, and resulted in re-interpretation of bibliometric conceptions.



Example: The notion of citations in information Example: The notion of citations in information science, bibliometrics and science policyscience, bibliometrics and science policy

Citations became a widely used measure of the impact of scientific publications, but the notion of what citations actually express sheds light on different aspects.

These different views might results in conflicts caused by re-interpretation and “perspective shift”.


Some examples:

According to Merton, citations are intimately connected with the reward system of science.

Cozzens: “Citation is only secondarily a reward system. Primarily, it is rhetorical-part of persuasively arguing for the knowledge claims of the citing document.”

Cronin: Citations are "frozen footprints in the landscape of scholarly achievement … which bear witness to the passage of ideas“.

Glänzel and Schoepflin: Citations give “a formalised account of the information use and can be taken as a strong indicator of reception at this level.”



The process of re-interpreting the notion of citation and its The process of re-interpreting the notion of citation and its consequencesconsequences

Information use

Reward system(quality measure)

Research evaluation/Science policy

Bibliometrics/Information science

citation

uncitednessuncitedness: unused information: unused informationfrequent citefrequent cite: good reception : good reception self-citeself-cite: part of scient. communication: part of scient. communication

interpretation

re-interpretationrepercussion(possible distortion of citation behaviour)

uncitednessuncitedness: low quality: low qualityfrequent citefrequent cite: high quality : high quality self-citeself-cite: distortion of impact: distortion of impact



5.5. BIBLIOMETRICS AS A TOOL IN RESEARCH BIBLIOMETRICS AS A TOOL IN RESEARCH EVALUATIONEVALUATION


5. Research EvaluationResearch Evaluation

Present-day use of bibliometricsPresent-day use of bibliometrics

The evaluation of scientific research performance is the most important application of present-days bibliometrics.

• Growing, often controversial, policy interest to use bibliometric Growing, often controversial, policy interest to use bibliometric techniques as techniques as output measures output measures of research productivity and of research productivity and efficiency.efficiency.

• Thus bibliometrics evolved to a standard tool in science policy and research management.

• Bibliometric components are used, among others, in national science reports, bibliometric indicators appear in the daily press, benchmarking results are used for research management and for allocation of research funding.


What can bibliometrics not be responsible for?What can bibliometrics not be responsible for?

• Bibliometrics can be used to develop and provide tools to be applied to research evaluation, but is not designed to evaluate research results.

• Bibliometrics does not aim at replacing qualitative methods by quantitative approaches.

• Consequently, bibliometrics is not designed to correct or even substitute peer reviews or evaluation by experts but qualitative and quantitative methods in science studies should complement each other.



Distorted behaviour based on policy use and misuse of Distorted behaviour based on policy use and misuse of bibliometric databibliometric data

One issue concerns the changes in the publication, citation and collaboration behaviour of scientists (both positive and negative) that the consistent policy use of bibliometric indicators might potentially induce.

Uninformed use and misuse are not always beyond the responsibility of bibliometricians. Unfortunately, bibliometricians do not always resist the temptation to follow popular, even populist, trends in order to meet the expectations of the customers.

Any kind of uninformed use or misuse of bibliometric results involves the danger of bringing bibliometric research itself into disrepute.



The problem of inappropriate use ranges from uninformed use, over selecting and collecting ‘most advantageous’ indicators to the obvious and deliberate misuse of data.

Uninformed use• incorrect presentation, interpretation of bibliometric

indicators or their use in an inappropriate context caused by insufficient knowledge of methodology, background and data sources

Misuse• intentionally incorrect presentation, interpretation of

bibliometric indicators or their deliberate use in inappropriate context

• tendentious application of biases or tendentious choice of (incompatible) indicators



But even correct use might have undesired consequences

• Author self-citations are shown in an unfavourable light. Authors might thus feel urged avoiding self-citations – a clear intervention into the mechanism of scientific communication.

• Indirect repercussions can be observed when bibliometric tools are used in decision-making and the scientific community recognises the feedback in terms of their funding.

• Butler (2004) showed on the example of Australia what might happen when funding is linked to publication counts. She found that this has stimulated an increased publication activity in the lower-impact journals.



Schematic visualisation of the feedback of policy use of bibliometrics on the scientific community



Possible positive effects• Scientists might recognise that scientific collaboration and

publishing in high-impact or even top journals pays off. Also their publication activity might be stimulated.

Possible negative effects• Exaggerated collaboration, even trends towards hyper-

authorship, inflating publication output by splitting up publications to sequences, inflating citation impact by self-citations and forming citation cliques, etc.

• Trend towards replacing quality and recognition by visibility at any price or towards preferring journals as publication channels in social sciences and humanities might be among these effects.

• A certain “champions league” mentality is spreading among scientists (“Shanghai ranking”, H-index, “IF Filter”).



6.6. BIBLIOMETRIC RESEARCH AND SERVICES BIBLIOMETRIC RESEARCH AND SERVICES

AT THE STEUNPUNT O&O STATISTIEKENAT THE STEUNPUNT O&O STATISTIEKEN


The Steunpunt O&O Statistieken (Research Centre for R&D Statistics – SOOS) was created in January 2002 after open calls for proposals.

SOOS is a unique and dedicated research center to support its science and innovation policy in Flanders.

This center has to develop an appropriate system of quantitative indicators to quantify R&D efforts at Flemish universities, research institutes and industry that can be used by policy makers to support the development of appropriate regional science and innovation policy for Flanders.

6. The Steunpunt O&O StatistiekenThe Steunpunt O&O Statistieken


ACTIVITY OVERVIEW

The activities of SOOS can be classified into four categories:

1. Developing an appropriate IT-infrastructure to support S&T indicator development;

2. Providing the Flemish government with appropriate S&T indicators on which to base its science and technology policy resource allocation decisions;

3. Executing specific studies on science and technology domains or science policy relevant topics for the Flemish government;

4. Doing original research in the field of S&T studies.



LONG- AND MEDIUM-TERM SERVICES FOR THE FLEMISH GOVERNMENT

I. The BOF-key

One of the major funding mechanisms for basic science in Flanders is the BOF-mechanism. The Bijzonder Onderzoeksfonds is a research fund of the Flemish government that is distributed to the Flemish universities on the basis of a specific allocation model. This allocation key comprises both input and output measures on the research performance of the universities involved. Part of the key is based on publication and citation data as derived from WoS by SOOS.



BOF-funding for Flanders amounts to 100 million EUR annually. This amount is distributed in basic science funding amongst the six Flemish universities (LUC/UHasselt, KUB, KUL, VUB, UA and UGent).

II. The IOF-key

The IOF-key is a distribution key to determine the annual allocation of 12 million EUR (to be augmented in the year 2006) amongst the six Flemish universities in order to support applied research.

The IOF-key is partially based on patent statistics derived from the USPTO, EPO and PCT databases. SOOS has been instrumental both in the development of and the data delivery for those distribution instruments.



III. The Flemish Indicatorbook

The Flemish Indicatorbook on Science, Technology and Innovation appears biannually, and does not only support government questions on R&D-proficiency in Flanders, but also aims to supply adequate and up-to-date information on the European R&D-potential of the Flanders region.

Two editions of the Indicatorbook have been published by SOOS so far: 2003 and 2005.



IV. Domain Studies (here bibliometrics only)

Aim of these studies is to conduct and to provide analyses of Flemish research and technology activity in the international context in highly important areas. Expected outcomes are relevant information about performance and competitiveness in Flanders.

Three domain studies have been prepared and published by SooS: Nanoscience and –technology, Biotechnology and molecular biology (both in 2003) and Stem-cell research (2004).

The studies are based on both publication and patent data extracted from the WoS, the USPTO and EPO databases.



Methodological and applied research is closely linked to the service tasks, for instance, in the context of methodological validation, of improvement of the efficiency of indicators or of the subject classification.

‘Basic research’ at the Steunpunt is structured in a systematic way to support policy-relevant research and services and to contribute to the advancement of the field of bibliometrics.

Similarly to the services, research is done at the national/regional, institutional and research-group level.


BIBLIOMETRIC RESEARCH PROJECTS AT SOOS


1. Development of a hierarchic subject classification scheme for evaluation purposes

2. The role of author self-citations in scientific communication

3. The myth of delayed recognition

4. Analysis of possible extension of bibliometric methods to social science and humanities

5. Studying the network of science communication

6. “Indicator research” (improving journal and subject impact measures and science indicators)



7. Bibliometrics beyond the WoS (e.g., proceedings literature)

8. Bibliometric approach to social sciences and humanities

9. Combining text-mining and bibliometrics

10.Mapping academic mobility and scientific collaboration in Europe

11.Research group dynamics and their impact on performance and visibility

12.Bisociation analysis of scientific communication


bibliometrics – use and limitations wolfgang glänzel ku leuven, belgium ispr, has, hungary

Documents

use of bibliometrics

bibliometrics use

wolfgang glnzelbibliometrics

sources of bibliometrics

introduction slide

hungary slide

limitations237 slide

science policy citations