bibliometrics and scientometrics

Bibliometrics and scientometrics

Part I: the classical performance indicators

Pablo Achard - NCCR Affective Sciences

Tuesday, February 24, 2009

P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

1.What are we talking about?

2.Basic indicators

3. Performance indicators at the micro level

4.Performance indicators at the meso and macro level


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Definitions

“Bibliometrics” is introduced by Pritchard in 1969:“the application of mathematical and statistical methods to books and other media of communication”

“Scientometrics” is the science of measuring and analyzing science. As such, it includes the bibliometrics of scientific books and articles; but it also takes into account funding, demography, geography, etc.


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

What is it used for

1. Science and Technology Studies (epistemology, science sociology, science history,…) and the scientific study of idea spread

2. Biblioeconomics

3. Science management

4. Rankings (used by journalists, students, scientists,…)


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

What is it used for

Lindberg, PhD Thesis


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Why is it becoming so important

1. Availability of large databases

2. Increased use of management tools in the research administration

3. Globalization of the education market


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

What is this talk focusing on?

The managerial use of performance indicators at the micro level (individuals, groups), and at the meso and macro levels (from departments to continents)


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

What is this talk focusing on?

The managerial use of performance indicators at the micro level (individuals, groups), and at the meso and macro levels (from departments to continents)

Therefore we will not deal with important topics like:- The study of idea spread- Demographics, education, collaborations- Links between Research and Development, public understanding of science


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


2.Basic indicators




P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Direct input indicators

• Number of scientists

• R&D budgets


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Direct output indicators

• Number of publications / books / abstracts

• Number of patents

• Number of PhDs


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Influence indicators

Measure the “passive” effects

• Number of citations

• Number of downloads

• Pagerank of a website


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Number of citations ≠ quality

The number of citations measures the reception

“if a paper receives 5 or 10 citations a year throughout several years after its publication, it is very likely that its content will become integrated into the body of knowledge of the respective subject field; if, on the other hand, no reference is made at all to the paper during 5 to 10 years after publication, it is likely that the results involved do not contribute essentially to the contemporary scientific paradigm system of the subject field in question” (Braun et al. 1985)


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Number of citations ≠ quality

Book evaluation vs citations: “The J-shaped distribution of citedness” (Nicolaisen 2002)


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Number of citations: issues


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Normalized indicators

Number of citations / number of expected citations for similar publications

The “Crown indicator” of Leiden University: Normalized by publication type, year and field


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

The myth of delayed recognition

Papers highly cited only after a period of 5 years = 60 out of 450’000 published in 1980 or 0.013%


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

IF


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

IF

Statistics matter: the mean value is a very rough description of a distribution

(W. Glänzel, 2003)


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Methodological pitfalls

Field delineation

Aggregation level

Time frame

Counting scheme

Data quality


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


2.Basic indicators




P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Warning 1: Mating...

You wouldn’t mary someone based on his/her picture on a mating website

But if you look for partners on a mating website, you should care about the picture you upload (using Photoshop is a matter of personal ethics)


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Warning 1: Mating...

You wouldn’t mary someone based on his/her picture on a mating website

But if you look for partners on a mating website, you should care about the picture you upload (using Photoshop is a matter of personal ethics)

Apply the same rules with evaluating someone’s scientific value and bibliometrics


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Warning 2: Distortion...

1. Funding agencies and science policy makers want to get the best science for their money. But there are more scientists than one can know and more fields than one can understand.

2. They fund / hire preferentially projects / scientists with the best score on a metric that correlates with the ‘quality’ they are looking for.

3. Scientists adapt their publication behavior to increase their score on this metric.


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Individual performance indicators

Number of publications / books / abstracts

Number of patents

Problem: no indication of the influence of the work

Distortions: multiplication of small papers (Least Publishable Unit); “honorary” authors


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


Number of citations received

Problems: one article can make it all (highly skewed); too long to build

Distortions: self-citations; citation exchange


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


H-index: "A scientist has index h if h of his or her Np papers have at least h citations each and the other (Np - h) papers have fewer than h citations each"


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


H-index: "A scientist has index h if h of his or her Np papers have at least h citations each and the other (Np - h) papers have fewer than h citations each"

Problems: very strong correlation with the number of publications; perfect to compare scientists... at the end of their career!

Distortions: same as publication numbers


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


Many variants of the h-index


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


Number of publications x IF of the journal in which they where published

Problems: highly noisy; very field-dependent; IF doesn’t determine future citation


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

The field dependency of IF

(Leydesdorff, 2008)


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

IF-citation correlation?

(Seglen, 1997)

Number of citations vs IF of the journal in which articles were published for each and every article of 4 researchers


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


Number of publications x IF of the journal in which they where published

Problems: highly noisy; very field-dependent; IF doesn’t determine future citation

Distortions: editors “hot topics”; worse at the macroscopic level


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

How to increase the IF of your university?


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


Just cut the departments with low citation rates!


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


Just cut the departments with low citation rates!

Do you think I’m kidding?


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


www.phdcomics.comTuesday, February 24, 2009

http://www.phdcomics.com

http://www.phdcomics.com

P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”


2.Basic indicators




P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

University rankings

Shanghai = alumni awards score + staff awards score + highly cited score + Nature and Science articles + articles + a mixture of all this/number of faculty

Times Higher Education = peer-review score + employer-review score + staff/student score + citation/staff score + international student score

UNIGE ranking 2007 = 105 ranking 2008 = 68


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

University rankings

Shanghai = alumni awards score + staff awards score + highly cited score + Nature and Science articles + articles + a mixture of all this/number of faculty

Times Higher Education = peer-review score + employer-review score + staff/student score + citation/staff score + international student score

UNIGE ranking 2007 = 105 ranking 2008 = 68

Let’s have a look at more serious indicators!


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Publications per country


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Publications vs GDP


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Citations vs GDP (normalized)


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Field dependency (again)

Clinical medicine (MED)Biomedical research (BRE)Biology (BIO)Chemistry (CHE)Physics (PHY)Mathematics (MAT)Engineering (ENG)Earth and space sciences (ESS)

(Glänzel, 2003)


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Citations vs GDP (normalized)


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

Conclusions?


P. A

char

d “B

iblio

met

rics

and

sci

ento

met

rics

”

“The use of a single index crashes the multidimensional space of bibliometrics into one single dimension.” (Wolfgang Glänzel)

“Not everything that can be counted counts, and not everything that counts can be counted.” (Albert Einstein)
