on the limits of bibliometrics

Faculté des arts et des sciences SciELO15 anos

1/15

Faculté des arts et des sciencesÉcole de bibliothéconomie et des sciences de l'information

On the Limits of Bibliometrics

Vincent LarivièreCanada Research Chair on the transformations

of scholarly communicationUniversité de Montréal

SciELO 15 anos

24 Outubro 2013


2/15

Outline of the talk

- Technical limits

- Coverage

- Limits of indicators

- Impact Factor and prediction of citations

- Citation window

- Adverse consequences


3/15Technical limits

- No database is covers all of the published literature.

- Particularly true in the social sciences and humanities.

- Local journals are not indexed by major databases and hence, local topics are not well covered

- No books were indexed (until recently).


4/15

Percentage of references made to journal articles in the natural and medical sciences (Larivière et al. 2006)

55%

60%

65%

70%

75%

80%

85%

90%

95%

1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000

Year

Average SCI Biomedical Research Clinical MedicinePhysics Biology Earth and Space SciencesChemistry Psychology MathematicsEngineering


5/15Percentage of references made to journal articles in the social sciences and humanities (Larivière et al. 2006)


6/15Proportion of references made to conference proceedings in the natural and medical sciences (Lisée

et al. 2008)

0% 5% 10% 15% 20%

Applied Physics

Metals & Metallurgy

General Engineering

Optics

Chemical Engineering

Library & Information Science

Human Factors

Acoustics

Mechanical Engineering

Industrial Engineering

Aerospace Technology

Miscellaneous Engineering & Technology

Nuclear Technology

Civil Engineering

Electrical Engineering & Electronics

Computers


7/15Coverage of journals as a function of the

language of publication (Archambault et al. 2006)

NSE SSH

Thomson ISI Ulrich Difference Thomson ISI Ulrich Difference

English 89% 78% 13% 90% 75% 20%

Czech 0.04% 0.3% -85% 0.2% 0.2% 8%

Russian 0.5% 0.9% -48% 0.3% 0.4% -24%

French 3.3% 3.4% -3% 3.2% 4.4% -26%

Multiple languages 0.2% 0.2% -14% 0.3% 0.5% -45%

Dutch 2.2% 2.2% 0% 1.3% 2.6% -48%

German 3.2% 3.9% -18% 3.0% 5.8% -50%

Japanese 0.4% 1.7% -74% 0.2% 0.6% -64%

Swedish - 0.1% -100% 0.1% 0.4% -69%

Spanish 0.6% 2.6% -75% 0.9% 3.0% -69%

Italian 0.1% 0.8% -83% 0.2% 1.1% -80%

Danish 0.04% 0.1% -50% 0.1% 0.3% -83%

Portuguese 0.1% 0.7% -85% 0.1% 1.0% -86%

Chinese 0.3% 2.4% -88% 0.04% 1.2% -96%

Polish 0.05% 0.7% -92% - 0.9% -100%

Arabic - 0.1% -100% - 0.3% -100%

Turkish 0.01% 0.2% -95% - 0.1% -100%

Other 0.3% 1.6% -80% 0.3% 2.4% -87%

Language


8/15

Median age of cited literature(Archambault et Larivière, 2010)

0

2

4

6

8

10

12

14

16

1980 1985 1990 1995 2000

Âge

méd

ian

de la

litt

érat

ure

cité

e

Année

Arts ethumanités

Sciencessociales

Sciencesnaturelleset génie

Médecine


9/15

How to correct (some of) these limits

- Creation of book databases

- Book Citation Index (but it still has an anglo-american bias!)

- Triangulate data sources

- Scopus vs. Thomson vs. Google Scholar

- Creation of local databases

- SciELO in Brazil

- Érudit au Québec


10/15

Impact factor and its limits

Cannot be compared across disciplines

Skewness of citation distributions

Assymetry between numerator and denominator

Short citation window – especially for SSH

Includes journal self-citations

Is less and less correlated with citations received by papers


11/15

Different citation practices: mean citation rates of papers after 2 and 10 years, 1900-2005

0

1

2

3

4

5

6

7

1900 1920 1940 1960 1980 2000

Mean - 2 year citation window

MED

NSE

SS

0

2

4

6

8

10

12

14

16

18

20

22

24

1900 1920 1940 1960 1980 2000

Mean - 10 year citation window

MED

NSE

SS


12/15

Weakening relationship between Impact Factor and citations (Lozano et al. 2012)

y = 0.0011x - 1.9418r2 = 0.4666

y = 0.003x - 5.7759r2 = 0.579

y = -0.0049x + 9.9863r2 = 0.4788

0,00

0,05

0,10

0,15

0,20

0,25

0,30

0,35

1900 1920 1940 1960 1980 2000 2020

Annu

al c

oeffi

cient

of d

eter

min

atio

n (r2

)


13/15

Decline of top journals (Larivière et al. 2013)


14/15

Adverse effect

- Change of research topics

Type of journal Brazil in abstractBrazilian Journals 31%USA Journals 15%Other Journals 22%


15/15

Archambault, É., Vignola-Gagné, É., Côté., Larivière, V. and Y. Gingras (2006). Benchmarking scientific output in the social sciences and humanities: The limits of existing databases, Scientometrics, 68(3): 329-342.

Larivière, V., Archambault, É, Gingras, Y. and É. Vignola-Gagné (2006). The place of serials in referencing practices: Comparing natural sciences and engineering with social sciences and humanities, Journal of the American Society for Information Science and Technology, 57(8): 997-1004.

Larivière, V., Archambault, É., Gingras, Y. (2008) Long-term variations in the aging of scientific literature: from exponential growth to steady-state science (1900-2004). Journal of the American Society for Information Science and Technology, 59(2): 288–296.

Larivière, V., Gingras, Y. and É. Archambault (2006). Canadian Collaboration Networks: A Comparative Analysis of the Natural Sciences, Social Sciences and the Humanities, Scientometrics, 68(3): 519-533.

Larivière, V. , Lozano, G.A., Gingras, Y. (2013). Are elite journals declining? Forthcoming in the Journal of the American Society for Information Science and Technology.

Lozano, G.A., Larivière, V., Gingras, Y. (2012) The weakening relationship between the Impact Factor and papers’ citations in the digital age. Journal of the American Society for Information Science and Technology. 63(11): 2140-2145.

Lisée, C., Larivière, V., Archambault, É. (2008) Conference proceedings as a source of scientific information: A bibliometric analysis. Journal of the American Society for Information Science and Technology, 59(11): 1776–1784.

Bibliography

on the limits of bibliometrics

Technology

facult des

medical sciences

social sciences

impact factor

information

references

american society

conference