Use of bibliometry for the evaluation of researchers and teams

in medicine and biology

A practical view from a School of Medicine

Dean Patrick Berche

Universitary School of Medicine Paris Descartes

Colloque de l’Académie des sciences "Évolution des publications scientifiques - Le regard des chercheurs" des 14-15 mai 2007

- High diversity of approaches and methodologies, as illustrated by the existence of many disciplines from basic biology to clinical research at the bedside

- Artisanal aspects of this research, contrasting with the necessity of heavy technological platforms and critical mass of researchers required to perform competitive research

- Very high number of scientific journals (>4600)

Main features of Research in Biology & Medicine

1. Importance of basic Research for breakthroughs in medicine: a continuum from cognitive research to clinical research at the bedside

2. Close relationship between Pedagogy and Research : no good pedagogy without research

3. Difficulties to program Research : is it possible to plan it? Many discoveries in medicine and biology have never

been planed, neither financially supported by institutions, at least at the beginning. Examples: HIV, Helicobacter pylori, RNA silencing…

At the most, the institution could define large and blurred fields, related to medical Research

Strategy to develop Research in a School of Medicine

4. Only matter individuals ! EVALUATION Originality, creativity, curiosity, capacity of wondering and questionning with respect to banal or unusual observations, capacity to interact and to create favorable scientific environment.

1. To allow ideally a complete liberty for researchers, without administrative constraints, without predefined scientific frames and priorities…

2. To favor a good scientific environment

- Create critical mass composed of many teams at the same location, with convivial places for meeting!! - Reinforce big technological platforms - Encourage interactions with other scientific fields (physics, chemistry, mathematics…).

What is the role of the institution to boost competitive research ?

3. To identify, evaluate and recruit the best researchers :

It is the main role and responsability of institutions:essential, difficult and dangerous task

The major objective is to improve knowledge

to better treat and cure patients.

- Creativity, originality and innovative potential of a researcher

- Aptitude to interfere with others teams to progress in their work

- Capacity to attract young scientists and to transmit skill and competence

What do we want to evaluate ?

Who do we want to evaluate ?

- Young scientists and medical doctors

- Confirmed scientists and medical doctors: achievements and competitivity of their research teams

- The national and international rank of Research of our Medical School

Hard to evaluate with bibliometric analysis

This evaluation is mainly qualitative:

- Originality of previous works and new project

- Is it a risky Research?

- Choice of the Research laboratory chosen for the PhD thesis

- Assessment of the real contribution of the scientist in the publications related to the PhD thesis

- Quality of publications estimated by reading the original articles - Oral presentation of works and answers to questions

Evaluation of young scientists

The bibliometric analysis must be considered caustiously at this stage, often mainly related to the quality of Research laboratory

Evaluation of senior scientists and associated Research teams

(1) A quantitative bibliometric analysis - The global productivity of a researcher and of a team, especially the

paucity or absence of significant publications over the last 5 years-period. - The number of articles in journals with high impact factors and in the

best journals of the discipline. - A comparison with recently recruited scientists in the same discipline.

(2) A qualitative bibliometric analysis- Proportion of articles associated with a leadind signature rank of scientist (1rst, 2rd , last or corresponding author). - Proportion and rank in the articles reporting original works, compared to reviews, letters, answers… - Citation index in articles > 5 years, but only for articles publishing original contributions where the scientist signs in useful position. - Proportion of joint-authorship articles- The repetition (‘’psittacisme’’) of articles according to the title and abstracts.

The bibliometric analysis allows to globally assess the impact of a senior scientist and his associated team.

Evaluation of senior scientists and associated research teams

Bibliometric analysis

The bibliometric analysis must be cautiously used, without forgetting the pitfalls and limits of this approach.

This analysis is only complementary of the qualitative analysis of original works, looking for the clue and creative ideas of the Research developed in publications, and to the true contribution of scientist in the published works and breakthroughs.

Other important elements for evaluation

- Number of Research grants (industrials, National Public Agencies, international grants [EEC]…)

- Collaborations: quality of national and international teams

- Number of licenses and creation of start-up

How to estimate the scientific production How to estimate the scientific production of an institution as Paris Descartes of an institution as Paris Descartes

Medical School?Medical School?

The estimation of the scientific level of institutions can be only based on publications of original works in useful position in

major journals.

0 20 40 60 80

Mont Ste Geneviève

Institut Pasteur

Université Paris 5

Université Paris 6

Université Paris 7

Université Paris 11

Université Paris 12

Université Paris 13

Université Versailles

Orsay Gif Saclay CEA

Genopole Evry

INRA

7 IF> 25

Nature, Cell, Science, Nature Genetics, Nature

Immunology, Nature Medicine, New England

Journal of Medicine

7724

19

2121

1

105

33

50

49

Articles 2000-2005 in 7 top biological journals IF >25Articles 2000-2005 in 7 top biological journals IF >25Articles 2000-2005 in 7 top biological journals IF >25Articles 2000-2005 in 7 top biological journals IF >25

Useful position of French authors: 1Useful position of French authors: 1rstrst, second and last position , second and last position

75 from 75 from Paris Descartes Paris Descartes Medical SchoolMedical School

0 10 20 30 40 50 60

Mont Ste Geneviève

Institut Pasteur

Université Paris 5

Université Paris 6

Université Paris 7

Université Paris 11

Université Paris 12

Université Paris 13

Université Versailles

Orsay Gif Saclay CEA

Genopole Evry

INRA

4 clin IF 18-30

New England Journal of Medicine,, Lancet, Nature Medicine, JAMALancet, Nature Medicine, JAMA

Publications 2000-2005 in 4 top clinical journals IF 18-35Publications 2000-2005 in 4 top clinical journals IF 18-35Publications 2000-2005 in 4 top clinical journals IF 18-35Publications 2000-2005 in 4 top clinical journals IF 18-35

4

851

2217

157

1

83

1

1

Useful position of French authors: 1Useful position of French authors: 1rstrst, second and last position , second and last position

51 from 51 from Paris Descartes Paris Descartes Medical SchoolMedical School

1. The bibliometric analysis must be performed by discipline, since the impact factors of international journals are related to the extent of the scientific communauty implicated in the field.

2. The signature leadind rank of scientist is crucial : is he a major contributor of the work or project? Is he associated as co-author (which only reflects the capacity to interact, and not the real contribution, including publications in major journals) ?

3. The importance of original works in publications must be clearly estimated. General reviews in major journals only indicate notoriety and recognition, not originality and creativity.

4. The IF of journals are excellent indicators, stable in time and based on the quality of the selection performed by independent international experts. However, major breakthroughs might be published in obscure journals of low IF.

5. The citation index does not distinguish between original works, technical notes, letters, and reviews, neither takes care of the position of authors, auto-quotations… It is an historical indicator requiring several years before significant citations of articles appear . It is useful to measure notoriety of a senior scientist.

Conclusion