LIC 1

Jan Reedijk, Leiden University

The value and accuracy of key figures in

scientific evaluations

Wenner Gren Symposium, Stockholm, May 23-25, 2013

Reedijk@chem.leidenuniv.nl

LIC 2

Why this title? • Editorial Boards of many scientific journals

appear, increasingly, to develop an unhealthy attitude and mentality to improve the Impact Factor of the journal.

• In many organisations scientists are mainly evaluated on the basis of citations, or – even worse – the citation levels (IF) of the journals in which they publish.

• In certain universities scientists are financed (grants and salary) via a mathematical formula in which impact factors of journals occur and/or h indices of the involved scientists.

• Errors in impact-analyses and citations can be quite large, and are also prone to manipulation.

LIC 3

Contents• Introduction • Definitions • Methods for ranking• Citation analysis: Examples (if

possible via online demonstrations)• Impact factor (Thomson-Reuters IF, TRIF) • Hirsch index (h)• Examples of errors, misuse and abuse• Recommendations and Final Remarks

LIC 4

Introductory comments• Citation and citation habits: vary by field• Impact factors of journals: vary strongly by

research field; • Group analyses: requires professionals• Personal Analyses: risky• So-called h indices per person: risky• Career (age) corrected h factors (many by

now): R-factor, g factor, hc factor, m-factor; • Effects due to non-citing or over-citing by

some authors: hard to analyse• Matthew effect: some publishers now

mention in the html/online version of a paper, how many times it has already been cited by “today”.

LIC 5

Recent uses of Citation analysis

• By librarians: which journals (not) to abort

• By university administrations: where to cut, or how much to pay groups, as research budgets

• By personnel departments: whom to promote• By hiring committees: selection of candidates • By research councils: who will get big grant?• #: all these often performed incompetently

• By professional scientists (scientometrics; bibliometrics): A few top institutions worldwide.

LIC 6

Classical errors ISI-WoS (now Thomson Reuters)

• Name of the institution incorrect, or missing, or changing over time: e.g. A.State University Leiden, 6305 hits B. Leiden State University, 9116 hits C. University Leiden, 20066 hits D.Leiden University, 57899 hits

• (all checked by May 22, 2013)C or D: 71814 (NOT 20066+57899)

• Names of AUTHORS mis-spelled; initials incomplete, or incorrect

LIC 7

Other known but often neglected errors in WoS

• Missing citations (i.e. those with an error) or not covered documents; citations to non-WoS journals, are ONLY allocated to first author, or totally neglected;

• Authors copy references from others, even if they are wrong: examples are numerous.

• Other Errors: less rich people use Scopus (Elsevier), or Google Scholar; even worse.But: Google includes patent applications and also many “soft” documents that are not research papers.

LIC 8

Sources of errors in WoS• Copying incorrect citations; or errors made by

authors in typing citation lists at the end of papers; some errors have over 20 identical repeats!

• Errors made by WoS in entering publications and citations in data bases, and in classifying citations and accrediting them to journals and authors; one letter wrong can be dramatic.

• Incomplete and misleading impact figures published by WoS (like citations to editorials; double counting for 2 languages, e.g. in Angewandte Chemie; counting certain citable notes as “non-citable” e.g. New Engl. J. Med).

LIC 9

A Classical case: Angewandte Chemie

• Two editions: English and German (but they concern the same papers)

• Wiley journal editors often require citations to BOTH German and English papers (to assist the German Readers of Angew.)

• WoS counts citations in total: So many papers published in Angewandte obtain significantly more citations (and therefore some 10-15% higher IF, and consequently the journal is also higher ranked)

LIC 10

Origin of citation analyses

• To find out who has used the results described in an (earlier) paper (“snowball effect’)

• This is still of great value for scientists! (… and not taught enough to students)

LIC 11

Scientists use WoS or Scopus for:

• Bibliometrics and Scientometrics: relations between the several fields

• Chemists, physicists, etc.: see the follow up of older (specific) literature

• Nowadays: Importance of citation numbers has developed to almost a fetisjism

LIC 12

Ranking• How does one create a ranking?• Of countries, • Of universities, • Of Faculties/institutes• Of journals (impact factor; also journal h)• Of research groups (h)• Of individual scientists (h; integrated

Impact Factor values)

LIC 13

Known rankings of complete universities

• Times: interviews of other count heavily in the ranking

• Sjanghai: past performances count heavily

• Leiden ranking: Only last 5 years; and including size of the university.

LIC 14

Risks and errors in analysis

Although quite a few bona fide and competent analysts and organisations specialized in citation analyses exist, the incompetence of many “analysts”, when using crude WoS data (or Scopus or Google Scholar) in discussing rankings of journal and/or authors, is an important factor that makes such analyses often unreliable.

LIC 15

Definition of IF (impact factor)• The common impact factor for journals is

covering a period of 2 years.• This number may change each year; • For the year 2007, the IF (published in July

2008), is defined as: The citations in 2007 (in any journal) to papers in THIS journal published in 2005 and 2006 divided by the number of appeared citable papers in those years.

• The IF(TRIF) is an average and not valid for any paper in the journal! Sometimes: explosions lasting 2 years, just due to a single paper.

LIC 16

Effect of an “explosion paper” on IF

• The effect of a single paper with astronomic citations, can disturb the TRIF for 2 years. Example:

• Paper from Sheldrick, 2008: Citations in 2009, 2010 to this paper were: 11828. NB: In 2011 still 8181 (but these do NOT add to TRIF 2009 and 2010)

• All other papers together from 2008 in that journal collected - for those 2 years - the number of only 569 citations!

LIC 17

Effect of explosion in citations

Thomson-Reuters Impact Factor (TRIF) for Acta Cryst. A, 2007-2011; the increases in 2009 and 2010 result from many citations to a single 2008 paper authored by G.M. Sheldrick.

LIC 18

Impact euphoria and TRIF-Mania

• Annual self-praising summer emails from publishing houses and journal staff:

• We are pleased to announce that once again almost 70% of our journals published in ****** and related subjects show a year-on-year increase in their 2-year Impact Factor, demonstrating the outstanding quality standards we apply to our products.

• We are pleased to bring you the latest TRIF figures from our **** journals. Thanks to the contributions of our authors, reviewers and editors, we publish 10 out of the top 20 Impact Factor . 

• This citation-obsessed world gradually invites to manipulation and perhaps even fraud

LIC 19

Hints for Journal editors and publishers:

• Beware of (even small) name changes in your journals. It may cost you citations, and consequently library subscriptions and even contributions of high-level articles. Never combine Volume numbers.

• Give accurate and unambiguous instructions to your readers how to cite papers in your journal! Be prepared to give instructions to your referees to check whether the citation list at the end of the paper is well balanced.

• To increase TRIF, make your journal more attractive to readers (citers) by raising the quality of papers. If you try to gain from increasing TRIF, this generates a motivation to use other, unethical strategies.

LIC 20

The devil in the rankings• University rectors

want a high ranking• So: scientists are

encouraged to publish in journals with high impact and scientists want to increase “h”: better grants, higher salaries

• Result: strategic behaviour, unethical behaviour (“you cite me, I cite you”)

• Journal editors want a high Impact Factor: good for status, attract better papers, and bonuses for editors

• So: editors invent tricks to improve the TRIF (reviews, cartels, editorials, “uncitable” papers being cited, etc.

• Result: Thomson-Reuters may de-list the journal from TRIF tables (for some time)

LIC 21

Non-ethical, or borderline behaviour to increase IF

• Require all authors to include at least 2 (recent) references to THIS journal in each paper.

• Write editorials (or apparent reviews) where you highlight the “best” papers from your own journalPublish the same editorial in a sister journal.

• Have hot reviews/papers printed in first issues each year, and appearing in Dec. of previous year.

• Start a new kind of discussion-inviting paper which is considered by ISI as “non-citable”, but which is neverthless likely to be (highly) cited.

LIC 22

The editorials do effect TRIF • Substantially increasing a journal’s TRIF

with citations from its own editorials does occur, though not on a large scale, at least from a study for 2004

• ‘Intra-publisher’ editorial citations should be further analysed (like cartel citations)

• NB: In 2012 some 50 journals have been de-listed (suppressed) by Thomson Reuters and (temporarily) lost their TRIF

LIC 23

Some used policies to increase the (TR)IF

• Editorials highlighting the best papers in the own journal from the last 2 years; but penalty from TR may be severe: de-listed for some time

• Publish more reviews, or other types of manuscripts that may invite citations in the next 2 years

• Intra-publisher citation stimulation; citation cartels? For a case and discussion see e.g.: http://scholarlykitchen.sspnet.org/2012/04/10/emergence-of-a-citation-cartel/

LIC 24

Fabricating Citation data• Selective (non-)citations to certain scientists

• Extensive self-citations of authors and journal editors in prefaces (often these can be corrected for)

• Citation cartels? “Medical Science Monitor” and “Cell Transplantation” are suppressed from the JCR (Impact factor listing) of Thomson Reuters, due to “anomalous citation behaviour"!! (And several innocent authors lost standing and salary, as their paper suddenly was not in a listed “ISI” journal anymore!)

LIC 25

Analysis of scientific output of persons

The h index (2005) and also very many (about

30) variations

LIC 26

H index and more (introduction)

• The h-index was devised by physicist Jorge Hirsch in 2005 to measure research impact.

• A scientist’s h-index is the highest number of papers they have published which have each obtained at least that number of citations:

• Example: E J Corey, with an h of 143, has published 143 papers which have each received at least 143 citations. Record holder: G. Whitesides: h=181 (as per May 1, 2013)

• This h index can only grow and will never decrease (Eugene Garfield has 124)

• Old people cannot be compared with young ones in the same field.

LIC 27

Recent example of errors in h• The h index of the top 500 (living) chemists

published by the RSC; free available annual pdf.• However, many mistakes: e.g. they neglected

all Dutch/Belgian people with names containing: “van”, “de”, “van der” etc. (in 1997 ISI changed the format from “deboer” to “de Boer”)So in WoS: 2 searches are needed for such names: van Koten G: 370, vankoten G: 472 papers (by May 2013)

• Last 10 years: Most cited chemist(s): Wang, Zhang, Liu, etc.) are all multiple authors.

• Different sources than WoS are also used; Google scholar and Scopus (Elsevier)

LIC 28

Comments on h index calculations

• Web of Science is accurate, but excludes patents, and many book chapters

• Scopus is not covering all literature before 1996, but includes book chapters and patents.

• Google Scholar h values can only be calculated accurately from a personal profile. It does include book chapters and patents and many other (non-research) documents, websites,etc.

• The use of “Publish or Parish” software (Harzing) for Google Scholar is less accurate and finds too many documents that are not research papers, even though they are cited.

LIC 29

Differences in h index calculations

Scientist WOS SCOPUS Google-HarzingPapers h Papers h documents h

Author 1 587 83 553 75 886 89Author 2 409 74 343 65 760 83

Author 3 356 55 377 54 527 58Author 4 259 48 231 41 314 47Author 5 224 33 180 29 251 33Author 6 291 39 295 41 352 41Author 7 676 88 584 81 888 91Author 8 208 47 198 31 297 43Author 9 273 92 176 88 342 96Author 10 570 68 464 57 757 68Author 11 165 34 171 30 222 38NB Google Scholar h values can only be calculated accurately from a personal profile.

LIC 30

Author recommendations• Never change your name (females, hispanics)

or modify your initials; keep them always the same in all of your papers.

• Make sure you have a chance to check your bibliometric data, before it is used in any assessment.

• If your h is asked, always mention date + source

• Do not rank the journals too much on published (2-year) TRIF factors, and do not rely on such inaccurate numbers in deciding where to publish.

• It is to be regretted that as long as politicians primarily judge science quality based on “covers” (=TRIF) of journals, scientists will chose for such journals. New SNIP is far better.

LIC 31

Research councils and science policy makers:

• Do not rank the quality of scientists based on (often inaccurate) citation ISI statistics and h values only. Consult anyway experts and do not rely on crude ISI data only! If you really want to use citation information in your analyses, you better do it good, otherwise don’t do it all.

• And: always let the scientists themselves check the data used in evaluations.

LIC 32

San Francisco Declation (May 17)

• Recommendations Research Assessment:Journal-based matrics are surrogates to measure quality of researchers

• Advice for: funding agencies and institutions

• Recommendations for: Publishers, WoS-Scopus-GoogleS and for Researchers

LIC 33

Recommendations for researchers1. When involved in committees making decisions about funding, hiring, tenure, or promotion, make assessments based on scientific content rather than publication metrics.2. Wherever appropriate, cite primary literature in which observations are first reported rather than reviews in order to give credit where credit is due.3. Use a range of article metrics and indicators on personal/supporting statements, as evidence of the impact of individual published articles and other research outputs . 4. Challenge research assessment practices that rely inappropriately on Journal Impact Factors and promote best practice that focuses on the value and influence of specific research outputs.

LIC 34

Final Remarks• One should never judge persons and

groups of persons on the base of citations only (h; integrated IFs).

• When one uses citations in evaluations: use them only by analysis of an expert and after verification of the data and numbers by those who are being evaluated.

• Imfact factors of journals are averages, field dependent and prone to manipulation; so these numbers as such are not suitable for ranking of journals.

LIC 35

When my h increases, the rest will follow!?

BC Gibb, Nature Chemistry, 4, 513–514 (2012)

LIC 36

Albert Einstein:Not all what counts can be

counted, and not all what can be

counted, counts!