possible inaccuracies occurring in citation analysis · relevant scientific documents. in this...

95

Possible inaccuracies occurringin citation analysis

H.F. Moed and M. VriensLISBON Institute, Science Studies Unit. University of Leiden,Stationsplein 242, 2312 AR Leiden, The Netherlands

Received 31 August 1988

Citation analysis of scientific articles constitutes an im-portant tool in quantitative studies of science and technology.Moreover, citation indexes are used frequently in searches forrelevant scientific documents. In this article we focus on the

issue of reliability of citation analysis. How accurate are cita-tion counts to individual scientific articles? What pitfalls mightoccur in the process of data collection? To what extent do

’random’ or ’systematic’ errors affect the results of the citationanalysis?

We present a detailed analysis of discrepancies betweentarget articles and cited references with respect to author

names, publication year, volume number, and starting pagenumber. Our data consist of some 4500 target articles pub-lished in five scientific journals, and 25000 citations to thesearticles. Both target and citation data were obtained from theScience Citation Index, produced by the Institute for ScientificInformation.

It appears that in many cases a specific error in a citationto a particular target article occurs in more than one citingpublication. We present evidence that authors in compilingreference lists, may copy references from reference lists in

other articles, and that this may be one of the mechanisms

underlying this phenomenon of multiple’ variations/errors.

1. Introduction

Citation analysis constitutes an important toolin quantitative studies of science and technology.Moreover, citation indexes are used frequently insearches for relevant scientific documents on a

certain topic. Authors of scientific publicationsrefer to earlier publications that bear some rele-vance to the claims the authors want to make. In

the list of cited references the author should indi-cate a publication in such a way that it is uniquelydefined, thus enabling a reader to look it up easily.In citation analysis, one creates a set of scientificarticles, and wishes to determine the number of

times each of these articles is cited during a cer-tain period of time. To be more specific, theanalyst wants to determine how frequently eacharticle occurs in the lists of cited references of

some collection of other publications from thescientific literature. The articles for which the

frequency of citation is to be determined will becalled target articles throughout this paper, sincethey constitute the ’targets’ in the referencingprocess. The publications that cite a specific targetarticle-that is, that contain a cited reference tothe target article-will be indicated as source pub-lications.

The above type of citation analysis is per-formed in most cases with the use of computerizeddatabases on scientific literature, created by theInstitute for Scientific Information (ISI), in

Philadelphia. ISI compiles three large databasesthat contain information on the cited references in

source articles, published in a large number ofscientific journals: the Science Citation Index(SCI), covering the Natural Sciences and Life Sci-ences, the Social Science Citation Index (SSCI) forthe Social Sciences, and finally the Arts and Hu-manities Citation Index (A&HCI). ISI publishesthe contents of the databases in a printed form. Inaddition, these databases are made available online

by several host computer organizations. Thesehosts store the databases on computer memoryand offer telecommunication facilities and soft-

ware to search the databases for relevant scientific

material. The online version of the SCI is called

SCISEARCH.ISI extracts the following information from a

cited reference:- the first author:- the title of the publication medium (journal,

book, report);- the year of publication;- (if present) the volume number:- (if present) the starting page number.

Suppose we have created a set of target articles,and for each article we have gathered all relevantbibliographic information, i.e. information on

at Harvard Libraries on April 21, 2015jis.sagepub.comDownloaded from

http://jis.sagepub.com/

96

Table 1

Example of discrepancies between target article and citedreferences

authors, journal title, publication year. volumeand starting page number. Other information maybe included as well, such as the title or data on theinstitutional affiliations of the authors (corporatesource data), but these types of information arenot relevant in the citation analysis itself. Gener-ally speaking, a citation to a specific target articleis identified by matching (either manually or bymeans of computer programs) the information oncited references extracted by ISI with the ap-propriate information on the target article.

In the data on target articles or cited references,errors and/or variations may occur with respectto the author names, journal title, publicationyear, volume and starting page number. Thesetypes of information will be called datafields. As a

consequence, discrepancies arise between the bib-liographic information of the target and the citedreference, although it may be clear-or at leasthighly probable-that the cited reference is actu-ally a citation to a particular target.

In Table I we present some typical examples ofdiscrepancies that may arise with respect to authorname and volume number. The target article con-cerned is cited 57 times in references not showinga discrepancy. However, in six references theauthor name is spelled differently. In three cases,the volume number shows a variation.

In this article, we will analyse discrepancies asindicated in Table 1. Questions that will be ad-dressed are:- How frequently do discrepancies between

targets and cited references occur?- Which types of discrepancies do occur?- Can we identify particular characteristics of

target articles that increase the probability ofbeing cited erroneously? For instance, are therespecific author names that are often misspelled?

- Do highly cited articles receive more citationsshowing discrepancies than articles cited lessfrequently?

In the example given in Table 1 we observed thatthe discrepancy ’SCHOEMAKER-SHOE-MAKER’ occurs in more than one cited reference

to the target article. We will indicate this phe-nomenon as ’multiple discrepancies’. In two casestudies we examined whether evidence could beobtained that copying references from otherarticles may be a cause underlying this phenome-non.

In our analysis we match (in different ways) adataset with data on target articles with a dataset

containing citations to these targets, both extractedin computer-readable form from SCISEARCH,and we analyse ’discrepancies’ between the vari-ous datafields in the two datasets. As a conse-

quence, our analyses are primarily related to theinternal consistency of the SCISEARCH data-base. I discrepancies between target articles andcited references in our database are detected, the

following possibilities exist:1. the data on the target articles contain a varia-

tion/error:2. the data on the cited reference contain a varia-

tion/ error.The variations or errors may occur in several

stages of the process of data-handling, and we candistinguish at least three possible sources:a. the scientific publisher or the authors of the

articles made the variation/error; if so, thevariation/error is present in the printed text ofthe article;

b. the variation/error occurred at ISI, either whenthe data were key-punched into the computer,or when computer manipulations on the datawere performed;

c. the error or variation occurred at the host

organization that brings SCISEARCH online(in our case: DIMDI), possibly in the processof loading the database.In order to obtain some insight into the sources

of the discrepancies, we compared in a limitednumber of cases the information in our datasets to

the ’original’ printed articles in the journals.The structure of this paper is as follows. Sec-

tion 2 deals with data collection and handling. Wewill descrihe the datasets that were used. In Sec-

tion 3 we give an outline of the methods appliedin our analysis. The various match-keys we ap-



97

plied will be defined, and a typology of dis-crepancies will be presented. The results of ouranalysis and a discussion are presented in Section4. Finally, a general discussion of the results andconclusions follows in Section 5.

2. Data collection and handling

Our data source is SCISEARCH, the onlineversion of the Science Citation Index, imple-mented in the host computer of the DeutschesInstitut fur Medizinische Dokumentation und

Information (DIMDI) at Cologne (FRG). Our setof target articles consists of articles, published infive scientific journals in one or two particularyears:- Journal of Nuclear Materials (JNM), years:

1982-1983;- Surface Science (SS), year: 1982;- European Journal of Pharmacology ( EJP),

years: 1983-1984;- British Journal of Pharmacology (BJP), years:

1983-1984;- Journal of Pharmacology and Experimental

Therapeutics (PET), years: 1983-1984.The first two journals are from the field of

materials science, while the other three belong tothe field of pharmacology.

The citation data were collected by selecting allreferences in which the title of the (cited) journaland the (cited) publication year correspond tothose of the articles in the target dataset. Sincecited journal titles are not unified in SCI-SEARCH, we had to determine in advance underwhich variations the titles of our journals ap-peared. Possibly, we overlooked some variationsthat are hardly ‘ predictable’ and that do not occurfrequently. Consequently, we may have missedsome citations to our target articles. However,these omissions do not affect the analyses pre-sented in this paper. The principal point here isthat the cited references were selected indepen-der~tlv of the first author name, volume numberand starting page number. Therefore, our databaseis appropriate for an analysis of discrepanciesbetween target articles and cited references with

respect to each of these datafields.Both target articles and selected cited refer-

ences were downloaded from SCISEARCH and

stored in two data libraries, containing separate

variables for the first author, volume number,starting page number and publication year. Thetwo data libraries were match-merged, using dif-ferent match-keys. Details of this process ofsuccessive match-merging are given in the nextsection. A detailed description of our method ofdata collection and handling is given in [1].

3. Method

In a computerized analysis we identified firstthe cited references in which all datafields (fullauthor name, publication year, starting page num-ber and volume number) were identical to the

corresponding datafields of specific articles in ourtarget dataset. Those citations showing no dis-crepancy at all will be symbolized by C throughoutthis paper, and in some cases will be indicated as

‘correct’ citations. Next, we created a subset of allthose cited references that showed discrepancies inat least one datafield.

Using this subset of citations not matched inthe first step, we determined in the second step thecited references that contained discrepancies inone datafield only. These citations will be sym-bolized as C°, and in some cases be indicated asone-datafield discrepancies. This analysis was per-formed with respect to first author name, volumenumber and page number. Technically, this wasachieved by running several match programs inwhich each datafield was successively excludedfrom the match-key. In 5% of the cases, a citedreference was matched to several target articlesrather than one. These cases were examined man-

ually, and the most probable correct match wasidentified. For each individual target article wedetermined the number of received citations,

showing the various types of discrepancies.Finally, we created a subset of cited references

not matched in the previous steps. These refer-ences contain discrepancies in at least two data-fields (e.g., full author name and volume number).In this set we identified only citations that do not

contain an indication of the volume number andthe page number. These are probably citations toarticles that were not yet published at the time theciting publication was printed. We will indicatethese citations as ’premature’. Further analyseswith these citations were not performed, since inmany cases it appeared to be very problematic to



98

establish which particular target article was in-tended to be cited. The results we present in thenext sections relate only to discrepancies in atmost one datafield.

With respect to the one-datafield discrepancieswe made a typology of discrepancies occurring inthe various datafields. We divided all cases into 19

sub-types, using a combined computerized andmanual approach. To give a few examples, with

respect to discrepancies between author names oftarget and of cited reference, we distinguishedbetween ’small’ differences (e.g. ‘AKASU’ vs.’AKAZU’), and completely different names(‘HAGGBLAD’ vs. ’PEDATA’), on the basis ofvisual inspection. In addition, we identified casesin which the cited author is co-author but not thefirst author of the (most probably) correspondingtarget article. Similar sub-types were distinguished

Table 2

Sub-types of discrepancies between targert articles and cited references



99

with respect to discrepancies in other datafields. Acomplete overview of all sub-types is presented inTable 2. With respect to sub-type Vl, we give amore detailed explanation. Scientific journals maypublish proceedings of scientific meetings or con-ferences. In some cases such proceedings are pub-lished in two journal volumes rather than one(e.g., vol. 108 and 109). We examined whetherauthors, in citing articles in such combinedvolumes indicate the wrong volume number, in thesense that they cite vol. 108 if the target article ispublished in vol. 109, or vol. 109 instead of vol.108.

For each sub-type we determined the frequencyof occurrence in our datasets, in order to identifythe sub-types that occur most frequently. In ad-dition, these data were used to analyse possiblecauses of the substantial differences that appear toexist among the five journals considered.

4. Results and discussion

4.1. Analvsis by type of discrepancv

The overall figures from our analyses are pre-sented in Table 3. As can be seen from this table,in the first step of the matching process we

matched 24433 citations to our 4514 target articles

published in all five journals. It appears that,relative to this total number of citations with no

discrepancies at all, 9.4% of the citations in ourcited reference dataset show a discrepancy in atleast one datafield. Roughly speaking, for everyten citations containing a bibliographic descrip-tion identical to that of a particular article in ourtarget dataset, one citation shows some kind of

discrepancy.Aggregating all journals, the percentages of dis-

crepancies between either author name, page num-ber or volume number are 2.8%. 1.4’~, and 1.0%

respectively. The ’premature’ citations contribute2%, and all other citations account for 2.3%.

Inspecting the results for each journal sep-arately, it appears that large variations exist amongthe journals. JNM shows the highest percentagesin most categories. Considering the other journals,EJP and SS have a high percentage of discrepan-cies between author names. SS and PET show a

high percentage of discrepancies between pagenumbers and volume numbers respectively. BJPhas the highest percentage of premature citations.Possibly the period between acceptance of anarticle by the editor and the actual publicationdate is longer for this journal than for the otherjournals.

Table 3

Overall figures



100

In order to obtain more insight into the natureof the discrepancies and into possible causes un-derlying the differences among the journals, wedivided the one-datafield discrepancies into 19

sub-types. The definition of these sub-types isgiven in Table 2 in Section 3. Some sub-types maybe expected to occur potentially with respect to alltarget articles in our dataset, whilst other sub-types

Table 4

Discrepancies by sub-type d

&dquo; C: the number of ’correct’ citations to all articles in the journal; C,: the number of ’correct’ citations to the subset of articles thatcould potentially generate the specific discrepancy; CD: the number of citations showing the specific discrepancy and given to thearticles in the particular subset.



101

will occur only with respect to a specific subset oftargets. For instance, small variations in volumeor starting page number (sub-type P4) can beassumed to occur in citations to each article, sinceall articles have a page number in their biblio-

graphic description. However, discrepancies oftype P3, in which target articles contain an ’L’ inthe page number field while in the cited referencethe ’L’ is omitted, obviously may occur only withrespect to target articles published in ’letter’ sec-tions of a journal. Therefore, for each sub-type wefirst established the subset of target articles thatcould potentially generate the discrepancy of thatsub-type. With respect to those sub-types for whichwe had no a priori grounds to restrict the occur-rence to certain classes of target articles, we con-sidered the set of all target articles published in aparticular journal.

The results are presented in Table 4. For eachjournal and sub-type three parameters are given.The first ( C~/C ) indicates the percentage of ’cor-rect’ citations to target articles in the subset thatcould potentially generate the sub-type of dis-crepancy (C,), relative to the total number of’correct’ citations to all articles in the journal (C).If the subset of target articles that could generatethe discrepancy is equal to the set of all articles inthe journals, C equals C~ and CIS amounts to100%. The second parameter (C D) indicates theabsolute number of citations-showing the specificdiscrepancy-to target articles in the subset. Fi-nally, the percentage of C °/C, is given.To given an example, 38% of the ’correct’ cita-

tions to all articles in the journal BJP are citationsto articles of which the first author name is of type’BEDDELL’ or ’SCHOEMAKER’ (A8). Thesearticles are cited seven times in cited references inwhich the author is of the type ’BEDELL’ or’SHOEMAKER’. These seven citations constitute

0.5% of all citations given to the specific type oftargets and not showing any discrepancy. Withrespect to sub-type P3, one observes that 15% ofall ’correct’ citations to the articles in SS, are to

targets that contain an ’L’ in the page numberfield, while the other journals do not contain thistype of article. The letters receive 12 citations inwhich the ’L’ is omitted or substituted by our. andthat constitute 2% of all ’correct’ citations of theseletters.

Table 3 shows that JNM has the highest per-centages of discrepancies between author name,

volume and page number among our five journals.Inspecting Table 4, one observes that the highpercentage of discrepancies between volume num-bers is partly explained by the ’combined volumeproblem’ (sub-type VI), since only JNM containscombined volumes. However, the percentages formost sub-types related to author names and pagenumbers are still high, relative to the other jour-nals. We are not able to identify anv particularcharacteristic of the target articles that might ex-plain the high percentages for JNM in these twodatafields. Probably the explanation has to befound in characteristics of the sources citing ourtarget articles. We suggest that in the field covered

by JN M - nuclear fusion and fissionresearch-proceedings of scientific meetings playan important role in the communication, and thatJNM receives relativelv more citations fromcamera-ready produced proceedings, published injournals processed for the SCI. In fact, JNM itselfcontains several of these proceedings in later years.Articles in camera-ready proceedings are probably’checked’ less accurately in the process ofcopy-editing than articles in typeset volumes, or iferrors in the text are detected, thev may not becorrected in all cases. As a consequence, citedreferences in such articles may contain more er-rors. Below, we consider only the other four jour-nals.

The relatively high percentage of discrepanciesbetween page numbers for SS can be explained atleast partly by the fact that SS is the only journalin our sample with a letter section. PET’s highpercentage of discrepancies between volume num-bers is explained only partly by sub-type V2. Infact, PET has a high percentage of citations toarticles with volume numbers of sub-type V2,compared to BJP, EJP and JNM (80% against 0%,5% and 38% respectively). However, for PET andSS these percentages of citations to targets in thissubset are hardly different (80 ~ against 79%),while the percentages of ’erroneous’ citations tothese targets differ by a factor of almost 10 (0.9vs. 0.1). We do not have an explanation for thedifferences with respect to this variation amongPET and SS.A striking result is that the percentage of dis-

crepancies between author names in PET and BJPis low compared to the other journals. We foundthat the articles in BJP and PET emanate mainlyfrom the United ICingdom and the USA respec-



102

tively, while the other journals publish morearticles from non-English speaking countries (par-ticularly the other European countries and Japan).We conjectured that the native language of theauthors is a factor of significance, but we did notfind strong evidence in favour of this hypothesis.The high overall percentage of discrepancies be-tween author names in EJP appears to be caused

to a large extent by sub-type Al. However, thissub-type is rather trivial. The discrepancies aredue to the fact that the name of the cited author is

in most cases abbreviated, at least in the SCI-SEARCH database. As a consequence, targetarticles with too long author names that are citedcorrectly, have a high probability of showing thistype of discrepancy. Obviously, this should bekept in mind in citation searches using this data-base. The high overall percentage for SS is causedmainly by sub-type A2. We do not have anexplanation why the number of citations to co-authors instead of first authors is relatively highfor this journal. Possibly it is a characteristic ofthe field (materials science, physics), in the sensethat hierarchical relationships between researchersmay be stronger in this field than in biomedicalfields. As a consequence, a stronger tendency ex-ists to cite the ’senior’ rather than the ’junior’researcher, even if the senior is not the first author.We calculated the mean value of the variable

C~/€,(%). based on the scores of the individualjournals. These statistics provide a rough estimateof the percentage of citations showing discrepan-cies to target articles in each class. The mean

Fig. 1. Scores for main sub-types of discrepancies.C~: The number of ’correct’ citations to the subset of articlesthat could potentially generate the specific discrepancy; C’°:The number of citations with the specific discrepancy. Thesub-types of discrepancies are mdicated by the particular ex-amples, presented in Table 2.

percentages for the main sub-types of discrepan-cies are summarized in Fig. 1. The sub-types areindicated by the corresponding example fromTable 2. Mean percentages of discrepancies higherthan 1.0 are found for articles in letter sections

(sub-type P3), for author names containing ‘ van’,’of. ‘Mac’ (A6) or with more than one initial (A4),and for targets in combined volumes (VI). Itshould be noted that the standard deviations are

rather high and exceed in many cases the mean.

4.2. Discrepancies hetween puhlication years

As indicated in Section 2, we selected the citedreferences in which the publication year corre-sponds to the year of publication of the targetarticles in our target dataset. For instance, weselected all citations to JNM containing 1982 or1983 as cited publication year. Counting the num-ber of citations in our cited reference dataset to

JNM by volume, we detected that two particularvolumes received hardly any citations. The articlesin these two volumes had publication year 1982 inour target dataset. However, inspecting SCI-SEARCH online we found 931 citations to these

volumes, but the cited publication year was 1981! tFor this reason these citations were not present inour dataset of cited references. In fact, these cita-tions are not included in the analyses presented inthis paper.

The percentage of citations to JNM showingthis discrepancy, relative to the number of cita-tions not showing any discrepancy is 46%. Thecause of the discrepancy is the following: the twovolumes are proceedings volumes and were an-nounced to be published in 1981. Due to severalreasons, they were actually published in 1982. Inthe heading of each article, 1981 was indicated aspublication year. On the cover of the volume,however, 1982 appeared. ISI assigned 1982 as thepublication year to the articles, while citing authorsgave 1981 in their references.

A similar discrepancy between target and citedpublication year was detected for an issue of EJP,published at the end of 1982. This issue was a partof the 1983-programme of the scientific publisher.In fact, in the heading of the printed articles, 1983appears as the publication year, but the issue wasactually published in 1982, and on the cover theyear 1982 was indicated. In SCISEARCH, thearticles in this issue as source articles have 1982 as



103

the year of publication, while citations to thesearticles contain 1983. We deleted these citationsfrom our dataset.

A further analysis of the journal JNM revealedanother dimension of the ’combined volume’

problem. The two volumes (vol. 103 and 104) werepart of a combined volume. The volumes were

clearly separated in the journal, and ISI assignedthe correct volume numbers to the target articles.However, in the heading of the printed articles,‘ vol. 103 & 104’ was indicated. Citations often con-

tained ’vol. 103’, even if the target articles were

formally published in vol. 104. In fact, 387 cita-tions showed this type of discrepancy. In addition,we observed that a citation to an article in a

combined volume-say, ’Vol. 103&104, pg. 117’-was processed by ISI in 37 cases in the follow-ing way: two references were key-punched: onecontaining ’vol. 103, pg. 117’, and another onecontaining vol. 104’ and either again ’pg. 117’ orno indication of the page number at all.

The number of discrepancies for these twocombined volumes is much higher than that forthe other combined volumes in this journal forwhich the results were presented in Table 4. Thisis probably due to the fact that in the heading ofthe printed articles in these volumes two volumenumbers appear (vol. 103 & 104), while the articlesin the other combined volumes contain only onevolume number in their headings.

4.3. Di’sc-repaticies related to the number of timescited

We examined whether highly cited articles re-ceive more citations showing discrepancies thanarticles that are cited less frequently. Therefore,we divided all target articles into classes accordingto the number of received citations showing nodiscrepancy ( C ), and calculated for each class thepercentage of citations showing a discrepancy inonly one datafield (author name, volume and pagenumber) relative to the total number of citationsthat are given to all targets in the class withoutany discrepancy (CDIC (%)). Discrepancies oftype Al (too long author names) are not included.

The results are presented in Fig. 2. In ourdatasets, the articles receiving 0-3 citations havethe highest percentage of citations showing dis-crepancies in one datafield (8.3%). This percentagedecreases to 2% as the citation frequency increases,

Fig. 2. Discrepancies related to citation frequency.C: the number of ’correct’ citations to all articles in the

journal; C°: the number of citations with a discrepancy in onedatafield only (first author name. volume number or startingpage number). The ’correct’ citation scores of the target articlesare grouped into classes. At the honzontal axis. only the lowerbound of each class is indicated. Consequently. ’0’ indicatesthe class of scores between 0 and 3 ’correct’ citations.

while for the class of highly cited articles, citedmore than 40 times, it increases to 4.2%. Roughlyspeaking, the lower cited articles receive, on anaverage, the highest percentage of citations show-ing discrepancies. Next follow the highly citedarticles (cited more than 40 times), while the mod-

erately cited articles (cited between 4 and 40 times)show the lowest percentage of discrepancies.

4.4. On »iuliiple discrepanCies. and their sources

With ’multiple discrepancies’ we indicate thephenomenon that there are seoera! publicationsthat cite a specific target article and that they allshow the same discrepancy. We found that almost83% of the discrepancies with respect to specifictarget articles occur only once, i.e. in one citedreference only. In 6.5% of the cases, a discrepancyis ’repeated’ in three or more cited references, andin 0.6% it occurs more than ten times. Discrepan-cies of sub-type Al are not included. For every100 articles cited ’correctly’ more than ten times,we had one article receiving hardly any correctcitations, but cited more than ten times in refer-ences showing a discrepancy. So roughly speaking,due to discrepancies we missed 1% of the articlescited more than 10 times.

With respect to multiple discrepancies occur-ring at least three times, we checked manuallywhether the information in our target dataset is

correct, by looking up a sample of the originalarticles in the journals. Out of 15 cases, we foundone case in which the bibliographic descriptions in



104

the printed text and in our target dataset did notcorrespond. A particular author had the initials’PBMWM’ according to the original text, and’BMWM’ in the corresponding source article inSCISEARCH online. This article was cited 14

times in references giving the correct initials.We picked out three cases in which we checked

a sample of the cited references. Thus. we com-pared 16 cited references in our dataset to thecorresponding references, printed in the originalarticle. The first case relates to a discrepancy inwhich an article in our target dataset has BASINDG’ as a first author-this name is equal to thename in the original text and is therefore correct-while all citations in our dataset to this article

contain ’BASKIN DA’. It appeared that the cita-tion entered first in SCISEARCH indeed con-tained the (incorrect) name BASIN DA’ in the

original text. However, a sample of five citingarticles entered later all gave the correct name in

the original text. As a consequence, the authornames in our citations dataset are-at least in five

cases-different from the name given in the origi-nal text. It seems to be a persistent error, since inthe SCISEARCH database updated till June 1988,the article receives 45 citations showing the dis-crepancy, and no correct citations at all. We sug-

gest that ISI has some internal procedures forunification of cited references, and that in thiscase the incorrect unification key was applied.

With respect to the other two cases, the infor-mation in our dataset of cited references was equalto that in the original text. One of these casesrelates to an article published by H.A. Singer in1983. This case will be discussed in Section 4.5

below.

4. S. Multiplication of erroneous Citations: a casestudy

We analysed in detail citations to an article byH.A. Singer, published in PET in 1983, in volume226, starting on page 690. We collected additionalcitation data for the (citing) year 1986. The article

Fig. 3. Citation relationship between articles citing Singer 1983 erroneously.Each dot indicates an article citing the specific 1 983-publicution by Singer erroneously. Solid lines between two dots indicate citationrelationships between the corresponding articles. For instance, Kontos 1984 Wes Furchgott 1983. The dashed lines between each ofthe three 1986-articles by Ramwell and the 1983-article by Furchgott indicate that the three Ramwell articles cite publicationsauthored by Furchgott, though nor the ones given in this figure. The articles citing Singer 1983 correctly are represented by twocircles. The circle at the nght-hand side contains 18 articles that do cite Furchgott’s review from 1983, while the circle at the leftindicates ten articles that do not cite this review.



105

is cited 44 times during the period 1983-1986.Twenty-eight citations give bibliographic informa-tion identical to that of our target, while in 16citations the cited volume number is 227 instead

of 226. We found that the information on the

target and on four cited references in our datasets

corresponds to the printed text of the originalarticles. We therefore assume that all errors were

made in the original text.In Fig. 3, articles citing our target erroneously

are represented as dots. For each erroneously cit-ing article we indicated the publication year andthe citing author. For most publications this is thefirst author. However, if an author appears inmore than one citing article as first author orco-author, we indicated his name even if he is notthe first author. The articles citing Singer 1983correctly, are represented in two circles. Of thesearticles we indicated only whether they cite Fur-chgott 1983 (right circle) or not (left circle).We conjectured that the multiplication process

of errors is caused by the fact that authors incompiling reference lists may copy references fromother articles. In order to provide evidence infavour of this hypothesis, we examined whetherwe were able to reconstruct the propagation of theerror on the basis of citation relationships, co-authorships or institutional cooperations.

Analysing Fig. 3 from this perspective, we ob-serve that the first article citing our target erro-neously is a review article by Furchgott, publishedin 1983. This seems to be a useful paper, since it

receives more than 300 citations in the entire

SCISEARCH database during 1983-1986. Of the15 articles citing Singer 1983 erroneously, eightactually cite this review.

Focussing on the seven articles that do not citethe 1983 review paper by Furchgott, two of these(Furchgott 1985 and Defeudis 1986B) are authoredby researchers who have published articles that docite this review. The three articles by Ramwell citeseveral other articles by Furchgott-though notthe ones that cite our target erroneously-andtherefore can be expected to be aware of theparticular review as well. For this reason, in Fig. 3the three articles by Ramwell are connected to theFurchgott-review by dashed rather than solid lines.Rosenblum 1986 cites Kontos 1984, who in turn

cites Furchgott’s 1983-review. In fact, the publica-tions by Rosenblum and Kontos emanate fromthe same department. With respect to Caldwell

1986, we could not detect any relationship interms of citations, co-authorships or institutionalaffiliations with the articles citing our target erro-neously.

Our results suggest that the authors citing ourtarget erroneously have copied the particular ref-erence from an important review paper containingthe error for the first time. There is evidence that

almost all authors had this particular review ontheir desks when they wrote their articles. We didnot prove that the particular authors copied erro-neous references from other articles. But in ourview they may have done so.

With respect to the process of multiplication oferrors, two other observations should be made.

First, it appears that not all authors citing Fur-chgott’s 1983-review, cite our target erroneously.In fact, of the 28 articles citing correctly, 18 alsocite the review paper. So, if any copying of refer-ence lists occurs, it does occur in a limited number

of cases.

The second observation is that the multiplica-tion process seems to vanish after a few years. In

an additional analysis we found that, in the period1987 - June 1988, only three erroneous citationsare given while the number of correct citationsamounts to 17. In fact, we observed that authors

apparently become aware of the error and correctit in later publications, e.g. Ramwell, who gavethree erroneous citations in 1986, but also pub-lished in 1986 an article that contains the correct

citation. It is entered in SCISEARCH later than

the other three articles.

5. General discussion and conclusions

As indicated in Section 4.4, we checked for 29

articles (targets or cited references) whether the

bibliographic information in our dataset fromSCISEARCH corresponds to the information inthe original printed articles. Of course our sampleis very small, and we should be careful in drawinggeneral conclusions with respect to the sources ofthe discrepancies in our datasets. In the ’ BASKIN’case, we suggested that the discrepancies were dueto irregularities in some internal unification pro-cess at ISI. However, we are not able to give anyaccurate estimate of the frequency with whichsuch irregularities occur. Apart from this case, we



106

found only one discrepancy in which ISI hasprobably made an error in key-punching the infor-mation into the computer. Therefore, it seems

justified to assume that the major part of thediscrepancies in our dataset are due to errors orvariations in cited references that are present inthe original text.

In Section 4.1 we found that for our five jour-nals aggregated, the percentage of citations in ourdatasets showing a discrepancy in at least onedatafield, relative to the number of citations not

showing any discrepancy, amounts to 9.4%. Cita-tions to ’unpublished’ papers are included andaccount for 2%. One can ask how large the per-centage of citations showing no discrepancy is,relative to the total number of ‘intended’ citations

to our targets. However, this total number ofcitations can hardly be determined exactly. Infact, in Section 4.2 we showed that a large numberof citations in our dataset were missing due to adiscrepancy between publication years. If we donot consider these systematic omissions, we maystill have missed citations to our targets, becauseof other types of errors or unexpected variationsin the cited journal title and publication year. Onthe other hand, our datasets of cited references

may contain citations to articles other than our

targets. If these two factors cancel out, and thetotal number of citations in our datasets is the

correct number, then we find that the number ofcitations not showing any discrepancy constitute91% of the total number of citations, while thecitations with discrepancies in one datafield only(author name, volume or page number) accountfor almost 5%.We observed large variations among the jour-

nals. These were partly explained by specific char-acteristics of target articles in those journals.However, large variations still remained unex-plained. Therefore our overall percentages givenabove indicate only the order of magnitude.

Roughly speaking, for every 100 received cita-tions not showing any discrepancy, two citationscontain a minor variation in either author name

(A9), volume number (V3) or starting page num-ber (P4), while the other discrepancies that mayoccur in principle with respect to each targetarticle, contribute another two citations. Conse-

quently, the minimum level for discrepanciesamounts to four, regardless of whether the targetarticle has specific characteristics that attract more

discrepancies, such as containing an ’L’ in thepage number field or being published in a com-bined volume.

Taking into account the number of times targetarticles are cited ’correctly’, we estimate on thebasis of Fig. 2, that this minimum level will beseven for articles cited 0-3 times, two or three forarticles cited between 8 and 40 times, and againfour for articles cited more than 40 times.

If we consider the main discrepancies due tothe specific characteristics of targets as identifiedin our analysis, we find for instance that for every100 correct citations to articles of the type bye-DDELL CR, V225, P17’, two additional citationsshow discrepancies since the author name containstwo subsequent identical consonants (type A8)and more than one initial (A3), and the volumenumber is of type V2. If the page number is ‘L17’instead of ’17’, another two citations should beadded. Thus, the total number of discrepanciesmay amount to six or eight, and may be evenhigher than ten for lowly or highly cited targets ofthese types. It is assumed that the various types of

discrepancies occur independently one fromanother.

It should be noted that the levels indicated

above relate to discrepancies in one datafield only,and therefore represent a lower bound, since dis-

crepancies in two or more datafields will raisethese levels.

In the case study presented in Section 4.5, wefound evidence that copying references from otherarticles may be a cause of the observed multipli-cation of errors in cited references. We did not

prooe that the particular authors copied erroneousreferences from other articles. However, our re-sults suggest that they may have done so. It shouldbe noted that we studied only one case. Character-istic in this case is that the first error was made inan important (highly cited) review article. It isinteresting to note that some authors publish morethan one article containing the error. So, not onlycopying references from other authors, but alsofrom one’s own articles is at stake here. The error

occurs frequently during the first three years, andthen seems to vanish. More cases should be studiedin order to obtain a more complete picture of thisphenomenon.

In Section 4.3 we found evidence that authorsare less accurate in citing articles with a lowimpact-measured by citation counts-than those



107

with a high impact. Considering the articles withhigh impact, authors possibly tend to think thatthe important documents in a field can be indi-cated less accurately since all readers know suchdocuments very well and can be expected to havea copy on their desks.

Summarizing, we present the following check-list of technical points that one should keep inmind in performing citation analysis accurately:a. Author names should be abbreviated. The

length of the last name should not exceed 15characters, and the maximum number of ini-tials is three.

b. Severe inaccuracies may arise due to dis-

crepancies between publication years of citedreference and target. We expect that the prob-ability for such discrepancies to occur is higherfor articles in special journal volumes such asproceedings, and in volumes or issues publishedat the beginning or at the end of a year.

c. Target articles published in a combined volumeconstitute a ‘risk group’ since often the ’wrong’volume number is cited, or ISI may processcited references to such targets in a strangeway.

Other risk groups are:d. Target articles in letter sections, containing an

’L’ in the page number field. In a citation, the’L’ may be omitted or may be substituted by’1’. We expect that a similar discrepancy mayarise for articles in sectionalized journals, con-taining an ’A’ or ’B’ in the volume numberfield, but we had no sectionalized journals inour sample.

e. Authors with two or more initials. Second orthird initial of the cited author may be missing.

f. Author names containing two subsequent iden-tical consonants or three subsequent conso-nants. One cites BEDELL instead of BE-

DDELL, or SHOEMAKER instead ofSCHOEMAKER.

g. Author names starting with ‘ van’, ’de’,‘ vondem’, ’Mac’ etc.: for instance one citesGILST WHV instead of VANGILST WH.

h. Volume or page numbers of three digits withtwo subsequent identical digits. One cites 227in stead of 277.

The implications of our findings for the reliabil-irv of citation analysis of articles on the level ofscientific journals, research groups or institutes,and entire countries will be discussed in a future

publication.

Acknowledgement

Part of the research described in this paper was

financially supported by Elsevier Science Pub-lishers. The authors thank Robert Braam for the

computerized drawing of the figures presented inthis paper.

References

[1] H.F. Moed, The use of online databases for bibliometricanalysis, in: L. Egghe and R. Rousseau, ed., Informetrics87/88 (Elsevier Science Publishers, Amsterdam, 1988)133-146.


possible inaccuracies occurring in citation analysis · relevant scientific documents. in this...

Documents