bibliometric analysis of interdisciplinary research › 2008 › 03 › reinter… · bibliometric...
TRANSCRIPT
Report to the
Higher Education Funding Council for England
Bibliometric analysis of
interdisciplinary research
Jonathan Adams, Louise Jackson, Stuart Marshall
November 2007 Contact details
The reporting organisation is: Evidence Ltd 103 Clarendon Road, Leeds LS2 9DF T/ 0113 384 5680 F/ 0113 384 5874 E/ [email protected] Evidence Ltd is registered in England, Company no 4036650, VAT registration 758467185 http://www.evidence.co.uk
Bibliometric analysis of interdisciplinary research
Contents
Executive summary ………………………………………………………………………………………...1
1 Introduction..........................................................................................................................3
2 Defining and indexing interdisciplinarity ..............................................................................4
3 Differential citation impact .................................................................................................15
4 Presumed interdisciplinarity...............................................................................................23
5 Implications for bibliometric assessment and suggestions for further work ......................26
6 Subject category assignment ............................................................................................28
Annex A Units of Assessment clustered on journal frequency.......................................................31
Bibliometric analysis of interdisciplinary research
Executive summary
The research community has from time to time argued for separate assessment of interdisciplinary research, but has not found an objective definition of such work. The notion is that interdisciplinary work is treated and valued differently from other work. It is hypothesised that, being marginal to core subjects, it would be systematically cited less often. If so, this might be a problem for a metrics-based system of assessment where peer judgments that distinguish work of this nature would be absent.
In this study, we have explored this notion further by:
• developing an objective description of research output according to an index of its interdisciplinarity;
• testing whether there is differential citation impact across the range of interdisciplinarity;
• evaluating our findings against presumed interdisciplinary outputs submitted to the Research Assessment Exercise (RAE) and outputs assigned to Thomson Scientific® Inc’s ‘Multidisciplinary’ category;
• considering options for handling interdisciplinary work within the proposed system of bibliometric-based indicators.
We collated article records supplied by Thomson Scientific to create databases of publications – source articles – for two large research-intensive UK universities. We analysed the work (other publication references) that cites and is cited by these source articles. To define whether a citation link to or from a source article was within or external to the article’s discipline, we used the Thomson journal categorisation.
We developed measures of interdisciplinarity by indexing the subject spread of the references cited by the source articles. We found that:
• there was a continuum of interdisciplinarity at article level within journal categories;
• the categories themselves fitted a broad but not categorical typology ranging from mono- disciplinary to multidisciplinary.
1
Using the citing references, within each journal subject category we analysed the citation count (impact) of each source article. We then compared the interdisciplinarity of each article against its rebased citation impact (normalised for year of publication and subject category). We found:
• no tendency for the most interdisciplinary articles to be less frequently cited on any consistent basis within or across categories;
• a weak tendency for the articles with highest citation impact to be neither very monodisciplinary nor very multidisciplinary in terms of their cited references.
Our comparative analysis of presumptive interdisciplinary datasets provided only limited further information.
Our findings have implications for the assessment of interdisciplinary research:
• There is no evidence to support an explicit intervention regarding articles that refer to work across a broader range of disciplines. They do not receive systematically fewer citations than more monodisciplinary publications.
• If impact is to be normalised at any categorical level, there is evidence that careful attention must be paid to the choice of baseline category, especially for articles in journals that are assigned to multiple categories. The Multidisciplinary category used by Thomson does not, however, necessarily contain articles that cross more subject boundaries than research in other Thomson categories.
• Since our index of interdisciplinarity may not satisfy all parties, the methodology should ideally be evaluated in due course through consultation with informed researchers.
In summary, there is no strong case on the basis of this analysis for research outputs to be treated differently for the purposes of bibliometric assessment on the grounds of interdisciplinarity. It is important, however, to exercise care in choosing the appropriate field against which to normalise the citation rates of more interdisciplinary outputs.
2
1 Introduction
This study has provided a preliminary exploration of the notion that interdisciplinary work might be at a disadvantage in research evaluation because it is systematically judged in different terms to core disciplinary activity. Specifically, the project considered whether published work which is more or less interdisciplinary might receive consistently different rates of citations from other publications.
Our approach was, first, to develop an objective methodology for indexing interdisciplinarity. Secondly, we tested for differential citation impact among articles with higher and lower levels of indexed interdisciplinarity.
The interdisciplinary nature of an article was indexed by analysing the diversity of the subject categories from which it drew its cited references. Within each subject category, we found that articles varied across a continuum from monodisciplinary to multidisciplinary, with no obvious discontinuities or clustering. Across subject categories, we found that some tended to be more generally multidisciplinary than others. Thus it was possible to characterise the subject categories as low, medium or high diversity.
When we compared interdisciplinarity with citation impact, we found no evidence that interdisciplinary work had consistently lower citation impact. Indeed, there was a tendency for many highly cited articles to be mid-range in terms of interdisciplinarity, and in most subjects there was a weak or very weak positive correlation between interdisciplinarity and citation impact. That is to say, publications which referred to articles across a broad range of subject categories were (marginally) more likely to attract a higher number of citations.
While our approach to indexing interdisciplinarity has been based on widely accepted methodologies, we suggest that their validity should be further tested against the perceptions of researchers.
The remainder of this report is split into four sections:
Section 2 Defining and indexing interdisciplinarity
Section 3 Differential citation impact
Section 4 Presumed interdisciplinarity
Section 5 Implications for bibliometric assessment and suggestions for further work.
3
2 Defining and indexing interdisciplinarity
This section describes how we indexed the interdisciplinarity of research outputs.
There is no clear or generally accepted definition of interdisciplinarity; it is a subjective term. Many researchers believe that they are working in an interdisciplinary area. This seems possible, given that it is widely agreed that good and innovative research often draws on discoveries at the margins of more than one discipline.1 But there are no criteria for judging whether this is true. This is problematic because some researchers also believe that interdisciplinary research, working outside the core of established disciplines, is at risk of being marginalised in research evaluation and assessment. A desirable outcome might therefore be lost if the best researchers were to shun interdisciplinarity because of this perception.
How can we determine whether such a risk exists and take steps to avoid it? Rather than relying on the opinion of the researcher or an expert group, it is preferable for assessment purposes to identify an objective (measurable) characteristic that marks out interdisciplinary research. Our study began, therefore, by developing an index of interdisciplinarity.
2.1 Methodology Research outputs can be characterised at different levels of aggregation, from individual outputs to researchers and research groups or departments. For this study, we focused on the article level. Our aim was to characterise the (relative) interdisciplinarity of an individual article so that this could be compared against the article’s citation impact.
What characteristics of an article might define its interdisciplinary nature? One approach is to look at the previous work referred to by the article in question. Arguably, work claiming interdisciplinary influence would draw on, or cite, work from a greater than average range of fields. Similarly, it would be expected that work claiming interdisciplinary utility would be cited by journals in a greater than average range of fields.
1 See for example the Government’s Science and innovation investment framework, 2004-2014.
4
There are precedents for this approach. Investigations into interdisciplinarity via the analysis of cited references have been carried out by Leydesdorff2 at journal level and Porter et al3 at article level. Within the bibliometric databases of Thomson is another precedent, in the way that the individual articles from multidisciplinary journals (such as Nature and Science) are categorised. These articles are individually assigned to subject categories according to the subject categories of the articles they cite and are subsequently cited by. For example, an article with many references to physics journals is probably a physics article and will be assigned to the Physics category.
Following these established approaches, the cited papers for any article should provide an appropriate source of material to evaluate the association between article and subject category. Accordingly, we based our definition of an article’s interdisciplinarity on the range of subject categories of the cited (and citing) references of the article.
Subject categories Whether a researcher’s work can be described as interdisciplinary depends on how finely or coarsely the discipline categories are defined. Fewer research outputs cross boundaries where subject categories are broad than would be the case if the categories were narrower. So, the level of interdisciplinarity identified by the analysis of cited references is determined by the level of aggregation of the subject categories.
For the purposes of the current study, we used the subject categories from the Thomson Current Contents® range of products There are 106 Current Contents categories in total, and the categories are grouped into seven discipline-specific products. A list of Current Contents® subject categories and further details about subject categorisation are included in section 6 of this report. Full journal lists and scoping notes for each Thomson category are available at http://scientific.thomson.com/mjl/
It would be quite feasible to apply our methodology at other levels of granularity. For example, it could operate at the finer level of Thomson’s Science Citation Index Expanded™ categories, or at the coarser level of UK RAE Units of Assessment (UoAs). We have shown elsewhere that several Current Contents categories map closely onto each UoA, based on RAE publications.4 For the same study, we also
2 Leydesdorff L, "Betweenness centrality" as an indicator of the "interdisciplinarity" of scientific journals. Journal of the American Society for Information Science and Technology, 58(9), 1303-1309, 2007. 3 Porter A L, Cohen A S, Roessner J D and Perreault M, Measuring researcher interdisciplinarity. Scientometrics, 72(1) 117-147, 2007. 4 Adams J, Benchmarking international research. Nature, 396, 615-618, 1998. Adams J, Bailey T, Jackson L, Scott P, Pendlebury D and Small H, Benchmarking of the international standing of research in England: report of a consultancy study on bibliometric analysis. CPSE, University of Leeds. 108 pp, 1998. ISBN 1 901981 04 5
5
calculated the similarity between UoAs, based on the frequency with which common journals were submitted to the 1996 RAE. The tree diagram at Annex A shows the similarity between UoAs and illustrates how a coarser level of aggregation would lead to lower levels of interdisciplinarity.
If the analysis were to be repeated at a coarser level of subject aggregation, although fewer articles would be identified as interdisciplinary, each article’s impact value would then be normalised against a broad (less appropriate) subject baseline. Elsewhere,5 we have explored the effect of changing the normalisation baseline and established the importance of selecting the appropriate level of ‘zoom’ for each analysis.
At Evidence we routinely use Current Contents categories as the preferred level of granularity, based on our experience of providing bibliometric analyses for a range of clients. Our databases are structured to allow a wide variety of analyses to be performed at this level. We discussed the options for introducing an alternative (finer or coarser) category system in background work for this project, but ultimately concluded that the resource implications would outweigh the possible benefits. Similarly, we reviewed the possibility of building in a measure of 'distance' between categories, but it was felt that this level of complexity was outside the scope of the project.
Defining the dataset The dataset for this analysis contained articles produced by two research-intensive UK universities during the seven-year period 1997-2003. The analysis was restricted to those outputs (sources, cited and citing articles) which had been assigned by Thomson to a Current Contents® subject category and which had been indexed as ‘articles’, ‘notes’ or ‘reviews’.
In total, the dataset contained 37,000 source articles in the Science, Technology, Engineering and Medicine (STEM) subject categories.6 There were upwards of 500,000 associated cited and citing articles. The discussion and presentation of results focuses on the 88 STEM subject categories. We have included Mathematics and some Social Science categories, however, since this makes for an informative comparison.
Of the 37,000 articles, 31,000 were from University A and 7,000 from University B. Some articles were collaborative between the two. In the text that follows, most of the detailed analyses relate to outputs from University A, the larger of the two. We have included comparative data from University B to substantiate
5 Adams J, Gurney K, Jackson L, Calibrating the zoom – a test of Zitt’s hypothesis. Scientometrics, 2007, forthcoming 6 The UK Government has announced that changes to research assessment after 2008 will initially affect only STEM disciplines (science, technology, engineering and medicine).
6
our claim that the findings are general, i.e. that they will apply to outputs from any research-intensive university. We have avoided repeating the same statements for B that we have already made for A.
We characterised the interdisciplinarity of the source articles by analysing the subject categories of their cited (and citing) articles, using only those citing articles that were published during the seven-year period 1997-2003. When calculating the citation impact for the source articles, we used citation counts to the end of 2006. We calculated the rebased impact (RBI) by normalising the number of citations a stated article had received in the context of the world average number of citations for articles published in the same subject category and year (see also section 3).
2.2 Defining interdisciplinarity indices We indexed the interdisciplinarity of each article by analysing the way in which its cited references were spread across subject categories. We prefer cited references to citing references as a first test of disciplinary spread because this best reflects authors’ perceptions of the diverse relevance of their work. It is also the case that new papers are likely to have few citations, and that articles with high citations are de facto ‘high impact’ irrespective of disciplinarity. At the outset, however, we compared both cited and citing information to evaluate consistency.
First, we defined each article’s subject category as its ‘source’ category. We chose three different approaches to measuring the reference category spread, resulting in three indices of interdisciplinarity. The first and simplest of the three indices is the proportion of cited references that lie outside the source category. The second index is a count of distinct cited categories. For the third index we used the Shannon diversity index.7 This combines information about the number of cited categories and the evenness of the spread of cites among categories; the index value increases with the number of cited categories and with greater evenness of cites among categories. Inevitably, it is the least transparent but also the most informative index.
The indices and their definitions are summarised in Table 1. We calculated a further three interdisciplinarity indices for each article by analysing the subject categories of the citing references.
7 The Shannon information index originally arose out of work on information theory and has subsequently been widely used, e.g. as a measure of biodiversity.
7
Table 1 Definitions of interdisciplinarity indices using analysis of cited references
Index Description Definition Minimum value
INDEX I Proportion of cited references outside source category
Number of cited references in categories other than the sourcecategory divided by number of cited references.
0
INDEX II Count of cited categories Number of distinct categories into which cited references fall. 1
INDEX III Shannon diversity index Sum, over all categories, of: the proportion of cited references in that category multiplied by thenatural logarithm of the proportion of cited references in that category.
0 (if all c ited references are in
the same category)(if reference
2.3 Analysing the interdisciplinarity indices
Category-level interdisciplinarity We calculated a category-level version of index II (number of cited categories) by counting the number of distinct categories cited by each of the source categories. Then from among the same set of cited references, we counted the number of categories citing each source category.
There was a range of category-level interdisciplinarity, but no obvious relationship between numbers of cited and citing categories. For roughly half of the source categories, the number of cited categories exceeded the number of citing categories; for the other half, the reverse was true.
Figure 1 shows these results for the 88 STEM subject categories arranged by decreasing number of cited categories. The number of categories cited by the source category is shown as a dark square. The vertical bars show the number of categories that cite the source category.
8
Figure 1 Category-level analysis of cited categories and citing categories
0
50
100
MU
LB
ILB
EHM
BGM
CB
MG
EXP
MED
PSO
BIOO
GS
NEU
CEL
SIA
IMM
PH
CG
NC
AS
EAR
MTH
ENV
PH
MS
OC
PU
BP
HS
EN
DO
NC
DG
XIN
FP
MC
HEM
PSLC
GX
CM
PM
TRP
EDC
VSR
EPP
SIA
RA
RAD
UR
OC
ARI/M S
UR
CM
LO
RT
HLT
APP
GEO
NU
TO
PHA
ICBTCF P
LO
RG
GAS
O/A
DER
EN
TV
ETE
MA
MEC
INC
RH
UA
QU
CM
AC
SES
PE
LO
TOP
SYC
ME
EEE
REH
CIV
DEN
NC
LG
NE
S/I
How many categories cite this category?How many categories does this category cite?
Number of categories
Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd. Categories and their abbreviated codes are listed in section 6; highlighted categories are reviewed in more detail later.
Article-level interdisciplinarity We found no clustering of articles at any particular level of interdisciplinarity within a subject category. Instead, there was normally a continuum of values for each of the three indices. The exceptions were extreme cases of monodisciplinarity where all of an article’s cited references fell within the source category.
Each of the three indices addresses a slightly different aspect of the interdisciplinarity of the cited articles: the values of indices II and III are connected (though not proportional) since low numbers of cited categories mean low diversity and vice versa. Index I (the proportion of cited references outside the source category) gives differing results from the other two because it is the only one of the three that differentiates citations to articles within the source category from citations to articles outside it. It is possible for an article with minimum values for indices II and III to have a maximum value for index I. This occurs when all the cited references are in the same category, but that category is not the same as the source category. As there is overlap between some of the Current Contents® categories, references outside the source category are not necessarily outside the source field. An example of this would be where a source article assigned to the Haematology category cites articles assigned to Cardiovascular and Haematology Research. Such an article would gain a high score for index I, while its Shannon diversity index would be low.
As a first step to evaluating the advantages and disadvantages of using the three indices, we compared the values of each index for each article in two different subject categories. For this comparison we chose Space Science (788 articles, a relatively monodisciplinary category) and Biochemistry and Biophysics
9
(1,775 articles, a relatively multidisciplinary category).
The articles were arranged in descending order of Shannon diversity with the respective values of all three indices plotted for each article. The results are illustrated in Figure 2.
Figure 2 Comparison of article-level interdisciplinarity using three indices
Space Science Biochemistry & BiophysicsIndex I and III Index II Index I and III Index II
Index I - Proportion of cited references outside source category Index I - Proportion of cited references outside source category
Index II - Number of distinct cited catgories Index II - Number of distinct cited catgories
– Index III - Shannon diversity index for cited categories – Index III - Shannon diversity index for cited categories
-3
0
3
0
5
10
15
20
-3
0
3
0
10
20
30
40
Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.
The left-hand chart in Figure 2 shows the spread of values of the three indices for the articles in the Space Science category. The maximum number of cited categories was 16, and the most frequently occurring value – the mode - was 2. Articles where both the Shannon index and the proportion of cites outside the source category were at zero can be described as monodisciplinary. This applied to about one-third of the Space Science articles.
10
The right-hand chart in Figure 2 shows that Biochemistry and Biophysics had a much higher proportion of multidisciplinary articles. The number of cited categories was higher (maximum 35, mode 8), and there was a much lower percentage of monodisciplinary articles in this subject category.
From this presentation we concluded that where only one index is used, to avoid redundancy in the analysis, then our preferred index should be index III – the Shannon diversity index. Its variation is broadly in line with, and certainly not contradictory to, the other two indices (see Figure 3), but it encapsulates more information. The disadvantage is that its interpretation is less transparent and straightforward.
Comparison of diversity indices for selected categories
We selected seven subject categories for closer scrutiny (see Table 2). These were categories with relatively high volume chosen from broader disciplinary areas across STEM, and also with varying levels of interdisciplinarity. As we shall be returning to these categories later to discuss differential citation impact, we also ensured that the selection contained categories with varying levels of rank correlation between interdisciplinarity and citation impact (see section 3).
11
Table 2 Comparison of interdisciplinarity indices for selected subject categories Figure 3 Average interdisciplinarity indices
Subject category CODE Article count Rebased impact INDEX I INDEX II INDEX III DIVERSITY
Cited categor Propn outsideShannon indexSpace Science SP 788 1.7 0.1 2.1 0.3 Low
Physics PHS 2208 1.8 0.2 2.4 0.5 Low
Mech Eng MEC 264 1.2 0.4 2.7 0.7 Medium
Envt & Geo GEO 184 1.4 0.6 3.0 0.7 Medium
Chemistry CMP 511 1.8 0.7 4.9 1.2 Medium
Biochem & Biophys BIL 1775 1.4 0.6 7.0 1.4 High
Microbiology MCB 761 1.5 0.6 6.9 1.5 High
INDEX I - Average proportion of cited references outside source categoryINDEX II - Average num ber of cited categoriesINDEX III - Average Shannon diversity index for cited references
0
5
10
SP PHS MEC GEO C
INDEX II
Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd. Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.
In Figure 3, the subject categories have been arranged (left to right) in ascending order of Shannon diversity. By comparing the heights of the bars for each subject, we see that each index places the subject categories in almost the same rank order. The notable exception is Chemistry, which is ranked highest when interdisciplinarity is indexed by the proportion of references outside the source category. We believe this is mainly due to an artefact of the Current Contents system. There are effectively two Thomson chemistry categories: ‘Chemistry’ (used in Figure 3) forms part of the Physics, Chemistry and Earth Sciences Current Contents® product, whereas ‘Chemistry and Analysis’ forms part of the Life Sciences product. The two categories inevitably cross-refer.
Variation of interdisciplinarity with category size
There was wide variation in size among the Current Contents® categories for each of the samples of articles from University A and University B. One reason for this could be that some distinct sub-disciplines had been aggregated into larger categories. If that were the case, then one might expect the articles in
12
those larger categories to cite fewer articles outside the source category – i.e. that the larger categories would appear more monodisciplinary. However, when we tested for a connection between the size of the category and the interdisciplinarity of the articles within it, we detected no variation of interdisciplinarity with category size. The results are illustrated in Figure 4. The subject categories have been arranged in order of descending size (frequency). The red triangle represents the average Shannon diversity index (cited references) for the articles in each subject category.
Figure 4 Diversity of articles’ cited references set against category size
Average Shannon diversity index for articles in this category (cited references)
Number of source articles in this category
0
500
1000
1500
2000
2500
PHS
BILAPPBEHM
GN
PHC
MU
LM
BGIM
MSP M
THEARM
CB
NEU
PSOM
TRD
GX
CM
PM
EDO
RG
PSLO
GS
CEL
ENV
ON
CG
NC
HEM
END
SIAAS EXPSO
CIN
FBIOIN
CPH
MR
EPPSIC
VSPU
BM
ECAR
AC
ARPEDPL C
GX
SUR
OR
TR
ADO
/AC
SEG
EOI/M U
RO
AICEL R
HU
PMC
OPH
DER
NU
TEN
TPSYH
LTC
ML
GAS
F VETC
ME
CIV
S/IBTCEM
AO
TOAQ
UC
MA
REH
NC
LM
ETD
ENISTLIBA/AEEEAERG
NE
GPM
AN
Subject categories in descending order of size
Number of articles
0.0
0.5
1.0
1.5
2.0Average Shannon diversity index
Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.
Using cited references rather than citing references
As noted earlier, although it is possible to index the interdisciplinary utility of a piece of work by analysing the subject categories of the articles that subsequently cite it, analyses based on indices that use these citing references may give an unbalanced view. This is because articles which have received a very low number of cites cannot, logically, show a high level of interdisciplinarity.
We compared interdisciplinarity calculated using cited references against the corresponding indices using citing references, and found that they were broadly in agreement. Index values based on cited references generally exceeded those based on citing references as might be expected since most articles refer to previous work, whereas some remain uncited. The average Shannon indices for cited and citing references are compared in Figures 5 and 6.
13
Figure 5 Comparison between average diversity for cited and citing references by category Figure 6 Shannon (cited) vs Shannon (citing)
0
1
2
AN CG
XEN
DC
ELO
NC
MED
MBG
REH
CM
ABTCBIOPM
CIM
MIN
FM
CB
EXPC
ML
NU
TC
VSG
ASVETF O
GS
HEM
BILR
HU
UR
OSO
CC
ARN
EUD
GX
DER
PHM
PEDG
NC
INC
PSYPSIBEHEN
TC
MP
MU
LEN
VR
EPSIAPU
BAS PL R
ADH
LTAQ
USU
RAICPSLM
TREM
APH
CM
GA/AO
RC
ME
OR
TPSOO
/AD
ENO
TOG
PMAPPO
PHN
CL
EEEEL I/M AR
AG
EOM
ETM
ECEARAERS/IC
IVC
SEPH
SISTLIBM
THSP G
NE
Average Shannon diversity index
Average Shannon diversity index (citing)
Average Shannon diversity index (cited)
0
1
2
0 1Average Shannon div
Average Shannon diversity index (c
Subject categories in descending order Shannon diversity index for CITED references
Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.
Validation and further development of the methodology
Having reviewed the relationships explored in the above sections and figures, we elected to concentrate our work for the remainder of the study on Shannon diversity indices based on cited references. We maintained comparisons across indices where informative.
If indices of interdisciplinarity were to be put to further use for research assessment purposes:
• it would be necessary to validate the results against the views of experts in the relevant fields;
• some measure of the inter-relatedness of fields might be developed so that citations to ‘distant’ fields would be weighted more heavily than citations to ‘neighbouring’ fields.8
8 Porter A L, Cohen A S, Roessner J D and Perreault M, Measuring researcher interdisciplinarity. Scientometrics, 72(1) 117-147, 2007.
14
3 Differential citation impact
The aim of this study was to test the notion that interdisciplinary research is cited relatively less frequently than other research. So, having developed and applied a methodology for indexing articles’ interdisciplinarity, it remained to establish whether there is a relationship between interdisciplinarity and citation impact.
Citation counts vary by year, because more recent articles have less time to accumulate citations, and by discipline, because of cultural differences among fields. To take account of this, it is customary to normalise the citation count for a given article by reference to the average citation count for any article published in the same year and in the same field. This is also called rebasing, so the impact (citation count) is referred to as rebased impact or RBI.
The RBI was calculated for each source article by dividing the number of citations it had received by the world average number of citations for articles published in the same subject category and year. We then correlated the RBI against each of our three interdisciplinarity indices for the source articles in each subject category.
The data in these analyses were highly skewed. Spearman rank correlation was therefore used, rather than a parametric correlation.
3.1 Rank correlation between citation impact and interdisciplinarity
We found that rank correlation between RBI and:
• Shannon diversity was positive or insignificant for most subject categories;
• number of cited categories was positive or insignificant for most subject categories;
• proportion of cited references outside the source category was negative for about half of the subject categories, and suspect that this was largely due to overlapping subject categories (see 2.3).
A positive rank correlation between RBI and interdisciplinarity means that the more interdisciplinary papers tended to be more highly cited. Conversely, a negative correlation would mean that more interdisciplinary papers received relatively fewer citations. Figure 7 shows the values for the correlation coefficient between RBI and each of the three indices for each STEM subject for University A.
15
Figure 7 Spearman rank correlation between citation impact and three interdisciplinarity indices for each subject category (University A)
-1
0
1
CGX HEM IMM END GNC OPH PMC REP SUR DEN NEU PSO HLT SOC MCB VET AQU ENT AS CMP PHC APP EAR NCL MET CSE IST CME GEO
Rank correlation between RBI and proportion of cited references outside source categoryRank correlation between RBI and count of cited categoriesRank correlation between RBI and Shannon diversity index for cited references
Subject categories in discipline order
Spearman rank correlation coefficient
Clinical medicine Life sciences Physical sciences Engineering
Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.
Rank correlation I: RBI against proportion of cited references that are outside the source category
In about half the categories, there was a negative rank correlation between index I and RBI, and for the other half the correlation was positive. This is illustrated by the position of the grey squares in Figure 7, half of which are below the horizontal axis and half of which are above. As noted above (in section 2.3), the category system used has a number of overlapping categories, which may lead to some spurious results for this index. If a high proportion of the cited references are in a closely related category, then the article will have a correspondingly high interdisciplinary index.
Rank correlation II: RBI against number of distinct cited categories
In more than 90% of the subject categories, there was a positive rank correlation between the number of categories cited and the article’s RBI. This is illustrated by the position of the pink diamonds in Figure 7, nearly all of which are above the horizontal axis.
Rank correlation III: RBI against Shannon diversity index for cited articles
In more than 90% of the subject categories, there was a positive rank correlation between interdisciplinarity indexed by the Shannon diversity index for the distribution of cited references across subject categories and the article’s RBI. This is illustrated by the position of the red triangles in Figure 7, nearly all of which are above the horizontal axis.
16
3.2 Did the rank correlation vary between universities?
Figure 7 shows the rank correlation for University A. Figure 8 adds the corresponding values for University B, shown as outlined shapes. It is clear that the results for the two institutions are very similar. We infer that the consistency of outcomes for these two universities could be generalised to other research-intensive universities. In the rest of this report we focus on the larger dataset (University A).
Figure 8 Spearman rank correlation between citation impact and three interdisciplinarity indices for each subject category (University A and University B)
-1.0
0.0
1.0
CGX HEM IMM END GNC OPH PMC REP SUR DEN NEU PSO HLT SOC MCB VET AQU ENT AS CMP PHC APP EAR NCL MET CSE IST CME GEO
Rank correlation between RBI and proportion of cited references outside source categoryRank correlation between RBI and count of cited categoriesRank correlation between RBI and Shannon diversity index for cited references Subject categories in discipline orde
Spearman rank correlation coefficient
Clinical medicine Life sciences Physical sciences Engineering
Solid shapes: University A. Outline shapes: U
Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.
17
3.3 Did the rank correlation vary with category size?
We interrogated the data to determine whether source category size (the numbers of articles in our sample in that category) affected the rank correlation between RBI and interdisciplinarity.
We found no apparent variation of correlation with category size (Figure 9).
Figure 9 Spearman rank correlation between RBI and diversity of articles’ cited references set against category size
Spearman rank correlation between RBI and average Shannon diversity index for articles in this category (cited references)
Number of source articles in this category
0
500
1000
1500
2000
2500
PHS
BILAPPBEHM
GN
PHC
MU
LM
BGIM
MSP M
THEARM
CB
NEU
PSOM
TRD
GX
CM
PM
EDO
RG
PSLO
GS
CEL
ENV
ON
CG
NC
HEM
END
SIAAS EXPSO
CIN
FBIOIN
CPH
MR
EPPSIC
VSPU
BM
ECAR
AC
ARPEDPL C
GX
SUR
OR
TR
ADO
/AC
SEG
EOI/M U
RO
AICEL R
HU
PMC
OPH
DER
NU
TEN
TPSYH
LTC
ML
GAS
F VETC
ME
CIV
S/IBTCEM
AO
TOAQ
UC
MA
REH
NC
LM
ETD
ENISTLIBA/AEEEAERG
NE
GPM
AN
Subject categories in descending order of size
Number of articles
-0.5
0.0
0.5
1.0Spearman rank correlation between RBI and Shannon diversity index
Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.
3.4 Closer inspection of rank correlation
The rank correlation coefficient summarises, in one value, the relationship between two values for each article in a subject category: the article’s interdisciplinarity and its rebased impact. The actual spread of values can best be visualised using a scatter-plot where each point represents an article and the position of each point is plotted according to its interdisciplinarity index (on the horizontal axis) and its RBI (vertical axis).
Figure 10 shows examples of scatter plots which illustrate the relationship between RBI and interdisciplinarity. They show the three indices for each of two subject categories: Space Science (relatively monodisciplinary) and Biochemistry and Biophysics (more multidisciplinary).
18
Figure 10 Sample scatter plots comparing articles’ RBI against three interdisciplinarity indices for Space Science and Biochemistry and Biophysics
Space Science Biochemistry & Biophysics
RBI INDEX I RBI INDEX I
RBI INDEX II RBI INDEX II
RBI INDEX III RBI INDEX III
0
20
40
0.0 0.5 1.0
Proportion of cited references outside source category
0
20
40
0 10 20
Number of cited categories
0
20
40
0 1 2 3
Shannon diversity index
0
20
40
0 10 20
Number of cited categories
0
20
40
0 1 2 3
Shannon diversity index
0
20
40
0.0 0.5 1.0Proportion of cited references outside source
category
Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd
In the scatter plots in Figure 10, each point represents an article. The position of each point represents the rebased impact and interdisciplinarity of each article.
For each subject there are three plots, to illustrate each of the interdisciplinarity indices.
The two topmost plots show, for each article, the proportion of cited references that lie outside the subject category (index I). For Space Science, most of the cited references for most of the articles are also in Space Science. By contrast, this is less the case for Biochemistry and Biophysics, where the articles are spread towards the right-hand side of the plot.
A similar contrast between the subjects is apparent from the two central plots for index II.
The bottom two plots illustrate the spread of values for the Shannon diversity index of cited references. The Space Science articles are to the left-hand, low-diversity end of the axis whereas most of the Biochemistry and Biophysics articles are in the middle.
The scatter plots allow us to visualise the patterns behind the correlation coefficients reported in Table 3 below.
For Space Science, the rank correlation between RBI and interdisciplinarity is very low on each index. For Biochemistry and Biophysics the correlation is positive and significant at the 1% level for indices II and III. There is no rank correlation with RBI for index I.
19
We selected a further five of the larger subject categories, across a range of disciplines and with a range of values for interdisciplinarity and rank correlation with RBI. The rank correlation for each of the categories is shown in Table 3, and Figure 11 illustrates how the correlation coefficients varied between the three interdisciplinarity indices for each subject.
Table 3 Rank correlation of RBI with interdisciplinarity indices for selected subject categories Figure 11 Rank correlation with RBI
Subject category CODE Article count Rebased
impact RCORREL I RCORREL II RCORREL III
C ited categori Propn outs ide Shannon indexSpace Science SP 788 1.7 -0.05 0.06 -0.02
Physics PHS 2208 1.8 0.00 0.14 0.06
Mech Eng MEC 264 1.2 0.20 0.27 0.26
Envt & Geo GEO 184 1.4 -0.01 0.29 0.25
Chemistry CMP 511 1.8 -0.21 0.29 0.19
Biochem & Biophys BIL 1775 1.4 -0.02 0.32 0.26
Microbiology MCB 761 1.5 0.03 0.32 0.22
RCORREL I - Spearman rank correlation of RBI against average proportion of c ited references outside source categoryRCORREL II - Spearman rank correlation of RBI against average number of cited categoriesRCORREL III - Spearman rank correlation of RBI against average Shannon diversity index
-0.3
0.0
0.3
0.5
SP PHS MEC GEO C
Correlation with RBI
Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd. Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.
We found that citation impact was positively correlated with rebased impact in most subjects for indices II and III. The correlation was significant at the 0.1% level for five subjects, excluding Space Science and Physics. For index I there was a weak negative correlation between interdisciplinarity and RBI for Space Science, Environmental Studies, Chemistry, and Biochemistry and Biophysics.
To investigate the spread of articles behind the correlation coefficients, we then compared the index III scatter plots for each of the seven subject categories. We also compared the results for University A with those of the smaller University B (Figure 12).
20
Figure 12 Scatter plots comparing RBI against Shannon diversity index for selected subject categories (University A and University B)
UNIVERSITY A UNIVERSITY B
Space Science
RBI20
Diversity
RBI20
Diversity
Physics
RBI20
Diversity
RBI20
Diversity
Mechanical Engineering
RBI20
Diversity
RBI20
Diversity
Environment
RBI20
Diversity
RBI20
Diversity
Chemistry
RBI20
Diversity
RBI20
Diversity
Biochemistry & Biophysics
RBI20
Diversity
RBI20
Diversity
Microbiology
RBI20
Diversity
RBI20
Diversity
0 3 0 3
0
10
20
0 1 2 3
0
10
20
0 1 2 3
0
10
20
0 1 2 3
0
10
20
0 1 2 3
0
10
20
0 1 2 3
0
10
20
0 1 2 3
0
10
20
0 1 2 3
0
10
20
0 1 2 3
0
10
20
0 1 2 3
0
10
20
0 1 2 3
0
10
20
0 1 2 3
0
10
20
0 1 2 3
0
10
20
0 1 2 3
0
10
20
0 1 2 3
Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.
There is a marked similarity between the scatter plots for University B, on the right, and those for University A. The distribution of the articles in relation to the axes follows a similar pattern in both institutions for each of the seven subjects.
The subject categories themselves have characteristic distributions according to whether they are more or less interdisciplinary. Those like Space Science and Physics are relatively monodisciplinary, with more articles towards the left-hand (low diversity) side.
In the lower diagrams, interdisciplinarity increases through Mechanical Engineering, Environmental Studies and Chemistry, where most articles are in the mid-range of diversity.
Biochemistry and Biophysics and Microbiology are the most multidisciplinary, with a heavier concentration of articles to the right-hand (high diversity) side of the plot.
Visual inspection suggests that the highest impact articles fall in the mid-range – neither monodisciplinary nor very multidisciplinary. This can be tested by analysing articles in decile bands, based on their Shannon diversity, and calculating average impact for each band (see below).
21
Figure 13 shows the outcome of analysing the data in Figure 12 by deciles, across the diversity range. The graph suggests that articles with the highest average rebased citation impact often occur in the mid-range of subject diversity of cited references, but this is not statistically significant. It is clear, however, that there is no fall in impact with increasing diversity or interdisciplinarity.
Figure 13 Average RBI by diversity decile band
0
1
2
3
4
Space Science Physics MechanicalEngineering
EnvironmentalStudies,
Geography &Development
Chemistry Biochemistry &Biophysics
Microbiology
Average RBI
The articles in each subject category were split into 10 bands of increasing diversity. The average RBI is shown by a grey bar for the articles in each band in each subject. The shapes suggest a very weak tendency towards higher RBI for articles with higher diversity within a subject category.
increasing diversity
Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd.
This analysis supports the observations from the correlations and scatter plots that some papers in the mid-diversity range have high impact. There is also no evidence that papers with a high diversity of cited material – which we believe have the characteristics of interdisciplinary papers – have unusually low impact.
22
4 Presumed interdisciplinarity
In the present study, we have used the spread of a paper’s cited references across subject categories to identify interdisciplinary research. In the original specification, we intended to test our methodology and findings against two presumptive datasets. One was the set of submissions to the 2001 UK Research Assessment Exercise (RAE2001) that was cross-referred between panels. The other was that set of articles which fell into the ‘Multidisciplinary’ category on Thomson Scientific’s databases. On closer examination it became apparent that these datasets were of limited utility for our purposes.
4.1 Multidisciplinary RAE submissions The ‘Refers’ table for RAE2001 contains information about (partial or whole) submissions that were referred, at the request of staff in an institution, to other UoA panels. Some of the referrals applied to one or a few members of staff, or even individual outputs, while others applied to whole research groups.
When we analysed the STEM referrals more closely, it became apparent that the majority of them were not to do with interdisciplinarity as such, but were more likely to be because of differences between the organisational structure of institutions and the way in which this related to the category structure of UoAs.
Once these and other referrals not relevant to purpose had been discounted, we were left with a small number of cross-referred items. These represented less than 0.05% of the submitted outputs. We felt that, with so little data to work on, it was not feasible to proceed with our original plan. This had been (i) to evaluate interdisciplinary indices for these outputs using the typology outlined above, and then (ii) to compare the citation impact of cross-referred outputs with the average impact for other outputs with the same UoA and grade.
We note that this very small volume of flagged interdisciplinary cross-referrals is some evidence that most STEM researchers do not, in practice, have a great deal of conflict with the assessment of discipline-based peer review panels. In terms of cost-benefit, it might be deemed that the penalty suffered for submitting interdisciplinary outputs would have to be very significant to justify making an adjustment to an evaluation system for such a small proportion of activity.
4.2 Thomson’s ‘Multidisciplinary’ category
Thomson Scientific has three different definitions for its ‘Multidisciplinary’ category, one for each of three different Current Contents® products. Some interdisciplinary journals are always assigned to the Multidisciplinary category, regardless of which product they are in, but some have different assignments for different products. Within a product, it is the whole journal – rather than individual articles – that gets assigned to a particular category.
23
It is important to be aware that an article’s category assignment may change from year to year, depending on the citations it has received.
There has been an increase in Multidisciplinary articles per year in the 2006 version of the UK National Citation Report (UKNCR) compared with the 2005 version. We believe this is because:
• articles are assigned to categories according to the citations they have received to date;
• journal coverage has changed.
This has an effect on rebased impact at the article level. As an example, one article published in Nature (2000) was assigned to the Multidisciplinary category in UKNCR 2005, but to Animal and Plant Science in UKNCR 2006. As a result, the RBI of this particular article decreased from 2.8 to 1.6 because of the difference in baselines for the different subject categories.
To improve our understanding of the effects of Thomson’s categorisation on interdisciplinary articles, we analysed one year’s worth of UK Nature articles. Figure 14 illustrates the spread of articles across categories, the differential citation rates and subject baselines, and the resulting RBI for UK articles published in Nature in 2000.
24
Figure 14 Average citation impact and frequency of UK Nature articles (2000) by subject category Figure 15 Scatter plot for Multidisciplinary category
RBI100
0
0
100
200
300
400
500
CG
XG
AS
EN
DC
ML
IMM
CEL
OG
SB
ILEN
VP
HM
MC
BP
HS
PH
CA
PPM
BGB
EHA
QU
EARP
LM
GA
SE
XPC
MP
EN
TB
IOM
UL
MET
SP
VET
CAR
AR
CG
NC
PSO
0
10
20
30
40
Average impact Average rebased impact
Category baseline impact Number of papers
Average impact Number of papers category baseline
rebased impact
0 1 2
Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd Source: data, Thomson Scientific® Inc; analysis, Evidence Ltd
The Multidisciplinary category is mid-range in interdisciplinarity on all three indices when compared with other subject categories. The plot in Figure 15 shows that the correlation between citation diversity and RBI is similar to mid-range subjects such as Chemistry or Mechanical Engineering (Figure 12).
This initial exploration of Thomson’s Multidisciplinary category suggests that its contents do not differ significantly from any other categories in the balance of interdisciplinary articles. This means that the assignment of an article to the Multidisciplinary category does not tell us much about the interdisciplinarity of its cited references. Interdisciplinary articles are to be found in all Thomson categories.
25
5 Implications for bibliometric assessment and suggestions for further work
If the premise that the relative interdisciplinarity of research should be reflected in the diversity of fields on which it draws is accepted, then there is no indication that interdisciplinary work suffers from a lack of esteem. We found no evidence that articles classified as interdisciplinary by the subject spread of their cited references receive systematically fewer citations than other comparable articles.
For this study, we looked at data drawn from two research-active but rather different universities: different in their volume of research, their diversity of capacity and the strengths of their departments. We found no substantive difference in comparable analyses made between the two institutions, so we feel that the outcomes of this study have some general applicability.
We used three different indices of diversity for the citations made by the source articles from the two institutions. We found no conflicting difference between these indices, which generally showed similar relativities among subject categories. We therefore preferred the Shannon diversity index as our guide because this was more information-rich, albeit less transparent.
Non-parametric correlation analysis revealed that articles with a more diverse range of cited material had a relatively higher rebased citation impact than articles for which a greater share of cited material was within the same journal category as the source article.
We found some limited but consistent indications that the most highly cited articles tended to be those in the middle range of interdisciplinary diversity. Tentatively, we would suggest that these are articles which remain attached to their core discipline but establish relevance in cognate areas. However, further examination is required to validate this.
In conclusion, if articles are indexed on their normalised citation impact, there is no reason to suppose that those which appear to be more interdisciplinary will be in any way systematically disadvantaged.
Before developing policy based on this finding, however, we suggest that our method of indexing interdisciplinarity should be tested against researchers’ perceptions and expert opinion. If this definition of interdisciplinarity proves acceptable, we would conclude that there is no reason to single out interdisciplinary research for differential treatment under a metrics-based system of assessment.
26
We also draw attention to our exploration of presumptive interdisciplinarity in associated datasets. Our analysis of Thomson’s Multidisciplinary category alerted us to the critical issue of using an appropriate base for normalisation. This was not the focus of the current study, but is an important finding in the context of the general use of bibliometrics for research assessment.
What therefore remains crucial to the successful use of bibliometric data for assessment purposes is the selection of an appropriate field-level baseline against which citation counts are normalised. This applies to all research outputs, but especially to those that fall into one or more subject category. It is particularly important in cases where the baseline citation rates of the possible categories are significantly different from each other.
27
6 Subject category assignment
There are 106 Thomson Scientific Current Contents® subject categories in total. Each article is assigned to one or more subject category according to the journal in which it is published. Exceptions to this are the journals Nature, Science and Proceedings of the National Academy of Sciences (PNAS), where articles are individually assigned to subject categories according to the subject categories of the articles they cite and are subsequently cited by. This means that an article’s category assignment can change over time.
Within a Current Contents® product, a journal is mapped to one or more of the subject categories, and every article within that journal is assigned to those categories. Journals can appear in more than one product. For example, the journal Plant Physiology is in Current Contents®/Agriculture, Biology and Environmental Sciences – assigned to the category Plant Sciences – and also in the Current Contents®/ Life Sciences product where it is assigned to the category Animal and Plant Science.
For the current study, we needed a method of dealing with multi-category journals that avoided double-counting. We drew up a one-to-one mapping between journals and subject categories. This ensured that cited and citing references could be assigned unambiguously to single subject categories, even where information on subsequent citations was lacking. Journals that had previously been assigned by Thomson to more than one subject category were re-assigned to the most frequently occurring category for articles in that journal, based on the UK National Citation Report for 2006. This new mapping was then used to re-assign source articles so that all the articles from each journal were assigned to the same category. (Under this methodology, all articles from Nature and Science were assigned to the Multidisciplinary category, and all articles from PNAS were assigned to Biochemistry and Biophysics.)
The 88 STEM (plus Mathematics) subject category titles and their abbreviated codes are listed below.
28
Thomson Scientific® Inc Current Contents® 2006 subject categories, alphabetically by category code
Code Current Contents(R) Subject Category Code Current Contents(R) Subject Category Code Current Contents(R) Subject
A/A Agriculture/Agronom y ENT Entom ology/Pest Control OGS Medical Research, Organs & SAER Aerospace Engineering ENV Environm ent/Ecology ONC OncologyAIC Anesthesia & Intensive Care EXP Experimental Biology OPH Ophthalm ologyAN Animal & Plant Science F Food Science/Nutrition ORG Organic Chem istry/Polym er SAPP Applied Physics/Condensed Matter/Materials Science GAS Gastroenterology & Hepatology ORT Orthopedics, Rehabilitation & AQU Aquatic Sciences GEO Environm ental Studies, Geography & Developm ent OTO OtolaryngologyARA AI, Robotics & Autom atic Control GNC General & Internal Medicine PED PediatricsAS Animal Sciences GNE Engineering Management/General PHC Physical Chemistry/Chemical BEH Neurosciences & Behavior GPM Geological, Petroleum & Mining Engineering PHM Pharm acology & ToxicologyBIL Biochem istry & Biophysics HEM Hem atology PHS PhysicsBIO Biology HLT Health Care Sciences & Services PL Plant SciencesBTC Biotechnology & Applied Microbiology I/M Instrumentation & Measurem ent PMC Pharm acology/ToxicologyCAR Cardiovascular & Respiratory Systems IMM Im munology PSI PsychiatryCEL Cell & Developm ental Biology INC Inorganic & Nuclear Chemistry PSL PhysiologyCGX Oncogenesis & Cancer Research INF Clinical Im munology & Infectious Disease PSO PsychologyCIV Civil Engineering IST Inform ation Technology & Comm unications Systems PSY Clinical Psychology & PsychiaCMA Agricultural Chemistry LIB Library & Information Sciences PUB Public Health & Health Care SCME Chemical Engineering MBG Molecular Biology & Genetics RAD Radiology, Nuclear Medicine &CML Chemistry & Analysis MCB Microbiology REH RehabilitationCMP Chemistry MEC Mechanical Engineering REP Reproductive MedicineCSE Com puter Science & Engineering MED Research/Laboratory Medicine & Medical Technology RHU RheumatologyCVS Cardiovascular & Hematology Research MET Metallurgy S/I Social W ork & Social PolicyDEN Dentistry/Oral Surgery & Medicine MGN Medical Research, General Topics SIA Spectroscopy/Instrum entationDER Derm atology MTH Mathem atics SOC Environmental Medicine & PubDGX Medical Research, Diagnosis & Treatment MTR Materials Science & Engineering SP Space ScienceEAR Earth Sciences MUL Multidisciplinary SUR SurgeryEEE Environm ental Engineering & Energy NCL Nuclear Engineering URO Urology & NephrologyEL Electrical and Electronics Engineering NEU Neurology VET Veterinary Medicine/Anim al HeEMA Engineering Mathem atics NUT Endocrinology, Metabolism & NutritionEND Endocrinology, Nutrition & Metabolism O/A Optics & Acoustics
29
30
Annex A Units of Assessment clustered on journal frequency
Source: Adams et al 1998 (see footnote 4)
0.00 0.16 0.32 0.48 0.64
Clinical Lab Sci...
Accountancy
Hosp. based...Com. based...Other stud. PharmacyBiochemistryBiol. sciencesPre-clin. stud.PhysiologyPharmacologyAnatomyVeterinary sci.Clin. DentistryFood sci...AgricultureEarth sci.Environ. sci.GeographyArcheologyMineral/mining...ChemistryMetallurgy...PhysicsChem. eng.Computer sci.Gen. Eng.Mechanical eng...Electrical eng...Civil eng.Pure maths.Applied maths.Statistical res...Nursing`Sports related...PsychologyEducationPolitics...Social policy...SociologySocial workCommunication...Built environ.Town/country...Economics...Business... stud.
LawLibrary and info...AnthropologyAsian stud.Middle east...Theology...American stud.Iberian...European stud.FrenchGerman, Dutch...EnglishHistoryItalianRussian...LinguisticsClassics...PhilosophyHistory of Art...Art and DesignDrama, Dance...MusicCeltic stud.
Engineering
Medical
Physical
Maths
Bio-Med
Environment
Social
Arts & Humanities
Czekanowski similarity index
1996 UoA
This tree diagram illustrates similarity in the frequency with which journals were submitted to RAE1996
31