An Observation of Research Complexity in TopUniversities Based on Research Publications

Ivan LeeUniversity of South AustraliaMawson Lakes, SA, Australia

Dalian University ofTechnology

Dalian, Liaoning, Chinaivan.lee@unisa.edu.au

Feng XiaDalian University of

TechnologyDalian, Liaoning, China

Göran RoosNanyang Business School,

Nanyang TechnologicalUniversitySingapore

ABSTRACTThis paper investigates research specialisation of top rankeduniversities around the world. The revealed comparativeadvantage in different research fields are determined accord-ing to the number of research articles published. Subse-quently, measures of research ubiquity and diversity, andresearch complexity index of each university, are obtainedand discussed. The study is conducted on top-ranked uni-versities according to Shanghai Jiao Tong Academic Rank-ing of World Universities, with bibliographical details ex-tracted Microsoft Academic Graph data set and researchfields of journals labelled with SCImago Journal Classifica-tion. Diversity-ubiquity distributions, relevance of RCI anduniversity ranks, and geographical RCI distributions are ex-amined in this paper.

Keywordsscientometrics; complexity modelling

1. INTRODUCTIONResearch output analysis has been an active research area,

and it is one major factor assessing the performance of auniversity, subsequently influence the allocation of educa-tion fund and research grants, the attractiveness to organi-sations interested in procuring applied research services, aswell as the attention from prospective students. Most pop-ular university rankings, such as Shanghai Jiao Tong Aca-demic Ranking of World Universities (ARWU), QuacquarelliSymonds World University Ranking (QS), and Times HigherEducation World University Rankings (THE), consider re-search output as one of the crucial factors towards rankinguniversities around the world. The rankings, from a globalprospective, may be considered the scientific competitive-ness of nations [Cimini et al. 2014].

While universities usually comprise of a number of fac-ulties or divisions and pursue research in different fields,

c©2017 International World Wide Web Conference Committee (IW3C2),published under Creative Commons CC BY 4.0 License.WWW 2017, April 3–7, 2017, Perth, Australia.ACM 978-1-4503-4914-7/17/04.http://dx.doi.org/10.1145/3041021.3053054

.

most university rankings (include the aforementioned exam-ples) do provide additional ranking break-downs associatedto research fields. Analysis of disciplinary specialisation hasalso attracted attentions in the research community, to ex-plore its relevance and differences to university ranks [Lopez-Illescas et al. 2011] [Robinson-Garcıa and Calero-Medina2014]. Mapping of research disciplines has also been in-vestigated, such as the study based on ISI Web of Science(WoS) [Leydesdorff and Rafols 2009] and the study basedon combined WoS and Scopus [Borner et al. 2012].

Further to analysing and visualising the distribution ofdifferent research fields, clarification of universities accord-ing to research specialty has also been investigated in thepast. Ortega et al. has applied a clustering algorithm toidentify specialisation of Spanish institutes based on theirresearch outputs [Ortega et al. 2011], and classify these in-stitutes as either Humanistic, Scientific, or Technological.Studies on the most frequently publishing universities inSpain suggest that university rankings should account fordisciplinary specialisation [Lopez-Illescas et al. 2011]. Li etal. has conducted another study of research specialisationat country-level in China, and the change of disciplinarystructure over time has been investigated [Li et al. 2015].Beyond the country-level, studies of research specialisationhas been reported for European universities [Daraio et al.2015] [Pastor and Serrano 2016], and for universities aroundthe world [Harzing and Giroud 2014]. While bias in researchgrant allocation to small universities are observed accordingto the study by Murray et al. [Murray et al. 2016], analy-sis of research specialisation may help improving the grantallocation model.

While the research outcome of a research institute comesin different forms, including research publications, copyrights,patents [Wong and Singh 2010] [Payumo et al. 2014]. Amongthese, research publication is possibly the most popular bench-mark to assess the research productivity of a university. Thispaper attempts a different approach to examine researchpublications produced by top universities around the world.Economic Complexity Index (ECI) [Ricardo et al. 2008], amodern technique for modelling the economic complexityaccording to the export record of countries in the world [Hi-dalgo and Hausmann 2009], has been adopted for analysingresearch complexity of universities. The original model mea-sures a country’s industrial composition by analysing its ex-port record, and combines the diversity of ubiquity to formthe economic complexity index, which can be used to pre-

1259

dict the economic growth. In this paper, instead of using theexport record, research publications are utilised to measurethe diversity and ubiquity of an institute’s research output.The complexity model for academic output analysis has beenrecently investigated [Guevara et al. 2016], and the work byGuevara et al. has focused on grouping the publications ac-cording to research fields, and analyse the proximity acrossdifferent research fields. This paper compares top universi-ties in terms of their research ubiquity and diversity, as wellas their complexity indices. Correlation between opportu-nity value and research complexity index is also investigatedin this paper.

The rest of this article is organised as follows: we startfrom describing how revealed comparative advantage are ob-tained through academic publications in section 2, followedby the definition of ubiquity and complexity of a university,and the concepts of research complexity index, capabilitydistance, and opportunity value in section 3. Sections 4 and5 covers the methods for data processing and the experi-mental results. Finally, our concluding remarks are drawnin section 6.

2. REVEALED COMPARATIVE ADVANTAGEIn economic analysis[Hidalgo and Hausmann 2009], export

record across different industrial sectors are utilised for com-plexity modelling. To evaluate the performance of researchinstitutions, publication outputs of different research disci-plines can be used to replace the export values. In our study,we use the SCImago Journal Classification which labels jour-nal in 27 research areas. Although, there are a number ofmulti-disciplinary publications, such as prestigious journalsNature and Science, accept articles from more than one re-search field. Let i denote an academic institute (university),and f denotes a research field. For a paper p(i, f) publishedin journals labelled with m disciplines where m ∈ Z andm ≥ 1, a normalisation factor mp(i,f) needs to be applied.We define the publication matrix Pif as the summation overall papers p(i, f) in research field f and has at least oneauthor from university (or institute) i, normalised by thenumber of research fields covered by the journal. Mathe-matically, Pif is expressed as

Pif =∑p(i,f)

1

mp(i,f)

. (1)

Revealed comparative advantage (RCA) is defined as thepercentage of academic publications of a particular disci-pline in an academic institute, divided by the percentageof all papers by the academic institute over all academicpublications. Thus, if the academic institute has a strongfocus in a particular research discipline, then the nominatorterm increases hence increase the overall RCA value. Whilethe academic institute may be big or small in terms of thevolume of research outputs, the normalising term in the de-nominator helps reduce the impact due to the institute’ssize. Mathematically, the revealed comparative advantageRCAij for research institute i and research field f can beexpressed as

RCAif =Pif/

∑f Pif∑

i Pif/∑

i,f Pif. (2)

It should be noted that RCA represents the level of special-isation, and not a measure of competitiveness of a researchfield.

The RCA values across different disciplines are used toconstruct the Mif matrix, and each element in Mif is aboolean value which indicate whether an academic institutehas revealed comparative advantage in a particular field ofresearch.

Mif =

{1 if RCAif ≥ t,0 if RCAif < t.

(3)

where t is a manually selected threshold value. In this paper,the threshold value follows the setup in Economic Complex-ity analysis [Ricardo et al. 2008] and t value is chosen as1, expect for investigating the correlation between the com-plexity indices and university ranks, where multiple t valuesare used. Further details are covered in sections 3 and 5.

3. DIVERSITY, UBIQUITY, AND RESEARCHCOMPLEXITY

With RCA values summarised in Mif , the measure of aninstitute’s research fields with reveal comparative advantage,or diversity Di, can be formulated as:

Di = k(i, 0) =∑f

Mif . (4)

Similarly, the measure of the degree of how many researchinstitutes has revealed comparative advantage in a researchfield can be obtained. This measure, known as the ubiquityUf , can be formulated as

Uf = k(f, 0) =∑i

Mif . (5)

Using the Method of Reflections [Hidalgo and Hausmann2009], the value of k(i, n) can be formulated in terms ofk(f, n− 1):

k(i, n) =1

k(i, 0)

∑f

Mifk(f, n− 1). (6)

Likewise, the value k(f, n) can be obtained in terms of k(i, n−1):

k(f, n) =1

k(f, 0)

∑i

Mifk(i, n− 1). (7)

Subsequently, each institute may be characterised by thevector ki = {k(i, n) | n = 1 . . . N} and kf = {k(f, n) | n =1 . . . N} where N is a pre-determined number of iterationsand N ∈ Z and N > 1. Equations (6) and (7) can be writtenas

k(i, n) =1

k(i, 0)

∑f

Mif1

k(f, 0)

∑i

Mifk(i, n− 2), (8)

which can be rewritten as

k(i, n) =∑i′

Mii′k(i′, n− 2). (9)

The research complexity index of an institute, RCI(i), iscalculated according to

RCI(i) =Ki −Ki

σ(K), (10)

1260

where Ki denotes the eigenvector of Mii′ according to [Ri-cardo et al. 2008], and σ(Ki) denotes the standard deviationof the K vector.

Similarly, the complexity of research fields can be mod-elled by field complexity index, FCI, where

FCI(f) =Qf −Qf

σ(Q). (11)

where Qf denotes the eigenvector of Mff ′ , following the

same process of obtaining Mii′ for k(i, n).The measure of proximity is based on the conditional

probability that an institutes has both reveal comparativeadvantage in both research fields f and f ′. Proximity φ(f, f ′)for a pair of research fields f and f ′ can be expressed as:

φ(f, f ′) =

∑i MifMif ′

max (k(f, 0)− k(f ′, 0)). (12)

Subsequently, capability distance can be obtained. The con-cept of capability distance show how far away for an instituteto pursue a research field it’s not currently specialised (i.e.without revealed comparative advantage). The capabilitydistance dif is defined as:

d(i, f) =

∑f ′(1−Mif ′)φ(f, f ′)∑

f ′ φ(f, f ′). (13)

The opportunity value OV of an institute i is defined as:

OV (i) =∑f ′

(1− d(i, f ′))(1−M(i, f ′))FCI(f ′) (14)

and a higher opportunity value indicates that the institutehas more research fields in close proximity or its researchfields are more complex.

4. DATA PROCESSINGIn this paper, we’ve chosen the Microsoft Academic Ser-

vice (MAG)[Sinha et al. 2015] in our study. In comparisonwith other popular collections such as DataBase systems andLogic Programming (DBLP) and American Physical Soci-ety’s Data Sets for Research (APS), with focuses on com-puter science and physics bibliography respectively, MAScovers most research disciplines. In comparison with popu-lar multidisciplinary datasets from ISI Web of Science (WoS)and Google Scholar Citations, the full collection of the MASdataset is open to the public to access and download throughMicrosoft Academic Graph (MAG). The MAG dataset wasthe official dataset recommended for the 2016 KDD CUPcompetition on predicting the publication acceptance andmeasuring the impact of research institutions [Sandulescuand Chiru 2016]. Although, it was also observed that MAG’spopularity in the bibliometric research community has beendeclining [Orduna-Malea et al. 2014], and this paper haschosen the collection of an earlier year (2014) for analysis.

This paper focuses on the comparison of research complex-ity between top ranked universities in the world. While thereare multiple world university rankings, each ranking systemdistinguishes itself from one another due to the choice ofindicators and the ranking scores formulae. At the sametime, these ranking systems exhibits a considerable overlapin the top 100 listed universities according to the study byMoed [Moed 2016].

Top 100 universities listed on the ARWU 2016 rankingoutcome has been selected in our study. It should be noted

that the selection of the ranking year doesn’t need to matchthe publication year. One potential extension of the studycould be the investigation of how research complexity in-fluence subsequent university ranking, which is beyond thescope of this paper and may be considered for future work.

The format of academic publication may be consideredsemi-structured data, and MAG applies algorithms to auto-matically parse details such as author and institutions [Sinhaet al. 2015]. In our study, institutional information is re-quired, and according to our observation there are multipleaffiliation IDs that may be associated with the same insti-tution. It is observed that punctuation, such as the commain “University of California, Los Angeles”, is removed in theMAG data set. Thus, adjustment needs to be made for insti-tution names extracted from ARWU. Apart from the chal-lenge of automated information extraction, some institutesmight have multiple well perceived (not necessarily official)names. For example, “University of Sydney”, “The Univer-sity of Sydney”, “Sydney University”appear as separated en-tries and they are combined in our study. In addition, well-perceived abbreviations such as “MIT” for “MassachusettsInstitute of Technology” are considered. Although, in thisparticular example, “MIT” can be a sub-string of anotheraffiliation, therefore only the abbreviations that fit withinword boundaries are taken. Lastly, we observed that someautomatically parsed entries appear to be short biographies,and we remove affiliation entries with 300 characters or more.After the affiliation name adjustment, we still observe thatsome of these top ranked universities yield very low numberof publications. Universities with less than 1000 publica-tions in 2014 according to the MAG data set are excludedfrom our study.

Commercial bibliography tools such as WoS and SCImagoprovide classification codes for identifying the research fieldsof different journals. Unfortunately, classification codes arenot available in the raw MAG dataset, which represents aconstraint from adopting the MAG dataset for research com-plexity analysis, and it requires significant effort to classifythe entire database. Instead of designing a new set of clas-sification codes for the MAG dataset, sciMAG2015 adoptsthe codes from SCImago Journal Classification and labelsall MAG journals [De Domenico et al. 2016]. In this paper,we use the 27 distinct area classification from sciMAG2015for research complexity analysis.

5. INSTITUTIONAL RCIThe ubiquity-diversity scatter plot of the selected insti-

tutes is shown in Figure 1. The mean diversity ki,0 par-tition institutes into two groups with the number of fieldswith revealed comparative advantage higher or lower thanthe average, whereas the mean ubiquity ki,1 divides insti-tutes specialised in niche or popular/standard research fieldscompares to the mean value. Combination of the two meandividers partitions the scatter plots into four quadrants, andthese quadrants represent:

• upper-left quadrant: non-diversified institutes publish-ing in standard areas;

• upper-right quadrant: diversified institutes publishingin standard areas;

• lower-left quadrant: non-diversified institutes publish-ing in exclusive areas;

1261

• lower-right quadrant: diversified institutes publishingin exclusive areas.

5 10 15 20

25

26

27

28

29

30

31

32

33

34

001002

003

004

005

006007

009010

012

013014

015

016

017

018

019

021025

026

027

028

029

030

031

032

034036

037

044 045

047

049

051

052054

056

057

058

059

060

061

062

063

064

065

066

067

070

071

072

074

075

076

079

082

083

084

088

089091 094095

096

097

100

Figure 1: Ubiquity-diversity scatter plot

Figure 1 shows that most universities are scattered overthe upper-left, lower-left, and lower-right quadrants, with24 (or 36%), 15 (or 23%), and 17 (or 26%) universities, re-spectively. The upper-right quadrant has the least numberof universities, with 10 (or 15%) universities found in thisquadrant and most of them positioned close to the meanubiquity or the man diversity line. According to the defini-tion of diversity and ubiquity, the sparse upper-right quad-rant shows that only a small number of universities havediversified research disciplines with high RCA values, andthese research specialisation span across research fields withmore universities also with high RCA values (i.e. standardareas that many universities claim specialisation.)

Compares Figure 1 with the matching plot for economiccomplexity by Hidalgo & Hausmann [Hidalgo and Haus-mann 2009], there’s a noticeable difference in the lower-left quadrant, with few economies scatter over this quad-rant. While the economic complexity analysis covers mosteconomies in the world, whereas the study in this paperfocuses on top universities which is a small fraction of alluniversities in the world, the difference may be associatedto the difference of the sample coverage. Investigations onuniversities with all ranking levels could be considered as apotential extension of this work.

Table 1 shows the RCI and OV values of selected univer-sities. The order of the list follows the ARWU 2016 ranks,with 66 universities due to incomplete data set for some uni-versities, as explained in section 4.

Scatter plots of RCI against ARWU ranking are shown inFigure 2. In the experiment, different RCA threshold valuesare used (i.e. the threshold for determining whether an in-stitute has revealed comparative advantage in a particularresearch field, according to Equation (2).) According to thefirst-order linear regression (dotted red line) plots, there’s noclear evidence of correlation between RCI and the universityrank. While the selected universities are mostly well-funded,extension of the study to cover smaller universities could beconsidered future work.

A scatter plot of opportunity value versus RCI can befound in Figure 3. Note, while part of the text labels in Fig-

0 10 20 30 40 50 60 70 80 90 100Rank

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

RC

I

(a) RCI with threshold t = 0.75

0 10 20 30 40 50 60 70 80 90 100Rank

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

RC

I

(b) RCI with threshold t = 1.00

0 10 20 30 40 50 60 70 80 90 100Rank

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

RC

I

(c) RCI with threshold t = 1.25

Figure 2: RCI-rank scatter plots with different thresholdvalues

1262

Table 1: RCI of top ranked universities

Rank Institute Name RCI OV

001 Harvard University -0.68 -2.20002 Stanford University -0.01 1.00003 University of California, Berkeley -1.74 -4.05004 University of Cambridge -1.03 -1.43005 Massachusetts Institute of Technology -1.40 -6.25006 Princeton University -1.37 -7.04007 University of Oxford -0.20 -0.03009 Columbia University 0.47 1.66010 University of Chicago 0.61 1.88012 University of California, Los Angeles 1.38 2.29013 Cornell University -0.66 -2.92014 University of California, San Diego -0.25 0.06015 University of Washington 1.49 1.64016 Johns Hopkins University 1.70 1.26017 University College London 1.01 1.37018 University of Pennsylvania 0.99 3.72019 ETH Zurich -1.50 -6.90021 University of California, San Francisco 1.40 1.48025 Duke University 1.08 4.04026 Northwestern University -0.20 0.36027 University of Toronto 1.27 2.71028 University of Wisconsin - Madison -0.29 -0.28029 New York University 0.63 2.65030 University of Copenhagen 0.80 1.65031 UIUC -0.82 -4.56032 Kyoto University -1.65 -4.04034 University of British Columbia -0.25 0.08036 UNC 1.05 4.06037 Rockefeller University 0.26 1.62044 Karolinska Institute 1.66 2.21045 The University of Texas at Austin -0.20 0.31047 Heidelberg University 0.42 1.26049 University of Southern California 0.72 2.91

Rank Institute Name RCI OV

051 University of Munich 0.53 1.78052 University of Maryland, College Park -1.08 -5.77054 University of Zurich 0.13 1.33056 University of Helsinki -0.28 -0.21057 University of Bristol -1.33 -5.26058 Tsinghua University -1.80 -4.54059 University of California, Irvine -0.04 0.68060 Uppsala University -0.50 -0.48061 Vanderbilt University 1.51 2.80062 Ghent University -0.64 -1.81063 McGill University -0.11 0.45064 Purdue University - West Lafayette -1.16 -4.52065 Aarhus University 0.59 1.22066 Utrecht University -0.52 -0.79067 University of Oslo 0.23 1.54070 University of Pittsburgh 1.58 2.21071 Peking University -1.26 -4.50072 Nagoya University -1.61 -2.51074 University of Groningen 0.21 1.99075 Boston University 1.32 2.77076 University of California, Davis -0.15 0.11079 Monash University -0.05 1.14082 University of Sydney 0.07 1.67083 McMaster University 0.74 1.74084 National University of Singapore -0.91 -4.87088 Moscow State University -1.91 -3.92089 The Hebrew University of Jerusalem 0.05 1.76091 University of Florida 1.14 2.20094 KU Leuven -0.58 -1.01095 Leiden University 0.66 0.92096 Osaka University -1.07 -1.91097 Rutgers 0.16 1.88100 University of Utah 1.35 1.70

ure 3 appear cluttered (especially for the points with RCIvalues close to zero), the values of ranking-RCI pairs can alsobe found in Table 1. Figure 3 illustrates a high degree of cor-relation between opportunity value and RCI. As shown inthe first and second order regression, opportunity value wellfit as a quadratic function of RCI. The third order regres-sion is an over-fit and it is not included in 3 to increase thereadability. According the definition in Equation (14), insti-tutes with high RCI values has more research fields withinclose proximity to pursue, or its current research fields areconsidered complex. The plot does not exhibit associationsbetween university rank to the OV-RCI pairs (i.e. there’sno cluster of closely ranked universities), similar to the ob-servation in Figure 2 that no evidence of associate betweenthe university rank has been found.

Figure 4 shows the geographical distribution of RCI val-ues. The colours represent the “band” of the RCI values,with markers in green for RCI > 0.5, red for RCI < −0.5,and yellow otherwise. The size of the marker representsthe magnitude (or value) of RCI. The geographical longi-tude and latitude details of the selected universities are ex-tracted from Google search as well as from the Wikipedia.It should be noted that most yellow markers are too smallto display due to their small magnitude/value (for example,

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2RCI

-8

-6

-4

-2

0

2

4

6

Opp

ortu

nity

Val

ue

001

002

003

004

005

006

007

009 010012

013

014

015016017

018

019

021

025

026

027

028

029

030

031032

034

036

037

044

045

047

049

051

052

054

056

057

058

059

060

061

062

063

064

065

066

067

070

071

072

074

075

076

079082 083

084

088

089091

094

095

096

097 100

Universities1st order regression2nd order regression

Figure 3: Opportunity Value vs RCI scatter plot

1263

the two Australian universities in Table 1), despite theirpositive/negative signs. It is observed that top universitiesin North America in general have high positive RCI values,Asian universities have the other extreme with very low neg-ative RCI values, whereas European universities exhibit tohave a good mixture of both. The difference could be due toa number of factors, such as diversification as a proxy of theuniversity age, the composition of the economy which leadsto bias in discipline-based research funds. These factors arebeyond the scope of this paper, and further investigationsmay be considered in the future work.

Further analysis, such as other factors associate to theRCI, as well as the relevance of RCI to the university rankingover time, will be considered the future work of this paper.

6. CONCLUSIONSThis paper examined research complexity based on the

research outputs of top ranked university around the world,From the study of diversity and ubiquity, most of the se-lected universities are either non-diversified institutes, ordiversified institute publishing in exclusive areas. From thestudy of research complexity analysis, there was no associa-tion observed between RCI values and the institution’s rank-ing. On the other hand, opportunity value highly correlateswith RCI according the scatter plot, suggesting that highRCI institutes has more research fields within close prox-imity, or its current research fields are considered complex.According to geographical distribution, it was observed thatmost North America institutes possesses high positive RCIvalues, Asian institutes possess low negative RCI values, andEuropean institutes have a good mixture of both. The ob-servation suggested further investigations are required of theunderlying factors.

7. REFERENCES[Borner et al. 2012] Katy Borner, Richard Klavans,

Michael Patek, Angela M. Zoss, Joseph R. Biberstine,Robert P. Light, Vincent Lariviere, and Kevin W.Boyack. 2012. Design and Update of a ClassificationSystem: The UCSD Map of Science. PLOS ONE 7, 7(July 2012), e39464.

[Cimini et al. 2014] Giulio Cimini, Andrea Gabrielli, andFrancesco Sylos Labini. 2014. The ScientificCompetitiveness of Nations. PLOS ONE 9, 12 (Dec.2014), e113470.

[Daraio et al. 2015] Cinzia Daraio, Andrea Bonaccorsi, andLeopold Simar. 2015. Efficiency and economies of scaleand specialization in European universities: Adirectional distance approach. Journal of Informetrics9, 3 (July 2015), 430–448.

[De Domenico et al. 2016] Manlio De Domenico, ElisaOmodei, and Alex Arenas. 2016. Quantifying theDiaspora of Knowledge in the Last Century. AppliedNetwork Science 1, 15 (2016).

[Guevara et al. 2016] Miguel R. Guevara, DominikHartmann, Manuel Aristaran, Marcelo Mendoza, andCesar A. Hidalgo. 2016. The research space: usingcareer paths to predict the evolution of the researchoutput of individuals, institutions, and nations.Scientometrics 109, 3 (Dec. 2016), 1695–1709.

[Harzing and Giroud 2014] Anne-Wil Harzing and AxeleGiroud. 2014. The competitive advantage of nations:

An application to academia. Journal of Informetrics8, 1 (Jan. 2014), 29–42.

[Hidalgo and Hausmann 2009] Cesar A. Hidalgo andRicardo Hausmann. 2009. The building blocks ofeconomic complexity. Proceedings of the NationalAcademy of Sciences 106, 26 (June 2009),10570–10575.

[Leydesdorff and Rafols 2009] Loet Leydesdorff and IsmaelRafols. 2009. A global map of science based on the ISIsubject categories. Journal of the American Societyfor Information Science and Technology 60, 2 (Feb.2009), 348–362.

[Li et al. 2015] Feng Li, Yajun Miao, and Jing Ding. 2015.Tracking the development of disciplinary structure inChina’s top research universities (1998–2013).Research Evaluation 24, 3 (July 2015), 312–324.

[Lopez-Illescas et al. 2011] Carmen Lopez-Illescas, Felix deMoya-Anegon, and Henk F. Moed. 2011. A ranking ofuniversities should account for differences in theirdisciplinary specialization. Scientometrics 88, 2 (Aug.2011), 563–574.

[Moed 2016] Henk F. Moed. 2016. A critical comparativeanalysis of five world university rankings.Scientometrics (Dec. 2016), 1–24.

[Murray et al. 2016] Dennis L. Murray, Douglas Morris,Claude Lavoie, Peter R. Leavitt, Hugh MacIsaac,Michael E. J. Masson, and Marc-Andre Villard. 2016.Bias in Research Grant Evaluation Has DireConsequences for Small Universities. PLOS ONE 11,6 (June 2016), e0155876.

[Orduna-Malea et al. 2014] Enrique Orduna-Malea,Alberto Martın-Martın, Juan M. Ayllon, and EmilioDelgado Lopez-Cozar. 2014. The silent fading of anacademic search engine: the case of MicrosoftAcademic Search. Online Information Review 38, 7(2014), 936–953.

[Ortega et al. 2011] Jose Luis Ortega, ElenaLopez-Romero, and Ines Fernandez. 2011. Multivariateapproach to classify research institutes according totheir outputs: The case of the CSIC’s institutes.Journal of Informetrics 5, 3 (July 2011), 323–332.

[Pastor and Serrano 2016] Jose Manuel Pastor andLorenzo Serrano. 2016. The determinants of theresearch output of universities: specialization, qualityand inefficiencies. Scientometrics 109, 2 (Nov. 2016),1255–1281.

[Payumo et al. 2014] Jane G. Payumo, Prema Arasu,Anas Miftah Fauzi, Iskandar Zulkarnaen Siregar, andDeni Noviana. 2014. An entrepreneurial,research-based university model focused on intellectualproperty management for economic development inemerging economies: The case of Bogor AgriculturalUniversity, Indonesia. World Patent Information 36(March 2014), 22–31.

[Ricardo et al. 2008] Hausmann Ricardo, Cesar A.Hidalgo, Sebastian Bustos, Michele Coscia, SarahChung, Juan Jimenez, Alexander Simoes, andMuhammed A. Yıldırım. 2008. The atlas of economiccomplexity: Mapping paths to prosperity. TheObservatory of Economic Complexity. HarvardHKS/CDIMIT Media Lab (2008).

[Robinson-Garcıa and Calero-Medina 2014] NicolasRobinson-Garcıa and Clara Calero-Medina. 2014.

1264

Figure 4: Geographical RCI distribution

What do university rankings by fields rank? Exploringdiscrepancies between the organizational structure ofuniversities and bibliometric classifications.Scientometrics 98, 3 (March 2014), 1955–1970.

[Sandulescu and Chiru 2016] Vlad Sandulescu and MihaiChiru. 2016. Predicting the future relevance ofresearch institutions - The winning solution of theKDD Cup 2016. (Sept. 2016).

[Sinha et al. 2015] Arnab Sinha, Zhihong Shen, Yang Song,Hao Ma, Darrin Eide, Bo-june Paul Hsu, and Kuansan

Wang. 2015. An overview of Microsoft AcademicService (MAS) and applications. In Proceedings of the24th International Conference on World Wide Web.ACM, 243–246.

[Wong and Singh 2010] Poh Kam Wong and AnnetteSingh. 2010. University patenting activities and theirlink to the quantity and quality of scientificpublications. Scientometrics 83, 1 (April 2010),271–294.

1265