pbl netherlands environmental assessment agency (pbl) · bibliometric study for pbl 8 2 data and...

33
PBL Netherlands Environmental Assessment Agency (PBL): Research performance analysis (2011-2016)

Upload: others

Post on 09-Aug-2020

4 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

PBL Netherlands Environmental

Assessment Agency (PBL):

Research performance analysis (2011-2016)

Page 2: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses
Page 3: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Center for Science and Technology Studies (CWTS)

Leiden University

PO Box 9555, 2300 RB Leiden

The Netherlands

September 16, 2017

Page 4: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

PBL – Netherlands Environmental Assessment Agency

Drs. Ton Manders

Sector Head, Sector Information, Data and Methods

Tel. +31 655 73 69 18

E-mail [email protected]

Project team

Alfredo Yegros-Yegros PhD, Project leader

CWTS B.V.

P.O. Box 905

2300 AX Leiden, The Netherlands

Tel. +31 71 527 7737

Fax +31 71 527 3911

E-mail [email protected]

Page 5: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Contents

Page 6: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

1 Introduction ...................................................................................................................................... 7

2 Data and methods ......................................................................................................................... 8

2.1 Database Structure .................................................................................................... 8

2.2 Data collection .......................................................................................................... 8

2.3 Bibliometric Indicators ............................................................................................. 10

2.3.1 Indicators of output ......................................................................................... 10

2.3.2 Indicators of impact ......................................................................................... 10

2.3.3 Indicators of scientific collaboration.................................................................. 13

3 Performance analysis: Netherlands Environmental Assessment Agency 15

3.1 Publication output ................................................................................................... 15

3.2 Citation impact ........................................................................................................ 17

3.3 Special indicators .................................................................................................... 20

3.3.1 Research profile analysis .................................................................................. 20

3.3.2 Collaboration analysis ..................................................................................... 22

3.3.3 Knowledge user analysis .................................................................................. 26

4 Conclusions .................................................................................................................................... 27

Appendix I: Calculation of the MNCS indicator ........................................................................ 29

Appendix II: Citation indicators ......................................................................................................... 32

Appendix III: Complete research profile ....................................................................................... 32

Page 7: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

7

1 Introduction

The Netherlands Environmental Assessment Agency (PBL) will have its research

evaluation taking place in 2017. PBL follows the Standaard Evaluatie Protocol (SEP) in

terms of its evaluation.

The mission of PBL is to support and guide policy through scientific research. As

such, the assessment of its scientific performance based on bibliometric analyses should

be placed in the right context, being different than universities or academic hospitals.

The structure of this report is as follows. Chapter 2, provides a general

introduction to the data collection, the methodology and an overview of the bibliometric

indicators that were calculated in the study. Chapter 3, presents the results of the

performance analysis and Chapters 4 briefly presents the conclusions drawn from this

study.

Page 8: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

8

2 Data and methods

In this chapter, we discuss the methods underlying the bibliometric analyses

presented in this report1.

2.1 Database Structure

At CWTS, we calculate our indicators based on our in-house version of the Web of

Science (WoS) database of Thomson Reuters. WoS is a bibliographic database that covers

the publications of about 12,000 journals in the sciences, the social sciences, and the arts

and humanities. Each journal in WoS is assigned to one or more subject categories. We

note that our in-house version of the WoS database includes a number of improvements

over the original WoS database. Most importantly, our database uses a more advanced

citation matching algorithm and an extensive system for address unification. Our

database also supports a hierarchically organized field classification system on top of the

WoS subject categories.

2.2 Data collection

In general, bibliometric performance analyses can be conducted in two ways. One

approach is to collect the publications produced by a research unit in the past and to

analyze these publications. The other approach is to start with the researchers currently

affiliated with a research unit and to collect and analyze their past publications,

irrespective of whether researchers produced these publications when they were

affiliated with the research unit of interest or not. The first approach leads to an analysis

that is completely backward looking. The second approach has a more forward looking

1 We refer to Moed (2005) for a general introduction into the use of bibliometrics and citation analysis for

research evaluation.

Page 9: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

9

focus, since it does not include publications produced by researchers who are no longer

affiliated with a research unit. PBL has requested CWTS to take the first approach in the

performance analysis presented in this report. Hence, we analyze the past publications of

PBL.

The performance analysis presented in this report focuses on publications from

the period 2011-2016. Only WoS indexed publications are considered. This means that

books, book chapters, journal publications not indexed in WoS, conference proceedings

publications, working papers, etc. are not included in the analysis. Each publication in

WoS has a document type, such as ‘article’, ‘book review’, ‘editorial material’, ‘letter’, or

‘review’. In our analysis, although we first show all the publication counts for all types of

document, we only take into account publications of the document types ‘article’ and

‘review’ to conduct the main analyses. In general, these two document types cover the

most significant publications and reflect best the research developments.

The publications of PBL were collected from the CWTS CI-system and sent to the

PBL to verify and check any wrong assignment of publications to PBL or any possible

missing publication by PBL during the period of analysis.

Page 10: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

10

2.3 Bibliometric Indicators

Three key aspects of PBL’s research performance are considered in our

performance analysis: publication output, citation impact, and scientific collaboration.

2.3.1 Indicators of output

To measure the total publication output produced by PBL, we use a very simple

indicator. This is the number of publications indicator, denoted by P. This indicator is

calculated by counting the total number of publications of a research unit.

2.3.2 Indicators of impact

Citation impact focuses on the number of times the publications of PBL have been

cited. Citation impact does not necessarily reflect the scientific quality of the work of PBL,

but it can be regarded as a proxy for the scientific impact of this work.

As already mentioned in Section 2.2, the period for the performance analysis is

2011-2016, however the citation analysis considers publications up to 2015 and citations

until 2016, as at least one complete year to receive citations is needed to calculate robust

indicators. In addition, as indicated previously, only publications of the WoS document

types ‘article’ and ‘review’ are taken into account. Hence, book reviews, editorials, letters

to the editor, etc. are not included in the analysis.

It is important to note that in this report we distinguish between two different

concepts of citation impact:

Total citation impact (TCS). The overall citation impact of the publications of a

research unit. Other things equal, a research unit with a larger number of

Page 11: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

11

publications will have a higher total citation impact. Hence, total citation impact is

partly determined by the size of a research unit.

Average citation impact per publication (MCS). The average citation impact of the

publications of a research unit. Average citation impact per publication equals total

citation impact divided by the number of publications of a research unit. Average

citation impact per publication makes it possible to compare research units of

different size. Research units with a selective publication strategy (favoring

‘quality’ over ‘quantity’) are the ones that tend to perform best when looking at

average citation impact per publication.

To measure the total or average citation impact of a set of publications, we start by

counting for each publication the number of times it has been cited. Since our analysis is

based on WoS data, only citations from WoS indexed publications are counted. We

normally do not count author self citations. A citation is considered an author self citation

if the citing and the cited publication have at least one author name in common. For each

publication, all citations received until the end of 2016 are taken into account, we use the

so called flexible citation window. This means that older publications have had more time

to receive citations and may therefore be expected to have higher citation counts than

more recent publications.

After counting the number of times publications have been cited, we calculate the

following indicators, considering both aspects, the total citation impact of a set of

publications as well as the average citation impact:

TCS. The total number of citations of the publications, excluding self-citations. This

is a very straightforward indicator that does not correct for the field and the year

Page 12: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

12

in which publications have appeared. The indicator therefore provides only a very

rough indication of the total citation impact of a set of publications.

Number of top 10% publications (PP(top 10%)). The number of publications

that compared with all other WoS indexed publications in the same field and the

same year belong to the top 10% most frequently cited. We refer to these

publications as top 10% publications.

MCS: Average number of citations per publication, excluding self-citations.

Pnc: Percentage of publications not cited by others (in the given time period)

MNCS: Average normalized number of citations of the publications of a unit,

excluding self-citations.

Total normalized journal score (MNJS). Average normalized citation score of the

journals in which a research group has published

Mean field normalized citation score (MNCS) in the traditional way; the actual

number of citations (without self-citations) is divided by the expected number of

citations on a paper basis. Here, the expected number of citations is based on the

world-wide average citation score without self-citations of all similar papers

belonging to the same field (journal subject category). In this way, a field

normalized score is calculated for each paper. Next, the MNCS indicator is

computed for each unit of analysis, by taking the average of these field normalized

citation scores for individual papers. A value above 1 indicates that the mean

impact for the unit is above world average whereas a value below 1 indicates the

opposite.

The mean normalized journal score (MNJS) indicates the average citation

impact of the journals in which the papers appeared that were published by the

Page 13: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

13

unit of analysis. The indicator is calculated based on the same principles as the

MNCS. It shows whether the publications originating from the unit of analysis were

published in top or in sub-top (in terms of citation impact) journals.

Number of highly cited publications (Ptop10%) in international journals of the

unit of analysis in the period;

Percentage of highly cited publications. (PPtop10%) The percentage of

publications published by the unit that are among the upper top 10% percentile of

the citation distribution for similar papers belonging to the same fields (journal

subject categories).

For those indicators that involve a normalization by scientific field, we used the

publication-level classification system developed at CWTS. This classification clusters

groups of publications based on their citation relations, so that similar publications based

on their citation profiles are grouped together. This approach leads to a more accurate

and fair comparison between areas of research compared to other available classification

systems2

2.3.3 Indicators of scientific collaboration

Collaboration is measured according to the degree to which the publications of PBL

indicate multiple research institutes, from the Netherlands or abroad. Collaboration is

measured then analyzing the affiliations indicated by the authors in their publications.

2

van Eck NJ, Waltman L, van Raan AFJ, Klautz RJM, Peul WC (2013) Citation Analysis May Severely

Underestimate the Impact of Clinical Research as Compared to Basic Research. PLoS ONE 8(4): e62395.

https://doi.org/10.1371/journal.pone.0062395

Page 14: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

14

We first identified publications authored by a single institution (‘no collaboration’).

We then identified publications that have been produced by institutions from different

countries (‘international collaboration’) and publications that have been produced by

institutions from the same country (i.e. ‘national collaboration’). These types of

collaboration are mutually exclusive. Publications involving both national and

international collaboration are classified as international collaboration.

Page 15: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

15

3 Performance analysis: Netherlands Environmental Assessment Agency

This section is devoted to present the performance analysis of the Netherlands

Environmental Assessment Agency, based on publications from 2011 to 2016.

The section is organized in three subsections. The first of these subsections presents

an overview of the scientific outputs produced by PBL. In the second subsection the main

indicators of citation impact are presented and the last subsection shows the so called

special indicator: a research profile analysis, a collaboration profile analysis and, finally,

the knowledge user analysis.

3.1 Publication output

The PBL has contributed to 430 scientific publications covered by the WoS during the

period 2011-2016. Therefore, on average, PBL has contributed to roughly 70 publications

per year.

Figure 1 shows the evolution over time of the number of publications. It can be

observed that 2012 and 2013 were the years with a lower publication output, with slightly

less than 70 publications in each of these two years. In 2014 the numbers improved and

remained relatively stable till the end of the period under analysis.

Page 16: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

16

Figure 1. Evolution over time of the number of publications

Table 1 shows the breakdown of PBL publications per document type and publication

year. Most of the contributions made by the PBL were done through articles (86.7%).

Reviews and editorial material represent each roughly 5% of the total output.

The remaining types of publications represent very little in the overall landscape of

publications produced by PBL.

Table 1. Breakdown of PBL output in the WoS by document type

Document type 2011 2012 2013 2014 2015 2016 Total (%)

Article 59 56 60 67 67 64 373 (86.7)

Book Review 1 1 2 4 (0.9)

Correction 1 1 (0.2)

Editorial Material 5 2 1 4 5 2 19 (4.4)

Letter 2 1 1 1 1 1 7 (1.6)

Meeting abstract 2 2 (0.5)

Review 4 2 3 5 5 5 24 (5.6)

Total (%) 72 (16.7) 61 (14.2) 66 (15.3) 78 (18.1) 79 (18.4) 74 (17.2) 430

The rest of the analyses will be conducted only for articles and reviews. All the other

document types are not included in the analysis as they are considered to be less

Page 17: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

17

representative of the research conducted at PBL. Articles and reviews together account

for more than 90% of the publications by PBL included in the WoS.

3.2 Citation impact

For the citation impact indicators only reviews and articles from 2011 to 2015 are

considered, and all the citations received till 2016.

Table 2 contains the main indicators on citation impact of PBL for the whole

period. A table with the scores of these indicators for each individual year in the period

can be found in Appendix II (table A2.1.).

Limiting the period to the years 2011 to 2015 and the document types to articles

and reviews, reduces the amount of publications produced by PBL to 328, these are the

publications analysed.

The internal coverage (int_cov) indicates the percentage of references included in

PBL publications that in turn point to scientific publications covered by the WoS. This

indicator is used to assess the suitability of bibliometrics to assess the scientific

performance of research organisations. A score around 60% indicates that a bibliometric

analysis provides a good overview of the research performance of PBL, based on scientific

publications3.

Looking at the main citation indicators, the mean normalized citation score

(MNCS) achieved by PBL is 3.31, more than three times above the world average value

(1). Also the proportion of its publications among the most highly cited publications is

three times above the worldwide reference value of 10% (33.7%).

The indicator of percentage of non-cited publications (Pnc) provides an interesting

additional point of view to get a more comprehensive picture of the citation impact of PBL.

The score of this indicator reflects that only 5.8% of all the publications remained uncited

till 2016, therefore the vast majority of publications contributed to shape the citation

impact of PBL.

3

Scores around 40% are problematic as indicate that publications in scientific journals are not

frequent for the research organization under analysis and therefore bibliometric analyses do not provide

a good overview of the research performance.

Page 18: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

18

Besides the citation indicators directly related to the publications produced by

PBL, the mean normalized journal score (MNJS) of 2.22 also indicates that PBL has been

publishing in journals in which publications are cited twice as much as in a worldwide

average journal. Therefore the journals to which PBL has targeted its publications are well

cited journals.

Table 2. Indicators of citation impact of the publications of PBL (2011-2015/16)

Indicator Score

P 328

Int_cov4 62.1%

TCS 7,700

MCS 23.5

Pnc 5.8%

MNCS 3.31

MNJS 2.22

Ptop10 110

PPtop10% 33.7%

Figure 2 and Figure 3 show the evolution of two indicators of citation imapact: the

mean normalized citation impact (MNCS) and the proportion of publications among the

most highly cited publications (Pp top 10%). Both present a similar pattern, although not

identical.

Both indicators suggest that publications with the highest citation impact

correspond to the years 2011 and 2014. However, even in those years in which the

citation impact is lower, it is still above 2 for the MNCS and around 25% for the Pp top

10%.

4

Percentage of references in PBL publications that are also covered by the WoS.

Page 19: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

19

Figure 2. Evolution of the Mean Normalized Citation Score

Figure 3. Evolution of the percentage of top 10% most cited publications

Page 20: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

20

3.3 Special indicators

This section of the report contains the last part of the analysis conducted to assess

the performance of the PBL. The special indicators described below correspond to the

research profile, the collaboration analysis and the knowledge user analysis.

3.3.1 Research profile analysis

This research profile analysis is devoted to describe the disciplinary profile of PBL

based on its scientific publications and to show both the amount of scientific publications

and the citation impact of PBL in each individual scientific discipline.

Figure 4 shows a breakdown of publications by scientific fields as represented in

the WoS. Next to each scientific field, between brackets, it has been indicated the value of

the mean normalized citation score (MNCS) achieved by the publications in that field. The

share of publications in each particular field is also shown in the figure, providing a visual

representation of the main fields also in terms of the number of publications.

Figure 4 only shows fields accounting for at least 1% of the PBL publications. A

table including a complete list of fields is provided in Appendix III (table A3.1)

Page 21: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

21

Figure 4. Distribution of publications over WoS fields

Page 22: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

22

3.3.2 Collaboration analysis

In order to assess the collaboration patterns of PBL, publications have been

divided in three groups. Those classified as no collaboration only include the PBL among

the affiliations. Publications classified as international collaboration are those in which

the PBL as well as at least another Dutch organization are involved, but without the

participation of foreign organisations. Any publication in which at least one foreign

organization is involved is classified as international collaboration.

As shown in Figure 5, the group of international collaborations accounts for most

of the PBL publications (61.9%). The second biggest group includes the publications

produced in national collaboration (31.7%), while the publications produced only by the

PBL represent 6.4% of all the publications.

In terms of citation impact, all the publications by PBL can be regarded as having

achieved a high citation impact (all above an MNCS of 1.2). However, there are differences

in the MNCS achieved by these three groups of publications: 4.26 in international

collaboration; 1.74 in national collaboration and 2.01 when there is no institutional

collaboration.

Page 23: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

23

Figure 5. Output and impact per collaboration type

Figure 6 and Figure 7 present the evolution of the number of publications and the

mean normalized citation score (MNCS) according to the type of collaboration. Figure 6

indicates that the number of publications produced in international collaboration has

always been higher than the number of publications produced through other

collaboration types, and this patter is even more marked in the second half of the period.

International collaborations are important not only because of the amount of

publications, but also because the level of citation impact achieved through this type of

collaborations, most of the times higher compared to other types of collaboration, as

shown in figure 7.

Page 24: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

24

Figure 6. Development over time of publications according to the type of collaboration

Figure 7. Development over time of the MNCS according to the type of collaboration

Page 25: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

25

Table 3 shows the organisations that co-publish more frequently with the PBL.

Some Dutch universities populate the table showing the top research partners of PBL, as

reflected in the scientific publications, but the table also includes a number of foreign

organisations such as the Potsdam Institute for Climate Impact Research (PIKP) or

International Institute for Applied Systems Analysis – IIASA, among others.

Table 3. Most frequently occurring research institutes co-publishing with PBL

Research organisation No. pub.

Utrecht University 131

Wageningen University and Research Centre 96

VU University Amsterdam 44

Potsdam Institute for Climate Impact Research (PIKP) 44

International Institute for Applied Systems Analysis - IIASA 42

United States Department of Energy 24

National Center for Atmospheric Research 22

University of Amsterdam 21

ETH Zurich 19

Joint Research Centre of the European Commission in Ispra 18

Page 26: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

26

3.3.3 Knowledge user analysis

The knowledge user analysis focusses on research organisations that ‘use’ the

knowledge generated by PBL. This analysis is based on the research organisations whose

publications include in the reference list one or more scientific publications produced by

PBL.

This analysis provides an interesting perspective on the citation impact generated

by PBL. It can also point to the identification of potential research partners for future

collaborations, as the citation relations reflect to some extent the existence of common

research interests between PBL and the citing organization.

Table 4 shows the organizations that most frequently refer to publications

produced by PBL and the percentage of all the citations received by PBL these

organisations represent.

Among the top knowledge users there are some Dutch organisations, like Utrecht

University and Wageningen University but also other important organisations from

different countries, such as the University of California in the US, ETH Zurich in

Switzerland or the Chinese Academy of Sciences.

Table 4 Knowledge user profile for the PBL by citing research unit, 2011‐2015/16

Organisation % Citations

Utrecht University 7.5% Wageningen University and Research Centre 7.2% Potsdam Institute for Climate Impact Research (PIKP) 6.6% International Institute for Applied Systems Analysis - IIASA 6.4% University of California 5.6% United States Department of Energy 4.7% ETH Zurich 4.2% National Center for Atmospheric Research 4.2% Chinese Academy of Sciences 4.1%

Page 27: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

27

4 Conclusions

This report describes the research performance analysis conducted to

assess, from a bibliometric perspective, the scientific activities of the Netherlands

Environmental Assessment Agency (PBL). The study has considered the scientific

publications produced by PBL in the period 2011 to 2016 and covered by the Web

of Science.

During the whole period of analysis, PBL has contributed to 430 scientific

publications. Most of these publications are articles or reviews (more than 90%).

The number of publications by PBL has been increasing in recent years, suggesting

an improvement in quantitative terms. However, additional information would be

required in order to draw more solid conclusions about the observed trend, such

as the number of PBL staff, or the amount of scientific publications included in the

WoS.

The analysis of the citation impact achieved indicates that publications

produced by PBL are cited well above world average values (MNCS=3.31 or Pp top

10%=33.7%). Also PBL published in well cited journals (MNJS=2.22). The scores

achieved are remarkably high considering the specific nature of PBL, a research

institute whose main mission is to support decision making in the areas of

environment, nature and spatial planning.

Additional citation impact indicators, like the percentage of non-cited

publications (Pnc=5.8%), also shows that most of the publications contributed to

the overall citation impact achieved by PBL.

Page 28: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

28

The PBL has been active in 58 out of the 250 fields represented in the WoS.

This disciplinary profile shows that PBL activity is highly concentrated in a small

number of fields related to environmental research areas and meteorology. Only

the three main fields in terms of number of publications ‘Environmental Sciences’,

‘Environmental Studies’ and ‘Meteorology & Atmospheric Sciences’ account for

more than 50% of all the scientific output produced by PBL. The scientific impact

achieved by PBL in each scientific field can be regarded as high (MNCS>1.2). There

are only a few scientific fields in which the citation impact can be considered low

(e.g. Urban studies), and this occurs in fields that are not representative of the

research conducted at PBL judging from the relatively low number of publications

involved.

The analysis of the collaboration profile of PBL indicates that research is

done predominantly in institutional collaboration, as more than 90% of the

publications are produced in collaboration with at least another organization.

Especially important are publications produced in collaboration with one or more

foreign organisations, in terms of both the amount of publications involved

(61.9%) and the specially high citation impact achieved (MNCS=4.26).

The role of foreign research partners is also reflected in the list of main

collaborating organisations. It is a mix of Dutch and foreign research

organisations, suggesting a balanced portfolio of research partners including both,

domestic and international.

Page 29: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

29

Appendix I: Calculation of the MNCS indicator

To illustrate the calculation of the MNCS indicator, we consider a

hypothetical research group that has only five publications. Table A1 provides

some bibliometric data for these five publications. For each publication, the table

shows the scientific field, to which the publication belongs, the year in which the

publication appeared, and the actual and the expected number of citations of the

publication. (For the moment, the last column of the table can be ignored.) The five

publications are all of them document type article. Citations have been counted

using a variable-length citation window. As can be seen in the table, publications

1 and 2 have the same expected number of citations. This is because these two

publications belong to the same field and have the same publication year and the

same document type. Publication 5 also belongs to the same field and has the same

document type. However, this publication has a more recent publication year, and

it therefore has a smaller expected number of citations. It can further be seen that

publications 3 and 4 have the same publication year and the same document type.

The fact that publication 4 has a larger expected number of citations than

publication 3 indicates that publication 4 belongs to a field with a higher citation

density than the field in which publication 3 was published. The MNCS indicator

equals the average of the ratios of actual and expected citation scores of the five

publications. Based on Table 1, we obtain

Page 30: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

30

Hence, on average the publications of our hypothetical research group have

been cited more than twice as frequently as would be expected based on their

field, publication year, and document type.

Table A1: Bibliometric data for the publications of a hypothetical research group.

To illustrate the calculation of the PPtop 10% indicator, we use the same

example as we did for the MNCS indicator. Table A1 shows the bibliometric data

for the five publications of the hypothetical research group that we consider. The

last column of the table indicates for each publication the minimum number of

citations needed to belong to the top 10% of all publications in the same field and

the same publication year and of the same document type.5 Of the five

publications, there are two (i.e., publications 2 and 4) whose number of citations

is above the top 10% threshold. These two publications are top 10% publications.

It follows that the PPtop 10% indicator equals

5 If the number of citations of a publication is exactly equal to the top 10% threshold, the publication

is partly classified as a top 10% publication and partly classified as a non-top-10% publication. This is done

in order to ensure that for each combination of a field, a publication year, and a document type we end up

with exactly 10% top 10% publications.

Page 31: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

31

In other words, top 10% publications are four times overrepresented in the

set of publications of our hypothetical research group.

Page 32: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

32

Appendix II: Citation indicators

Table A2.1. Indicators of citation impact of the publications of PBL (2011-2015/16)

Period P Int_cov TCS MCS Pnc MNCS MNJS Ptop10 PPtop 10%

2011-2015 328 62.1% 7,700 23.5 5.8% 3.31 2.22 110 33.7%

2011 63 61.6% 3790 60.2 0.0% 4.70 1.97 23 35.8%

2012 58 57.1% 1118 19.3 1.7% 2.33 1.99 15 26.1%

2013 63 64.2% 1284 20.4 4.8% 3.03 2.11 23 36.5%

2014 72 63.5% 1165 16.2 5.6% 4.05 3.19 32 44.5%

2015 72 63.1% 343 4.8 15.3% 2.41 1.74 18 24.7%

Appendix III: Complete research profile

Table A3.1. Research profile of the PBL

WoS field % P MNCS

Environmental Sciences 94.0 3.38

Environmental Studies 37.8 2.15

Meteorology & Atmospheric Sciences 35.8 6.23

Ecology 17.0 3.50

Economics 15.1 1.71

Geography 13.8 2.18

Geosciences, Multidisciplinary 12.0 7.17

Energy & Fuels 8.9 1.36

Planning & Development 8.6 2.69

Engineering, Environmental 8.5 1.90

Agronomy 6.5 2.88

Agriculture, Multidisciplinary 6.4 1.80

Urban Studies 6.0 0.66

Business 5.3 3.44

Water Resources 4.9 1.74

Biology 4.8 5.82

Green & Sustainable Science & Technology 4.4 2.44

Geography, Physical 3.5 5.10

Agriculture, Dairy & Animal Science 3.1 4.57

Computer Science, Interdisciplinary Applications 3.0 1.48

Oceanography 2.5 2.86

Marine & Freshwater Biology 2.5 1.14

Public, Environmental & Occupational Health 2.4 2.97

Transportation 2.3 0.76

Biodiversity Conservation 2.2 4.23

Thermodynamics 2.0 2.18

Public Administration 2.0 2.16

Page 33: PBL Netherlands Environmental Assessment Agency (PBL) · Bibliometric Study for PBL 8 2 Data and methods In this chapter, we discuss the methods underlying the bibliometric analyses

Bibliometric Study for PBL

33

Fisheries 2.0 0.77

Limnology 1.5 1.90

Entomology 1.5 0.69

Transportation Science & Technology 1.0 1.36

Remote Sensing 1.0 0.85

Social Issues 1.0 1.39

Evolutionary Biology 0.5 11.75

Forestry 0.4 3.96

Anthropology 0.4 4.95

Engineering, Civil 0.4 2.27

Sociology 0.3 2.71

Toxicology 0.3 13.17

Food Science & Technology 0.3 4.97

Soil Science 0.3 13.34

Agricultural Economics & Policy 0.3 4.16

Chemistry, Multidisciplinary 0.3 0.41

Engineering, Chemical 0.3 0.41

Nutrition & Dietetics 0.3 4.16

Genetics & Heredity 0.1 11.12

Zoology 0.1 2.50

Plant Sciences 0.1 17.28

Archaeology 0.1 16.91

Medicine, General & Internal 0.05 4.45

Ornithology 0.04 3.75

Psychology, Multidisciplinary 0.03 1.81

Biochemical Research Methods 0.02 1.76

Mathematical & Computational Biology 0.02 1.76

Social Sciences, Mathematical Methods 0.02 30.78

Astronomy & Astrophysics 0.01 16.91

Demography 0.01 16.91

Paleontology 0.01 16.91