Slide 1

Research evaluation at CWTS Meaningful metrics, evaluation in context Ed Noyons, Centre for Science and Technology Studies, Leiden University RAS Moscow, 10 October 2013

Slide 2

Outline Centre of science and Technology Studies (CWTS, Leiden University) history in short; CWTS research program; Recent advances.

Slide 3

History in Short 25 years CWTS 3

Slide 4

25 years CWTS history in short (1985-2010) Started around 1985 by Anthony van Raan and Henk Moed; One and a half person funded by university; Context is science policy, research management; Mainly contract research and services (research evaluation); Staff stable around 15 people (10 researchers); Main focus on publication and citation data (in particular Web of Science).

Slide 5

25 years CWTS history in short (2010 - ) Block funding since 2008; Since 2010 moving from Services mainly with some research to: Research institute with services; New director Paul Wouters; New recruitments: now ~35 people. 5

Slide 6

CWTS Research programme Research and services 6

Slide 7

Bibliometrics (in context science policy) is...

Slide 8

Opportunities Research Accountability => evaluation Need for standardization, objectivity More data available

Slide 9

Vision Quantitative analyses Beyond the lamppost Other data Other outputs Research 360 Input Societal impact/quality Researchers themselves

Slide 10

Background of the CWTS research program Already existing questions New questions: 1.How do scientific and scholarly practices interact with the social technology of research evaluation and monitoring knowledge systems? 2.What are the characteristics, possibilities and limitations of advanced metrics and indicators of science, technology and innovation?

Slide 11

Current CWTS research organization Chairs Scientometrics Science policy Science Technology & innovation Working groups Advanced bibliometrics Evaluation Practices in Context (EPIC) Social sciences & humanities Society using research Evaluation (SURE) Career studies

Slide 12

Back to Bibliometrics A look under the lamp post 12

Slide 13

Recent advances at CWTS Platform: Leiden ranking Indicators: New normalization to address: 1.Multidisciplinary journals 2.(Journal based) classification Structuring and mapping Advanced network analyses Publication based classification Visualization: VOSviewer

Slide 14

The Leiden Ranking http://www.leidenranking.com 14

Slide 15

Platform: Leiden Ranking http://www.leidenranking.com http://www.leidenranking.com Based on Web of Science (2008-2011); Only universities (~500); Only dimension is scientific research; Indicators (state of the art): Production Impact (normalized andabsolute) Collaboration. 15

Slide 16

Leiden Ranking world top 3 (PPtop10%) 16 PPtop10%: Normalized impact PPtop10%: Normalized impact Stability: Intervals to enhance certainty Stability: Intervals to enhance certainty

Slide 17

Russian universities (impact)

Slide 18

Russian universities (collaboration)

Slide 19

Impact Normalization (MNCS) Dealing with field differences 19

Slide 20

20 Background and approach Impact is measured by numbers of citations received; Excluding self-citations; Fields differ regarding citing behavior; One citation is one field is more worth than in the other; Normalization By journal category By citing context.

Slide 21

Issues related to journal category-based approach Scope of category; Scope of journal. 21

Slide 22

Journal classification challenge(scope of category) (e.g. cardio research)

Slide 23

23 Approach Source-normalized MNCS Source normalization (a.k.a. citing-side normalization): No field classification system; Citations are weighted differently depending on the number of references in the citing publication; Hence, each publication has its own environment to be normalized by.

Slide 24

24 Source-normalized MNCS (contd) Normalization based on citing context; Normalization at the level of individual papers (e.g., X) Average number of refs in papers citing X; Only active references are considered: Refs in period between publication and being cited Refs covered by WoS.

Slide 25

Networks and visualization Collaboration, connectedness, similarity,... 25

Slide 26

VOSviewer: collaboration Lomonosov Moscow State University (MSU) 26 WoS (1993-2012) Top 50 most collaborative partners Co-published papers

Slide 27

Other networks Structure of science output (maps of science); Oeuvres of actors; Similarity of actors (benchmarks based on profile); 27

Slide 28

Publication based classification Structure of science independent from journal classification 28

Slide 29

Publication based classification (WoS 1993- 2012) Publication based clustering (each pub in one cluster); Independent from journals; Clusters based on Citing relations between publications Three levels: Top (21) Intermediate (~800) Bottom (~22,000) Challenges: Labeling Dynamics. 29

Slide 30

Map of all sciences (784 fields, WoS 1993-2012) Each circle represents a cluster of pubs Surface represents volume Distance represents relatedness (citation traffic) Distance represents relatedness (citation traffic) Physical sciences Earth, Environ, agricult sciences Biomed sciences Cognitive sciences Social and health sciences Maths, computer sciences Colors indicate clusters of fields, disciplines

Slide 31

Positioning of an actor in map Activity overall (world and e.g., Lomonosov Moscow State Univ, MSU) o Proportion Lomonosov relative to world; Activity per field (world and MSU) o Proportion MSU in field; Relative activity MSU per field; Scores between 0 (Blue) and 2 (Red); 1 if proportion same as overall (Green). 31

Slide 32

Positioning Lomonosov MSU 32

Slide 33

Positioning Lomonosov MSU 33

Slide 34

Positioning Russian Academy of Sciences (RAS) 34

Slide 35

Alternative view Lomonosov (density) 35

Slide 36

Using the map: benchmarks Benchmarking on the basis of research profile Distribution of output over 784 fields; Profile of each university in Leiden Ranking; Distributions of output over 784 fields; Compare to MSU profile; Identify most similar. 36

Slide 37

Most similar to MSU (LR) universities FR - University of Paris-Sud 11 RU - Saint Petersburg State University JP - Nagoya University FR - Joseph Fourier University CN - Peking University JP - University of Tokyo 37

Slide 38

Density view MSU 38

Slide 39

Density view St. Petersburg State University 39

Slide 40

VOSviewer (Visualization of Similarities) http://www.vosviewer.com http://www.vosviewer.com Open source application; Software to create maps; Input: publication data; Output: similarities among publication elements: Co-authors Terms co-occurring Co-cited articles 40

Slide 41

More information CWTS and methods www.cwts.nl www.journalindicators.com www.vosviewer.com noyons@cwts.leidenuniv.nl 41

Slide 42

THANK YOU 42

Slide 43

Basic model in which we operate (research evaluation) Research in context

Slide 44

Example (49 Research communties of a FI univ) Positive effect Negative effect

Slide 45

RC with apositiveeffect Most prominent field Impact increases Most prominent field Impact increases

Slide 46

Rc with anegative effect Most prominent field Impact same Most prominent field Impact same Less prominent field Impact decreases Less prominent field Impact decreases

Slide 47

Wrap up Normalization Normalization based on journal classification has its flaws; We have developed recently an alternative; Test sets in recent projects show small (but relevant) differences;