the multi-dimensional research assessment matrix

The Multi-Dimensional Research Assessment Matrix

Henk F. MoedElsevier, Amsterdam, the Netherlands

Seminar Research Evaluation in Practice, National Geographic Society, Washington DC, 17 October 2012

Short CV Henk F. MoedYears Position

1981-2009

Staff member at Centre for Science and Technology Studies (CWTS), Leiden Univ.

2009 Professor of Research Assessment Methodologies at Leiden University

2010 – Sept 2012

Elsevier, SciVal Dept. Senior Scientific Advisor

As from Sept 2012

Elsevier, AGRM Dept. Head of Informetric Research Group

Contents

1 Boundaries of the playing field; rules of the game

2 The multi-dimensional research assessment matrix

3 Towards more sophisticated indicators: combining big data sets

(o)Bibliometrics makes sense

Which country has these main collaborators?

Main collaborators

UK

China

Germany

Canada

USA


Main collaborators

Argentina

USA

Portugal

France

Chile

Brazil


Main collaborators

Thailand

India

Singapore

Iran

UK

Malaysia


Main collaborators

France

Hungary

Germany

Italy

Bulgaria

Romania


Main collaborators

UK

China

USA

Nigeria

Australia

Netherlands

South Africa

Contents

1 Boundaries of the playing field; rules of the game



(i) Bibliometric research assessment is

potentially a proper tool to consolidate academic freedom

(ii) Bibliometric tools help establishing

a longer term perspective in research funding

(iii) One must be cautious using

“societal benefit” as an assessment criterion of basic research:

it can not be measured in a politically neutral way

(iv) Citations measure scientific-

scholarly impact rather than quality or validity

(v) Citation counts in social sciences

and humanities may be influenced by political ideologies

Citation impact and ideology

0

500

1,000

1,500

2,000

2,500

3,000

CITATION YEAR

NR

CIT

ES

MARX

LENINFall of the Berlin wall

in Nov. 1989

(vi)The future of research assessment

lies in the intelligent combination of

metrics and peer review

(vii)Case study on funding policies of a National Research Council reveals

biases in peer review

Affinity Applicants – Evaluation Committee

0 Applicants are/were not member of any Committee

1 Co-applicant is/was member of a Committee, but not of the one evaluating

2 First applicant is/was member of a Committee, but not of the one evaluating

3 Co-applicant is member of the Committee(s) evaluating the proposal

4 First applicant is member of the Committee(s) evaluating the proposal

For 15 % of SUBMITTED applications an applicant is a member of the evaluating Committee (Affinity=3, 4)

0

10

20

30

40

50

60

70

% A

PPLI

CATI

ONS

AFFINITY APPLICANTS-COMMITTEEProjects 63.2 10.2 11.5 5.9 9.1

0 1 2 3 4% S

UB

MIT

TED

A

PPLI

CAT

ION

S

AFFINITY APPLICANT - COMMITTEE

Probability to be granted increases with increasing affinity applicants-Committee

30

40

50

60

70

80

% G

RA

NTE

D A

PPLI

CA

TON

S

AFFINITY APPLICANTS-COMMITTEE

Projects 37.0 46.9 60.1 62.6 74.0

0 1 2 3 4

AFFINITY APPLICANT - COMMITTEE

% G

RA

NTE

D

APP

LIC

ATIO

NS

Logistic regression analysis: Affinity Applicant-Committee has a significant effect

upon the probability to be granted

MAXIMUM-LIKELIHOOD ANALYSIS-OF-VARIANCE TABLE (N=2,499) Source DF Chi-Square Prob ------------------------------------------------------------- INTERCEPT 1 18.47 0.0000 CITATION IMPACT APPLICANT 3 26.97 0.0000 ** Rel transdisc impact applicant 1 0.29 0.5926 AFFINITY APPLICANT-COMMITTEE 2 112.50 0.0000 ** Sum requested 1 45.47 0.0000 ** Institution applicant 4 25.94 0.0000 ** LIKELIHOOD RATIO 199 230.23 0.0638

(viii)Data must be accurate and verifiable

(ix)Assessed researchers must have the

opportunity to check data and comment on outcomes

(x)A framework is needed to characterize and position

bibliometric indicators and products

Contents

1 Boundaries of the playing field and rules of the game



The Multi-Dimensional Research Assessment Matrix

Expert Group on the Assessment of University-Based Research (AUBR, 2010)

Multi‐dimensional Research Assessment Matrix (Part)Unit of assessment

Purpose Output dimensions

Bibliometric indicators

Other indicators

Individual Allocate resources

Research productivity

Publications Peer review

Research group

Improve performance

Quality, scholarly impact

Journal citation impact

Patents, licences, spin offs

Department Monitor research programs

Innovation and social benefit

Actual citation impact

Invitations for conferences

Institution Increase regional engagement

Sustainabi-lity & Scale

Internat. co-authorship

External research income

Research field

Promotion, hiring

Research infrastruct.

citation ‘prestige’

PhD com-pletion rates




Other indicators




Research group

Improve performance












Research field

Promotion, hiring




Read column-

wise




Other indicators




Research group

Improve performance












Research field

Promotion, hiring







Other indicators




Research group

Improve performance












Research field

Promotion& hiring




MD-RAM: Example 1

Individual Hiring/promotionProductivity &

impactPhD date, place,

supervisor;Invitations for

conferences

Publications in international jrnls;





Other indicators




Research group

Improve performance












Research field

Promotion, hiring




MD-RAM: Example 2

Research group Monitoring

research program Scientific impact Collaborations

Topicality

Publications in international jrnls;

(Trend in) actual citation impact

Research assessment methodologies: Important considerations

• Methodology must be fit-for-purpose

• What is the primary “problem” to be solved?

• Be aware of unintended effects

• Change a methodology every 5-10 years

• What is an acceptable “error rate”?

• Wrong in individual cases benificiary for the system as a whole

Contents

1 Boundaries of the playing field and rules of the game



Journal full text data

Journal usage data

Patents

Trade jrnls

Social media

Journal articles + citations

Books

Conference Procs

Newspapers

Unit of assessment

(a)Downloads vs. Citations

What do full article downloads measure?

Authors vs. readers

Authors Readers

?

Hypothesis on degree of correlation between downloads and citations

Readers

Authors

ReadersAuthors

Strong Weak

Usage vs. citations per main field

Societal

Scientific

?PSYCHOL

(b)Patent citations to journal articles:

The technological impact of research

The Technological Impact of Library Science Research: A Patent Analysis [Halevi et al, 2012]

PATENTS (TotalPatent)

42 LIBRARY SCIENCE JOURNALS (Scopus)

Citations by patent examiners and inventors

The patents focus on electronic information administration, navigation, and products and services management in commercial systems.

The articles feature information retrieval and indexing, information and documents management systems which pertain to electronic and digital libraries development

Cited articles: keywords in titles

Citing patents: keywords in titles

(c) International scientific migration

International migration vs. co-authorship

Relationship Definition Comment

International co-authorship

Authors from institutions located in different countries jointly publish a paper

Country relates to where authors work, NOT to their nationality International

migrationA scientific author moves from one country to another

Map of countries with Ratio migration/collaboration > 1.2

TO COUNTRY

FROM Country

No. Co-authored papers

No. Migrating authors

Ratio % Migration / % Co-authorship

PAK IND 276 118 3.6PRT BRA 1,971 423 3.4IND PAK 276 96 3.3NLD IRN 492 80 3.1USA IND 12,013 3,307 2.8NLD PAK 145 21 2.8CHN TWN 3,979 1,048 2.6USA IRN 3,039 780 2.6BRA PRT 1,971 352 2.4MYS NGA 122 31 2.1

Language similarity drives migration stronger than it drives co-authorship

Political tensions affect migration less than they affect co-authorship

(d)Citation context analysis

Combining citation data from Scopus with full text article data from ScienceDirect

The use of contextual citations analysis to disclose the thematic and conceptual

flow of cross- disciplinary research:

the case of the Journal of Informetics 2007

(Gali Halevi et al., to be published, 2012)

Emerging sectional themes (OUTSIDE DISCIPLINE)

The themes sequence in the word clouds below might suggest that the individual output evaluation done by structured peer review leads to an acknowledgment of

the importance and evolution of networks rather than individuals

INTRODUCTION FINDINGS & DISCUSSION

CONCLUSIONS

(e)Book Citation Index

Citation flows between books and journals

A Scholarly Book Citation Index: Approaches

a) Add selected book series

b) Add books from selected publishers

c) Add selected individual book titles

Thank you for your attention!

Elsevier Bibliometric Research Program:

www.ebrp.elsevier.com

the multi-dimensional research assessment matrix

Documents

affinity applicantcommittee

research funding

evaluating committee

citation impact applicant

national research council

staff member

rel transdisc impact

sensewhich country