Bibliometric Indicators to Measure Frontier Research Does the ERC Fund Frontier Research?

AIT:

Edgar Schiebel

Marianne Hörlesberger

Thomas Scherngell

Dirk Holste

CNRS INIST:

Ivana Roche

Dominique Besagni

Claire François

Pascal Cuxac

DBF - Development of a bibliometric model for the identification of frontier research

Coordinated Support Action

support the ongoing monitoring and evaluation work as well as future

strategy and policy development

The ERC peer review system is at the very heart of the ERC's operations

and a crucial element in realising its scientific strategy. Analysis is needed

to monitor the effectiveness and efficiency of the peer review process

(including its implementation) and to understand the particular dynamics and

considerations at play in the ERC Monitoring process of selecting

successful applicants, taking account of the interplay between scientific

and administrative aspects of the process.

2 14.03.2013

DBF - Basics

From 2009-09-01 to 2013-02-28

Partners:

AIT Austrian Institute of Technology

Institute for Scientific and Technical Information (INstitut de l’Information

Scientifique et Technique - INIST) of the French National Center for

Scientific Research (CNRS)

3 14.03.2013

DBF - stages

Design, implement and test a bibliometric approach – based on aspects of

frontier research identified in grant proposals

Compare the expert-based peer review and bibliometric evaluation and draw

lessons learned

Engage with stakeholders and identify aspects of the bibliometric approach

to support the selection of (high-quality, risk-affine and reward-delivering)

frontier research

4 14.03.2013

Content

Basic ERC review process

Model and approach (basic process, attributes for frontier research)

Indicators (definitions, calculation, results and discussion)

Risk

Applicability

Interdisciplinarity

Novelty

Econometric decision model and results (today - afternoon)

5 14.03.2013

Basic ERC Review Process

6 14.03.2013

Proposals for research grants

ERC peer review panels

yes no Frontier research Other research

Model and Approach

Does the ERC Promote Frontier Research in Europe?

7 14.03.2013

Application of bibliometric

indicators for frontier

research on proposals

Do the decision criteria for

„yes“ or „no“ fulfill attributes of

frontier research?

Application of the decision

modell on the outcome of the

ERC peer review panels

Define Frontier Research

Use the HLE-Group Definition for Frontier Research

8 14.03.2013

Proposals for research grants

ERC peer review panels

yes no Other research

Attibutes for frontier research*

•Risk

•Applicability

•Interdisciplinarity

•Novelty

(*) Frontier research. The European Challenge, High Level Expert Group, EC 2005

How to understand attributes of frontier research? with a perspective to establish bibliometric indicators

9 14.03.2013

Frontier research

key attribute (EC, 2005)

Interpretation

Risk: “Frontier research is an intrinsically risky

endeavor…Researchers must be bold and take

risks.…”

Does the investigator take personal risk by

extending his existing research background by a

new research issue?

Applicability (Pasteuresqueness): Frontier

research “… may well be concerned with both

new knowledge … and with generating

potentially useful knowledge at the same time.”

Does the activity of the investigator indicate

applicability of his research?

Science of interdisciplinary nature: Frontier

research “(…) pursues questions irrespective of

established disciplinary boundaries. It may well

involve multi-, inter- or trans-disciplinary

research …”

Does the investigator use research issues,

concepts, methods, etc. from different disciplines?

Novelty: “...stands at the forefront of creating

new knowledge and developing new

understanding...”

Does the proposal use the latest research as it‘s

intellectual base? (Timeliness)

Is the research agenda part of a new upcoming

research front in the field? (Innovativeness:

Similarity to new research issues) EC (2005): Frontier research. The European Challenge, High

Level Expert Group

Risk Indicator

Does the investigator take personal risk by extending his existing

research background by a new research issue?

10 14.03.2013

Proposal Principal investigator

Proposal data:

Reference list

Reference lists of

previous publications:

Risk indicator:

similar: no risk

different: high risk

Risk Indicator: Definition

Key Attribute Risk: “Frontier research is an intrinsically risky

endeavor.…Researchers must be bold and take risks.…”, (EC, 2005)

Formalisation:

Extending the familiar research environment

• means to establish or enter a new or different knowledge base

• Knowledge bases are documented in the cited references of

– the proposal

– the publications of a scientist and

Hypothesis: The less similar the two reference profiles are, the more

independent and risky is the research

11 14.03.2013

Calculation of the RISK indicator

12 14.03.2013

ERC data Research

project proposal

-

External data

Data preperation

Extraction of the cited references

Standardizing the notation of the references

Searching for the PI and his/her references

Extracting the cited references

Cutting the latest work

Calculating the frequencies of the cited references Si

mila

rity

by

cosi

ne

of

refe

ren

ce v

ecto

rs

2006

Results for Risk Indicator: Panel PE7 2009 / StG

13 14.03.2013

Project ID cos

239700 0

239827 0

239987 0

239726 0,0220

240166 0,0223

239986 0,0234

240049 0,0241

240432 0,0263

240475 0,0293

239640 0,0387

240631 0,0393

240236 0,0443

240108 0,0490

239954 0,0583

239720 0,0595

240205 0,0676

240555 0,0717

240241 0,0786

240218 0,0873

240686 0,1032

Sorted by ascending cosine (cos),

the first twenties: more “risky” part

of the distribution

The cosine for the project ID

239700 = 0 indicates that the

references listed in the project and

the references of previous

publications of the PI are disjoint.

The higher the cosine, the closer

to the value 1 the more references

are common in the proposal and in

the recorded paper of the PI.

3 marked proposals are successful

ones

PE7 Systems and communication engineering: electronic, communication, optical and systems engineering

Discussion of the Risk Indicator

Comparison of cited references:

indicates differences in the intellectual base of the proposal against previous publications

does not include qualitative aspects

is just a hint for possible risk: necessary but not sufficient

At present state high workload due to standardization of references time consuming search for previous publications

14 14.03.2013

Applicability (Pasteuresqueness) Indicator

Does the activity of the investigator show a probability for the

application of his research?

15 14.03.2013

Proposal

Indicator: Number of

patents

Self citations in

the reference list Extraction of

Journals

Reference list of

journals for

applied or basic

research

Indicator: share

of applied journals

Pasteuresqueness Indicator: Definition

Key attribute applicability: Frontier research “… may well be concerned with

both new knowledge about the world and with generating potentially useful

knowledge at the same time. Therefore, there is a much closer and more

intimate connection between the resulting science and technology, with few

of the barriers that arise when basic research and applied research are

carried out separately.” (EC, 2005)

Formalisation of the Indicator:

Number of patents

Hypothesis: The patents applied or invented by the PI represent a

general attitude of the PI of whether or not he or she is driven by the aim

to create application relevant results

Share of Journals where the PI publishes categorized as applied vs.

fundamental research

Hypothesis: The category of the journals in which the PI uses to publish

can give an indication of the orientation applied/fundamental of the PI’s

research

16 14.03.2013

Calculation of Pasteuresqueness

17

Project IDPasteuresqueness -

Patents

Pasteuresqueness -

Applied part of PI's

publications

242837 3 1

242783 1 1

243028 1 1

242673 0 1

242859 0 1

242878 0 1

242949 0 1

241260 0 0.9523801

243171 0 0.9473684

243195 2 0.9375

242915 5 0.9090909

243073 3 0.875

242596 0 0.8333333

243113 1 0.7619048

241684 0 0.75

243033 0 0.75

242726 0 0.7368421

242771 0 0.7

242564 0 0.6666667

242699 3 0.5555556

241222 0 0.5333333

242623 0 0.4545455

242820 0 0.2272727

242381 0 0.2

243118 0 0.1538462

240381 0 0.125

242293 0 0.0625

242641 0 0.0625

242754 0 0

242772 0 0

242796 0 0

243024 0 0

243137 0 0

Example: Panel LS9 / Call 2009 / Starting Grant

18

the proposals sorted by

decreasing value of

Pasteuresqueness (journals)

Higher number of patents in the

first half, lower number of patents

in the second half

the 7 successful proposals of 32

are highlighted in green

2 successful with higher

pasteuresqueness

5 successful with lower

pasteuresqueness

Only 2 of 7 successful proposals

with patents

LS9: Applied life sciences and biotechnology: agricultural, animal, fishery, forestry and food sciences; biotechnology, chemical biology,

genetic engineering, synthetic biology, industrial biosciences; environmental biotechnology and remediation

Discussion of the Pasteuresqueness Indicator

Simple calculation, direct and straight forward, conclusive

High workload due to data availability (pdf-files)

Future improvement:

Electronically available data

available list of categorized journals

Additional alternative: PI’s co-authors’ affiliations of previous publications – share of companies, hospitals and applied research organizations

19 14.03.2013

Indicator Interdisciplinarity

Does the investigator use research issues, concepts, methods, etc.

from different disciplines?

20 14.03.2013

Proposals

ERC panels

•PE1

•PE2

•…

•PS1

•…

Assignement

disciplinary tagged keywords by

home panel: „DNA“ - Biology

Extraction of

keywords

Indicator:

number of keywords with

different home panels

Proposals

Indicator Interdisciplinarity: Definition

Key attribute applicability: Frontier research “(…) pursues questions

irrespective of established disciplinary boundaries. It may well involve multi-,

inter- or trans-disciplinary research that brings together researchers from

different disciplinary backgrounds, with different theoretical and conceptual

approaches, techniques, methodologies and instrumentation”, (EC, 2005)

Formalisation of the indicator:

Number of terms from different disciplines

Hypothesis: The more research issues concepts, techniques,

methodologies and instrumentation from different disciplines are

addressed the higher is the interdisciplinary character of the proposal

21 14.03.2013

Interdisciplinarity

22

Example: Panel PE1 / Call 2009 / Starting Grants

23

Proposals of panel PE1 sorted by

decreasing value of addressed

panels

Successful proposals are marked

All 11 successful within the first 12

of 39 proposals

Project ID Share of addressed panels

239885 95

239694 94

239807 94

240518 93

239748 92

239781 92

240123 92

239983 87

239784 85

240127 85

239959 82

239870 81

239902 76

240428 76

240269 73

240074 72

240223 71

240053 70

240471 70

240157 69

239800 68

239814 68

239952 67

240121 67

240693 67

240008 66

240192 66

PE1 Mathematical foundations: all areas of mathematics, pure and applied, plus mathematical foundations of

computer science, mathematical physics and statistics

Discussion of the Interdisciplinarity Indicator

Simple calculation of that indicator, direct and straight forward, theoretically conclusive

Data: all information like abstracts electronically available

Can be calculated for each new call, not sensitive on changing panel definitions or pre-classifications

24 14.03.2013

Indicator Novelty

Does the proposal use the latest research as it‘s intellectual base?

(Timeliness) Is the research agenda part of a research front in a

broader field? (Innovativeness: Similarity to new research issues)

25 14.03.2013

Proposal

Indicator Timeliness:

year of cited

references

Textual

information Research fronts:

PASCAL

Database

Indicator Innovativeness:

similarity of the proposal to

research fronts

Thank you for your attention