Staying Ahead of The Curve:Rethinking Research Performance

Yukun Harsono

Vice President Product MarketingIREG-4

Astana, Republic of Kazakhstan

2

LEAN TIMES ARE HERE

3

THE NEW REALITY…

4

OTHER FORCES THAT SHAPE LEAN RESEARCH

Trend Exacerbated by Economic Downturn

GOVERNMENT POLICIES ON RESEARCH

Research Program Sponsor Description

Governments are actively guiding their national research agenda

Assess the quality of research in universities and colleges in the UK

Enable funding bodies to determine how to allocate grants across research projects

Detail by institution and by discipline those areas that are internationally competitive, together with emerging areas

Identify thematic domains for future European support

Part of EU’s strategy to become “the most dynamic competitive knowledge-based economy in the world”

................................................................................................................................................................................................................................................

.

5

................................................................................................................................................................................................................................................

.

6

SCOPUS POWERING PERFORMANCE ANALYSIS

7

CWTS found that, in general, Scopus covers much of the same material as the Web of Science (WoS) and ISI Proceedings and in some areas provides a higher level of coverage. Overall, the report found that Scopus should be considered as a potential alternative to the WoS for the purposes of the REF.

“The overall percentage of WoS-covered, science-related papers found in Scopus increased over the years, from 89 per cent in 1996 to 97 per cent in 2005. In other words, in these fields and for published articles and reviews the WoS now constitutes almost a complete subset of Scopus.”

CWTS found that, in general, Scopus covers much of the same material as the Web of Science (WoS) and ISI Proceedings and in some areas provides a higher level of coverage. Overall, the report found that Scopus should be considered as a potential alternative to the WoS for the purposes of the REF.

“The overall percentage of WoS-covered, science-related papers found in Scopus increased over the years, from 89 per cent in 1996 to 97 per cent in 2005. In other words, in these fields and for published articles and reviews the WoS now constitutes almost a complete subset of Scopus.”

"Science and technology play a crucial role in a country's economic growth," said Mr. Hiroyuki Tomizawa, Principal Administrator, Economic Analysis and Statistics Division, Directorate for Science, Technology and Industry, with the OECD. "We selected Scopus for its breadth of coverage including journal titles from over 100 nations as well as its advanced features. Together, these advantages will enable the OECD to execute more sophisticated statistical analyses to guide our member countries."

"Science and technology play a crucial role in a country's economic growth," said Mr. Hiroyuki Tomizawa, Principal Administrator, Economic Analysis and Statistics Division, Directorate for Science, Technology and Industry, with the OECD. "We selected Scopus for its breadth of coverage including journal titles from over 100 nations as well as its advanced features. Together, these advantages will enable the OECD to execute more sophisticated statistical analyses to guide our member countries."

Using Scopus in 2007 has resulted in our being able to track for 127 more institutions than in 2006 resulting in better showing for many universities. A better representation from non-English language journals has led to better results from many countries where English is not the first language

Using Scopus in 2007 has resulted in our being able to track for 127 more institutions than in 2006 resulting in better showing for many universities. A better representation from non-English language journals has led to better results from many countries where English is not the first language

RESEARCH ASSESSMENT EXERCISEHIGHER EDUCATION FUNDING COUNCIL FOR ENGLAND (HEFCE)

Using Scopus in 2007 has resulted in our being able to track for 127 more institutions than in 2006 resulting in better showing for many universities. A better representation from non-English language journals has led to better results from many countries where English is not the first language

Using Scopus in 2007 has resulted in our being able to track for 127 more institutions than in 2006 resulting in better showing for many universities. A better representation from non-English language journals has led to better results from many countries where English is not the first language

"Science and technology play a crucial role in a country's economic growth," said Mr. Hiroyuki Tomizawa, Principal Administrator, Economic Analysis and Statistics Division, Directorate for Science, Technology and Industry, with the OECD. "We selected Scopus for its breadth of coverage including journal titles from over 100 nations as well as its advanced features. Together, these advantages will enable the OECD to execute more sophisticated statistical analyses to guide our member countries."

"Science and technology play a crucial role in a country's economic growth," said Mr. Hiroyuki Tomizawa, Principal Administrator, Economic Analysis and Statistics Division, Directorate for Science, Technology and Industry, with the OECD. "We selected Scopus for its breadth of coverage including journal titles from over 100 nations as well as its advanced features. Together, these advantages will enable the OECD to execute more sophisticated statistical analyses to guide our member countries."

CWTS found that, in general, Scopus covers much of the same material as the Web of Science (WoS) and ISI Proceedings and in some areas provides a higher level of coverage. Overall, the report found that Scopus should be considered as a potential alternative to the WoS for the purposes of the REF.

“The overall percentage of WoS-covered, science-related papers found in Scopus increased over the years, from 89 per cent in 1996 to 97 per cent in 2005. In other words, in these fields and for published articles and reviews the WoS now constitutes almost a complete subset of Scopus.”

CWTS found that, in general, Scopus covers much of the same material as the Web of Science (WoS) and ISI Proceedings and in some areas provides a higher level of coverage. Overall, the report found that Scopus should be considered as a potential alternative to the WoS for the purposes of the REF.

“The overall percentage of WoS-covered, science-related papers found in Scopus increased over the years, from 89 per cent in 1996 to 97 per cent in 2005. In other words, in these fields and for published articles and reviews the WoS now constitutes almost a complete subset of Scopus.”

8

AUSTRALIAN RESEARCH FRAMEWORK

Publication Data for National Assessment

Elsevier provides citation information for the Excellence in Research for Australia (ERA) initiative Cluster One evaluation

Refine affiliation data: “In order for the ARC to process publication records for citation analysis, all publication outputs must be submitted with a Scopus unique article identifier. This allows the ARC to match the citation numbers for each publication outputs quickly and accurately post-submission.”

9

GLOBAL RESEARCH LANDSCAPE CHANGING

Countries ranked by published output in 2007

10

GLOBAL RESEARCH LANDSCAPE CHANGING

Countries ranked by output growth 1997-2007

Percent

11

RESEARCH RESPONDS TO GLOBALIZATION

Research now...

Crosses national boundaries Exhibits high mobility of resources, people,

ideas, technologies and infrastructure Overarches multi-jurisdictional regulation Responds and reacts to public and private

pressures Networks energetically in real and virtual

space/time Balances on the knife edge of Competition

and Collaboration

Source: Prof. Daryl Le Grew, University of Tasmania

12

RESEARCH MORE EXCITING BUT ALSO MORE CHALLENGING

Q: How many hours per week searching and gathering information?

13

Researchers spend more time looking for information than analyzing and applying it

Q: How many hours per week organizing, analyzing and applying information

Source: 2007 Survey by Outsell Inc. on 6,300 knowledge workers

A:

FUNDING PRESSURES

Q: What do you estimate is the approval rate for National Science Foundation grant applications by new researchers?

14

Source: American Academy of Arts and Sciences (www.scienceprogress.org)

Competition for funding is intense and will continue to intensify

Q: What do you estimate to be the average age when biomedical researchers receive their first grant from the National Institute of Health?

A:

15

Increased focus on EFFECTIVENESS & EFFICIENCY

IMPROVE PERFORMANCE across all workflows that make up the

research enterprise

16

RESEARCH EXECUTIVES: KEY CHALLENGES

Need constant flow of information and intelligence about internal performance, benchmarks and comparative performance

Require matrix of tools linking:- Reputation and Ranking- Growth and Development

with - Research Performance - Research Strategy

Implications “Have my strategic decisions

been effective?” “Are we capitalizing on new hot

areas e.g. stem-cells?” “Where should my strategic

focus be?” “Who are my true competitors

per competency?” “How is my competency

portfolio performing?”

Questions

Source: Prof. Daryl Le Grew, University of Tasmania

HOW WE MEASURE TODAY

Total number of papers Total number of citations Average number of citations per author Average number of papers per author Thomson Reuters’ Impact Factor Bergstrom’s Eigenfactor Egghe's g-index SCImago’s Journal Rank Hirsch's h-index

17

18

SAMPLE MEASURE: PUBLICATION OUTPUT

Drawbacks Bias against smaller institutions Quantity ≠ Quality

19

SAMPLE MEASURE: REFERENCE VS OUTPUT

Drawbacks Inter-disciplinary differences Not granular enough

20

NEW MEASURE FOR A NEW REALITY

New bottom-up approach needed for identifying scientific leadership

EXISTING APPROACH TO PERFORMANCE MEASUREMENT

ISSUES

Journal-based classification

Assign journals to categories

Science categorized into small number of fields and subfields (i.e. highly aggregated classification system)

Example: National Science Foundation groups science into 13 fields and 127 subfields

Number and scope of journals too limitingRisk: Too few journals included in analysis & potential for English-speaking bias in journal selection

Level of aggregation too high-levelRisk: Limited view of performance data on the level where it counts – that of school or large focused laboratory

Inter-disciplinary boundaries changingRisk: Not understanding how different disciplines interact with one another on the institutional level

Risk: Not seeing new emerging research areas

21

A NEW APPROACH

22

Partitioning the literature

This literature is partitioned into ~40,000 highly detailed subject areas.

Data

We start with the Scopus literature database, which indexes over 1.2 million articles per year, and covers a broad set of scientific areas.

Brain Research

Biotechnology

Earth Sciences

Math & PhysicsChemistry

Biology

Infectious DiseaseMedical SpecialtiesHealth Services

Social Sciences

Engineering

Computer Science & EE

Humanities

# of papers

23

Additional information

Strengths can be identified for: Universities Regions Nations

Details about strengths: Market shares Growth rates Top competitors Top researchers

Identifying areas of strength

24

DISTINCTIVE COMPETENCY #26

25

Math & PhysicsChemistry (1)

Earth Sciences

Biology

Biotechnology

Infectious Disease (1)

Medical Specialties (27)

Health Services (1)Brain Research (2)

Social Sciences (1)EngineeringComputer Science

Humanities

Description:

Clinical cancer researchMolecular endocrinologyMolecular neuroscience

DISTINCTIVE COMPETENCY #26

26

THE ADVANTAGES?

Bottom-Up Cross-Silos Granular

Identify “Emerging Areas”

Identify “Super Connectors”

Identify “Rising Stars”

1 2 3

28

IDENTIFYING NATIONAL STRENGTHS

Source: Klavans & Boyack, “US vulnerabilities in science and engineering,” 10th International Conference on Science and Technology Indicators, Vienna, Austria, Sept. 17-20, 2008

Strengths – Top 40 NationsStrengths – USA

US strengths (transparent) are overlaid here on the strengths (dark) from another 40 nations

Strengths – Overlaid

29

ELSEVIER?

We own 25% of the world’s scientific literature and we are usually perceived as simply a STM publisher…

But the reality is that...

30

We have been indexing and analyzing the world’s research output over the last few years, not just Elsevier-owned content

We have been spotting and building new advanced tools on top of our multi-publisher citation dataset to help universities take the guesswork out of research performance assessment

We solicited help from key partner institutions to co-define what performance intelligence needs to bring to light in order to help drive success for universities

SUMMARY

Efficiency and effectiveness are realized by taking a deeper look at the research activity workflows and identifying improvement opportunities

Progressive institutions will demonstrate lean research by adopting new performance intelligence tools to identify research strengths to focus on

With more rigorous analytics and better insights into research performance these institutions will be better positioned to gain standing among their peers

31

Economic downturn has intensified focus on research effectiveness and efficiency, bringing a new reality for us all…

Thank You For Listening