BEYOND BIBLIOMETRIC

INDICATORS: A 360°

EVALUATION OF RESEARCH

IMPACT

W. Nicol Keith, Glasgow University

The professional development

framework to realise the

potential of researchers

Major new approach to researcher

development

Builds the UK research base

Develops world-class

researchers

Enhances the personal, professional

and career development of

researchers

We train & expect our

researchers to be well

rounded

Should we consider research

assessment & impact in the same way?

Why do we need research assessment?

Who do we assess?

Institute: a group of researchers with an articulated shared mission, operating within one or

more research programmes/units under the same management

Defining the context: What do we assess?

Examples of complex impact assessments in practice

How do we assess? Evidence for the REF

REF Examples of Impact & evidence

Examples of complex impact assessments in practice:

Training future leaders in field of cancer control

Fellowship Portfolio • The UICC has a range of Fellowship schemes & these can be viewed as

a portfolio.

• An impact evaluation of the UICC Fellowship portfolio was conducted The survey spanned years 1997- 2010 and over 600 Fellows reported on the perceived impact of the Fellowship on their research capacity, skills/knowledge & professional influence levels.

• Evaluation to date highlights the power of the portfolio as each has unique features which together deliver training relevant to gaps in skills and knowledge.

Who do we assess?

Background to the review

Summary of Grantees

– ICRETT

• Survey spans years 1997- 2010

• 1120 number of grantees contacted

• 447 grantees responded

• Map shows geographic distribution of respondents

YY

• Survey spans years 1998-2010

• 139 number of grantees contacted

• 95 grantees responded

• Map shows geographic distribution of respondents

ACSBI

• Survey spans years 1998-2010

• 103 number of grantees contacted

• 57 grantees responded

• Map shows geographic distribution of respondents

Overview of Fellowship Satisfaction & Performance

Success is measured across several areas (Research Capacity & Output; Skills & Knowledge; Professional Influence).

YY grants establish scientific projects and as such generate high numbers of joint publications.

ICRETTs are short term grants and for technology/skills transfer which lead to publication, increased knowledge or implementation of skills in for example a clinical setting.

ACSBI fellowships are advanced training grants which establish young investigators

Evidence of activity: Joint publication

Case study: YY Evaluation & Key Findings

• The objectives of the YY Fellowship are – To initiate, set up or pursue bilateral cancer research projects with collaborating investigators abroad

– To exchange and exploit complementary skills and material

– To receive training in advanced experimental research methods and techniques

• It is aimed at individuals with scientific or medical qualifications & a minimum of two years postdoctoral

experience who are actively engaged in cancer research

• Survey spans years 1998-

2010

• 139 number of grantees

contacted

• 95 grantees responded • number of invalid emails

unknown

• Map shows geographic

distribution of respondents

Professional Development

• Making an impact on Project Management & Leadership skills

• Focusing the future Research Direction

• Building the Research Programme

Increasing Research Capacity & Output

Building Skills & Knowledge

Increasing Professional Influence at Home & in the wider

International Community

Reasons for Success

• Reasons for Success – The survey indicates that YY Fellowships have a positive impact on all major key

performance indicators associated with a well rounded, successful international cancer researcher.

– In order to gain additional insight into the possible underlying factors driving success, the associations between variables was analysed using cross tabulation analysis. This analysis complements the basic survey output.

– Cross tabulation analysis is useful for showing a side by side comparison of survey questions to determine how they are interrelated. In statistical terms, it is a joint distribution between two (or more) variables.

• Summary of Findings – An Effective HOST/MENTOR Increases Fellowship Success – Publishing with the MENTOR increases chances of gaining Funding – The increased Knowledge & Skills resulting from the Fellowship help define future

Research Direction & Build the Research Programme – A successful Fellow has increased Project Management & Leadership skills – A successful Fellow is Internationally competitive – A supportive HOME environment is important to the implementation of training

An Effective MENTOR / GRANTEE Relationship Increases

Fellowship Success

Fellowship Success

• An effective Mentor/Grantee relationship

results in:

• Achieving Objectives

• An increase in Research Skills

• Increased Knowledge

• Publications

• This allows for:

• increased Funding

• Increased Professional Influence

• A positive impact on defining Research

Direction & Building a Research

Programme

• Together, these result in Improved

Management & Leadership Skills & an ability to

Compete at an International Level

A supportive HOME environment allows successful

implementation of training

For the benefits of

the Fellowship

experience to be

fully exploited, the

Home Supervisor &

Institution needs to

be fully supportive

of the returning

Grantee

Benefits to the Home Institute

Skills/tech transfer

Publications grantee/host supervisor: publications are an

internationally recognized indicator of success and

reputation. Fellows publications are contributing to the world

knowledge of cancer.

Returning fellows have an increased ability to attract

funding. The flow of funds into home institutes increases the

research capacity and knowledge generation.

The home benefits from the networks established by the

fellows as these extend influence internationally.

Impact Case Study: Leadership, influence &

capacity building

Testimony of Dr Rajesh Dikshit of ICRETT and ACSBI Fellowship opportunities: YouTube Fellowship impact

presentation at WCC

Impact Evaluation of the

American Cancer Society international

fellowships for beginning investigators

(ACSBI)

Aligning Sponsor strategy and training

opportunities

Geographic distribution of grantees

After 2007

Between 2002 -2007

Before 2002

ACSBI

• Survey spans years 1998-2010

• 103 number of grantees contacted

• 57 grantees responded

• Map shows geographic distribution of respondents

Area of work

Area of work & date

Job title

Job title & date

UICC Experience: Who or what do we assess?

• Impact assessment is context dependant – Low-middle income

countries

– Lab training / service development /implementation of technology

• Use it to fulfil current needs & identify future needs

• Complex research situations require more than impact factors, H-index & funding income

A few thoughts on evidence of influence &

impact……Social media & open access

• Some Advantages

– Stay current on new research

– Find resources

– Share & discuss ideas

– Develop local / international hubs

– Career advice

– Job opportunities

– Evidence of reputation, influence & impact

Researcher ID: visualising influence

• Geographic

locations for

publications

that have

cited our

work

• Geographic

locations of

co-authors

LinkedIn

• Endorsements for Skills

& Experience

• Recommendations

• Publications

Example of Science Networking Site: ResearchGate

• Assess impact of

your publications

– Downloads

– views

Example of Science

Networking Site:

ResearchGate

• Assess impact of

your institute

– Downloads

– Views

– collaborations

Example of Science Networking

Site: ResearchGate

In conclusion: Why, Who & What

Choose the right tools to measure impact

Look at the appropriate evidence