Evaluating Research Dynamics Using Evaluating Research Dynamics Using Network AnalysisNetwork Analysis

(in the context of EU funded R&D) (in the context of EU funded R&D)

Frank Cunningham DG Information Society and Media

European Commissionand

Caroline WagnerGeorge Washington University

CES/AEA Conference, Toronto October 27 2005

Shift Towards Evidence Based Policy Shift Towards Evidence Based Policy Making in EUMaking in EU

Commission has become more an ‘executive arm’ of the European Institutions

Manages a budget of €100 B /year

Rising (possibly) to €1000 B in period 2007-13

Shift towards ‘evidence-based’ policy development reinforced by enlargement to 25 Member States

EU R&D Framework ProgrammesEU R&D Framework Programmes

Currently in the 6th Framework Programme

Total budget of €17B over 4 years (2003-6)

Information Society Technologies (IST) €3.8B

Annual EU investment in IST of €1B

EU investment about 3% of total IST-RTD in Europe.

Current Evaluation and Impact Current Evaluation and Impact Assessment MethodologiesAssessment Methodologies

Commission has considerable experience in evaluation and monitoring of effectiveness of EU RTD investments over the last 15 years.

Focused mainly on ‘outputs’ rather than impacts.

Focused mainly on ‘project’ level and on the ‘private-return’ to RTD participants.

Now increased attention is given to results-based assessment and macro-economic and social impacts of EU RTD.

Gaining a better Understanding Gaining a better Understanding of Impact of EU RTD Fundingof Impact of EU RTD Funding

Work under way in DG INFSO since 2002

Established that EU funding has created an intensely linked network of European research collaborations.

New tools of ‘data mining’ and ‘link analysis’ were applied to EU IST databases.

Confirmed that network of collaborations is ‘small world’ with strong clustering

High degree of connectivity dominated by a small number of ‘hubs’

New Evaluation MethodologiesNew Evaluation Methodologies

Innovative firms do not innovate in isolation but in interaction with other ‘actors’

In ‘Innovation Systems’ the links between innovation actors are crucial

Collaborative R&D Programmes are one important way of strengthening linkages

Partnerships through collaboration networks helps to achieve ‘critical mass’ of dispersed EU R&D resources.

Series of Evaluative StudiesSeries of Evaluative Studies

DG INFSO is commissioning a series of evaluative studies to answer evaluative questions about the networking aspects and dynamics of:

IST RTD Innovation & Deployment

Part of our strategy for network evaluation and impact assessment

Series of Evaluative StudiesSeries of Evaluative Studies

1. ERAnets Study (RAND Europe) (subject of this presentation)

2. Evaluation of Progress towards ERA for IST (CESPRI) (subject of following presentation)

3. Networks of Innovation in IS Development and Deployment in Europe (tenders in evaluation process)

ERAnets Study (RAND Europe)ERAnets Study (RAND Europe)

Main Objectives were to Assess:

The degree to which IST Researchers in Europe collaborate.

How integration of IST Research in ERA changed with introduction of new FP6 Instruments.

Carried out from Jan 2004 to March 2005

Collaboration Has Grown SignificantlyCollaboration Has Grown SignificantlyEurope - 1990Europe - 1990

Collaboration Has Grown Significantly Collaboration Has Grown Significantly Europe - 2000Europe - 2000

ERAnets Study ERAnets Study Breaking New Ground in EvaluationBreaking New Ground in Evaluation

An Evaluative study supporting the Evaluation and Monitoring Unit of DG Information Society

Analysed the networks in IST Research and compared the current networks emerging in the Framework Programme with those emerging in other frameworks and communities

Providing insight into the structuring effects of FP6 on European IST RTD

Developed, and refined tools and datasets that can be used by the INFSO Evaluation and Monitoring Unit and others

FP6 Networks of Excellence ProduceFP6 Networks of Excellence Produce Highly Clustered Network Highly Clustered Network

NoEs, call 1 retained, projects as cliques, component 1, diameter 3, average path length 2.1 (colour is number of steps away from the centre, from 1-3)

ERAnets Study – Main Findings (1)ERAnets Study – Main Findings (1)

Main Findings show that IST RTD funding at the European level is most effective in:

Connecting universities and business

Connecting research in different themes and disciplines

Integrating new Member States

Integrating key patent-holders; and

Integrating small businesses

ERAnets Study – Main Findings (2)ERAnets Study – Main Findings (2)

Main Findings also show that:

FP6 network participants are more tightly interconnected than previous FPs

FP provides a major integrating function

…however

Large institutes and companies act as ‘gate-keepers’

SMEs have been ‘crowded out’

Next steps: 2005-6Next steps: 2005-6 Broadened RTD network analysis to the “ERA in the

Global context”: CESPRI (in 2005) Extending network analysis from RTD to include

innovation and deployment : to be contracted in 2006. Clarifying the “causality links” between IST-RTD

and FP and “Lisbon” objectives : Technopolis (10/04-8/05)

Initiating the systematic use of key indicators for monitoring the output and impact of IST-RTD at the Strategic Objective-level;

Identification and assessment of the capabilities of “system dynamic” model and tools for macro-economic/social impacts of investments in IST-RTD and ICT deployment : to be contracted;

Further InformationFurther Information

DG INFSO Evaluation Activitieshttp://europa.eu.int/comm/dgs/information_society/

evaluation/index_en.htm 

IST RTD (FP 6 Thematic Priority)http://www.cordis.lu/ist/ 

Communication: i2010 - A European Information Society for growth and Employment

http://europa.eu.int/information_society/eeurope/i2010/index_en.htm 

Further InformationFurther Information

Frank CunninghamDG INFSO Evaluation and Monitoring Unit

European Commission

Tel : + 32-2-296-8119Office BU 31 0/52

Ave de Beaulieu 311160 Bruxelles

Belgium

Email: frank.cunningham@cec.eu.int