Long term impacts of the EU's Framework Programmes for research and technology development American Evaluation Association, 5 November 2011 Neville Reeve PhD, European Commission DG Research and Innovation Framework Programme (FP) is a multi-annual research funding programme implemented at EU level 7th Framework Programme ( ) - 50 billion 27 Member States more than 100 participating countries tens of thousands of transnational co-funded projects What is the FP? Successive FPs 1952:ECSC Treaty; first projects started :EURATOM Treaty; Joint Research Centre set up 1984:First Framework Programme ( ) 1990: Third Framework Programme ( ) 1994: Fourth Framework Programme ( ) 1998: Fifth Framework Programme ( ) 2002: Sixth Framework Programme ( ) 2006: Seventh Framework Programme ( ) FP Budgets + Ideas Frontier Research (Individual Grants) Capacities Research Capacity (Infrastructures) People Marie Curie Actions (Mobility) Cooperation Collaborative research (Multi-partner projects) JRC non-nuclear research Euratom direct actions JRC nuclear research Euratom indirect actions nuclear fusion and fission research FP7 ( ) - The Structure FP7 in figures First four years of FP7 245 concluded calls more than 77,000 proposals received 59,000 proposals evaluated (312,000 applicant organisations and individuals) 12,000 proposals retained for negotiations (69,000 participants) 169 countries involved requested EU funding of 20,4 billion Changing priorities Changing FP Objectives Early phase expansion from ICT and energy Second phase widening objectives Third phase European value added and social objectives Fourth phase European Research Area Fifth phase Frontier research, innovation Joint Programming ERANET+ ERANET JTI Artemis Eniac Clean Sky IMI FCH Art. 169 AAL Bonus EMRP JTI PPP PEOPLE INFRASTRUCTURES HEALTH NMP SPA SEC CSH ELSA SETPLAN ICT ENV ENE TRS EUROSTARS Fundamental Applied Innovation Deployment Development ICT-FET (Flag Ships) ERCSMEs and SME Associations EC Funds National and Regional Funds (external view of the current landscape) eHealth eIdentity ICT for TT Energy efficiency Programmes Instruments New thematic initiatives Wind Energy 6B Solar Energy 16B Bioenergy 9 B Carbon Capture & Storage 13B Electricity Grid 2 B Sustainable Nuclear Energy 7 B Smart Cities PPP Energy Efficient Buildings Future of Factories Green cars Future Internet Alzheimer Agriculture, Food Security & Climate change Health and Diet Cultural Heritage eHealth Smart grid TT, mobility & logistics Content Large Scale Demos & trials Ageing (More Years Better Lives) Climate Knowledge (Clik-EU) Seas and Oceans Antimicrobial resistance Urban Europe Water challenges KBBE EERP * NER300 * Eureka! Evaluation and Monitoring Roadmap Annual Monitoring FP6 Ex-post evaluation FP7 Progress report FP7 Interim evaluation Ex ante Impact Assessment FP Ex post evaluation FP Principles to describe activities Independence Coverage Evidence Coordination Impact Communication Independence Decentralised approach All evaluations carried out by independent external experts contractors selected through rigorous public procurement process - expertise, quality of tender, price experts selected according to expertise, with balancing issues such as geography, background, gender Use of non-European experts Wide consultation on Terms of Reference for FP evaluations Programme Committees Coverage (Nearly) all thematic areas covered by one or more evaluations during each FP Specific evaluations for Instruments (JTIs, PPPs, IPs, NoEs) Increasing variety and complexity of MS impact evaluations FP7 projects to advance techniques Evidence Combination of different evidence types quantitative and qualitative Indicators base in the annual monitoring reports Major advance provided by CORDA has supported new types of analysis including networking and bibliometrics 64 individual evaluation studies (DG RTD) in last 3 years FP7 Interim Approach and Sources EVIDENCE BASE Statistical data Reviews & evaluations ERC RSFF Marie Curie etc. Independent studies CONSULTATIONS Stakeholders Self-assessments Other sources NCP survey Papers submitted EXPERT GROUP Deliberation Successive drafts Own expertise HEARINGS Commission Agencies Researchers Stakeholders SUPPORTING EXPERTS Specific topics Evaluation study structuring who coordinates in FP6 Evaluation study structuring hubs and coordination in FP6 Coordination Commission network across the research family European RTD evaluation network around 35 members twice yearly meetings Evaluation manual Control Standards Impact of evaluations Clear link between FP6 ex post, FP7 Progress report, FP7 Interim evaluation and Horizon 2020 Similar evidence from earlier evaluations Council conclusions Recent debate on wider participation Communication FP level reports sent to Institutions with follow-on debates and opinions Dedicated website Conferences Member States (EUFORDIA, Half Time Highway) Commission (Building the future knowledge base) Participation in international academic meetings What have we learnt? Funds high-quality R&D: appraisal is tough; competition is fierce; participants perform better than non-participants in bibliometric terms; they include the scientific elite Growth of the FP has paralleled growth in high-quality international co-publication Attracts the more excellent researchers in their fields - engages the more research-intensive companies in theirs It is by design a pre-competitive, collaborative programme, primarily producing intermediate knowledge outputs as well as technical and market network relationships that are re-used in other R&D and business processes Participants who enter projects with a deliberate product or process innovation objective are more likely to obtain short-term results than others and this FP is a place to exploit existing strength generally too competitive to allow capacity building that has to be done with national resources Most participants have only a fleeting relationship with the FP. However, Strong core of established players and networks whose composition slowly shifts over time - most participants have only a short relationship with the FP - new participants appear to learn the value of networked R&D and increasingly to participate in open innovation But..we know very little of the details of how networks work, how networking relates to strategy or how network shape relates to success and more Despite trend to larger instruments in recent Framework Programmes, bigger networks do not seem to be more productive than smaller ones in fact, the evidence there is suggests the opposite FP often associated with pre-normalisation R&D and the development of technical standards Most participants believe that FP participation increases their competitivity but the indirect nature of the FPs effects (through intermediate knowledge outputs) makes them very hard to track Parts of the FP that focus on smaller firms and more direct results have been evaluated using a cash benefit-cost approach, which suggests high benefit-cost ratios. Smaller firms benefit less than larger ones (and are generally less satisfied with FP participation) But the long term view is missing. Evaluation system fits the policy cycle but misses longer term effects Complexity of the picture limits taking a longer term view But we now have more than 30 years of experience. What's it done for jobs and growth? Thank you for your attention Evaluation webpage:u/research/evaluations/index_en.cfmu/research/evaluations/index_en.cfm Building Blocks for the Future Strategy prior to HORIZON 2020 launch; coverage of all areas and components; detailed timetables Governance interaction with Member States and stakeholders; expert advice Coherence Cross-cutting studies on transversal issues (scientific quality, job creation, impacts, ); common templates and methodologies; common key indicators Evidence focus on throughput, output and impact; integration of reporting information - project reviews/ project outcomes; Transparency Evaluation strategy; website