joint report on the final evaluation of socrates ii ... · separate programme-level reports on...

Joint Report on the Evaluation of the Socrates II, Leonardo da Vinci and eLearning programmes Executive Summary

ECOTEC

Joint Report on the Evaluation of the Socrates II, Leonardo da Vinci and eLearning programmes Executive Summary

January 2008

ECOTEC

u Priestley House 12-26 Albert Street Birmingham B4 7UD United Kingdom

T +44 (0)121 616 3600 F +44 (0)121 616 3699 www.ecotec.com

ECOTEC

Contents PAGE

1.0 Introduction............................................................................................... 1 1.1 Background ................................................................................................................1 1.2 Evaluation framework ................................................................................................2 1.3 Methodology ...............................................................................................................3

2.0 Impacts ...................................................................................................... 7 2.1 Impact in the schools sector .....................................................................................7 2.2 Impact in the VET sector..........................................................................................10 2.3 Impact in the higher education sector....................................................................13 2.4 Impact in the adult education sector ......................................................................16

3.0 Efficiency of the three programmes..................................................... 20 3.1 Efficiency of Socrates II ...........................................................................................20 3.2 Efficiency of Leonardo da Vinci II ...........................................................................20 3.3 Efficiency of eLearning ............................................................................................21 3.4 Efficiency of the Committee system.......................................................................21

4.0 The common impacts of the programmes and the contribution to the Lisbon goals..................................................................................... 23

4.1 Introduction ..............................................................................................................23 4.2 Creation of a European education area..................................................................24 4.3 Improvements in teaching practice and approaches to learning and

management .............................................................................................................25 4.4 Impacts on policy and practice at EU and Member State levels ..........................26 4.5 Increased proficiency in EU languages..................................................................26 4.6 Socio-economic impacts .........................................................................................26 4.7 The contribution of the programmes to Lisbon-related objectives .....................27

5.0 Strategic Conclusions ........................................................................... 29 5.1 The relevance of the programmes ..........................................................................29 5.2 The effectiveness of the programmes: objectives, impacts and sustainability..29 5.3 Balancing grass-roots needs and top-down direction..........................................30 5.4 Modes of intervention ..............................................................................................30 5.5 Decentralisation and the role of Member States ...................................................31

ECOTEC

5.6 Overall contribution to the Lisbon agenda ............................................................31

6.0 Recommendations ................................................................................. 32 6.1 Strategic recommendations ....................................................................................32 6.1.1 Building on the achievements of 2000-06...................................................................32 6.1.2 Achieving the right balance between grass-roots needs and top-down direction .......32 6.1.3 Realising the potential for synergy between actions...................................................33 6.1.4 Improving synergies within countries..........................................................................33 6.1.5 Giving greater priority to social disadvantage.............................................................33 6.2 Operational recommendations................................................................................34 6.2.1 Developing indicators and targets ..............................................................................34 6.2.2 Improving monitoring..................................................................................................34 6.2.3 Improving efficiency....................................................................................................34 6.2.4 Improving dissemination and sustainability ................................................................35 6.2.5 Structuring cooperation ..............................................................................................35 6.2.6 Improving how equality and accessibility issues are handled.....................................36

ECOTEC

1

1.0 Introduction

1.1 Background

In December 2006 ECOTEC Research & Consulting Ltd was commissioned to undertake the Joint Final Evaluation of the Socrates II, eLearning and Leonardo da Vinci II (LdV) programmes1 over the period 2000-2006. Collectively, these programmes accounted for in excess of 3,800 million euro of public spending2 and constituted a major way in which the European Commission had the potential to make a contribution to the achievement of the Lisbon goals in education and training3. This document is the Executive Summary for the Final Joint Evaluation Report.

The Evaluation Terms of Reference set out the following questions to be answered for all three programmes:

• Relevance: The extent to which an intervention’s objectives are pertinent to needs , problems and issues to be addressed

• Coherence and complementarity: The extent to which the intervention logic is non-contradictory and the intervention does not contradict other interventions with similar objectives

• Effectiveness: The extent to which objectives set are achieved • Efficiency: The extent to which the desired effects are achieved at a reasonable cost • Sustainability: The extent to which positive effects are likely to last after an intervention

has terminated. This was commissioned as a joint evaluation in order to be able to assess (a) individual and collective impacts, and in addition to that, (b) combined effects and (c) those combined effects in relation to Lisbon. This report is the main joint report of the evaluation. Separate programme-level reports on Socrates II, Leonardo da Vinci II and eLearning are available in annexed volumes with individual executive summaries. In order to avoid any potential conflict of interest stemming from the involvement of ECOTEC in the management of the Leonardo da Vinci II programme in the UK, the implementation of the

1 The legal basis for the evaluation was contained in the following Decisions of the European Parliament: Socrates – art. 14.2 Decision 253/200/EC; Leonardo – art. 13.2 Decision 1999/382/EC; eLearning – art. 12.2 Decision 2318/2003/EC. 2 Socrates: 1,850 million €,; LdV: 1,982 million €; eLearning: 44 million €. 3 See for example, Report from the Education Council to the European Council on The concrete future objectives of education and training systems. http://ec.europa.eu/education/policies/2010/doc/rep_fut_obj_en.pdf

ECOTEC

2

Leonardo evaluation was sub-contracted to ECORYS NL. A series of measures and quality checks were imposed to maintain this situation of independence.

1.2 Evaluation framework

The Terms of Reference for the evaluation required us to evaluate the combined effects of the programmes and also their combined effects in relation to the Lisbon goals. This required us to look both 'top-down' from policy and 'bottom-up' from the programmes.

The bottom-up component of this process was relatively conventional and comprised the reconstruction of intervention logics for each programme/action. The top-down element was more challenging as it required us to analyse how the Lisbon agenda had been translated into the field of education and training, and how it linked to the programmes; and then to construct an appropriate approach to evaluating the collective impact of the programmes in relation to Lisbon. This process concluded that it would be inappropriate to reconstruct an intervention logic in relation to Lisbon and that the evaluation should assess the collective contribution of the programmes to the Lisbon goals through the relevant body of education and training policy (notably the Copenhagen process in relation to VET and the Bologna process in HE).

Further, it was determined that the evaluation should carefully extract coherent sets of outputs, results and impacts that were common across the programmes, that did justice to the wide range of activities carried out under them and that would also reflect the general ways in which they might have contributed to Lisbon; the resulting categorisation would then be used to structure the evaluation. Outputs were categorised into three broad types: mobility activities; networks and partnerships; and methods/tools and frameworks. Impacts were of two broad types: practice and policy and socio-economic. The programmes were designed to deliver practice-related (and to a lesser degree policy-related) impacts more than socio-economic ones. It was not the prime purpose of most of the streams of funding to have a major direct socio-economic impact but rather to stimulate change within education and training systems, which might ultimately feed through into socio-economic benefits.

The evaluation also took into account the various structures and level of EU competence and influence within the various educational sectors that the programmes targeted (namely, schools, VET, higher education and adult education) when considering effects. For example in the highly developed higher education sector there is a major role identified regarding the Lisbon process not just in education but in providing the foundation for European science, technology and innovation. This sector has reached the earliest agreement of all the sectors at European level on common priorities (via the Bologna

ECOTEC

3

process), with comparatively strong progress towards the achievement of the infrastructure needed to form a genuine European area of higher education with easy cross-border movement for staff and students. In contrast, in the schools education sector where the EU’s competence is more limited and the principle of subsidiarity is strongly maintained with no equivalent of the Bologna and Copenhagen processes. European policy interventions in schools thus target comparatively small sums of money at individual institution and teacher levels. The adult learning sector too is relatively undeveloped, even embryonic, across many member states with varying participation rates and a very low base in terms of established cross-border co-operation and European dialogue in 2000. Similarly the VET sector, which is extremely varied across Member States in terms of sector structure and has seen some agreement on priorities through the Copenhagen process which is still in the early stages of implementation, with need for wider and more rapid progress recognised.

1.3 Methodology

The evaluation methodology was designed to address the issues at the heart of the evaluation framework given above, alongside the more ‘conventional’ evaluation questions as set out by DG Budget1. The broad evaluation questions were translated into a series of evaluation sub-questions and indicators (see Table 3.1, Chapter 3 of the Joint report).

The initial step was to reconstruct the logic for intervention for the programmes using documentation and interviews with Commission staff. The intervention logics were then translated into a series of meta-level impact indicators against which the programmes were evaluated.

This was followed by a programme of primary and secondary research. This included collation and analysis of programme documentation and financial data involving analysing key documents about each programme and Action such as: policy and background documents relating to establishing the programmes, calls for proposals, applications, the role of committees, assessment, selection, monitoring, project details, previous evaluations and dissemination. In particular, one purpose of the review was to review the management and implementation of the three programmes and explore their strengths, weaknesses, problems and solutions.

1 Evaluating EU Activities: A Practical Guide for the Commission Services; DG Budget, November 2003.

ECOTEC

4

We also undertook a series of web surveys targeted at successful applicant organisations under each of the three programmes1. The surveys were publicised by email to lists of successful applicant projects provided by the Commission. In addition National Agencies2 provided support in promoting the survey in their countries.

The surveys received the following response rates. In Socrates II, invitation emails were sent to 13,464 Socrates II participants from centralised actions. National Agencies were asked to forward the invitation to participants from decentralised actions. A total of 4,499 responses were received to the surveys, this represents an overall response rate of 33%. The survey focused on the main actions (Comenius, Erasmus, Grundtvig, Lingua, Minerva) and also covered the Arion sub-action and mobility in Grundtvig (Grundtvig 3) and Comenius (Comenius 2.2). In Leonardo over 46,500 email invitations were sent out, and 7,149 responses were received reflecting a response rate of approximately 15%. In eLearning emails were sent to all 62 funded projects and a total of 29 responses were received, representing a response rate of 47%. A further survey of participants in Action 3 (eTwinning) also received 1,551 responses from schools across Europe. Of those, the majority, 1,017, had completed a project through eTwinning.

We also conducted qualitative in-depth interviews with project co-ordinators, project partners, Erasmus co-ordinators, National Agencies, Commission and Executive Agency staff, Ministries, and other stakeholders3. In total we interviewed 128 individuals in Socrates II, 186 individuals for LdV and 86 individuals for eLearning. We developed a series of project case studies; these are presented in the individual programme reports. We also looked at the impact of the programmes at national level in five countries to aid understanding of how Member States contribute to delivering on Lisbon through the development and implementation of lifelong learning strategies and the reform of education and training systems. A sample of projects from the three programmes was also assessed (45 Socrates, 58 Leonardo and 11 eLearning projects) through an independent project scrutiny exercise using a template analysis framework primarily to enable us to assess the level of bias in survey responses, although it also generated other useful information which could be used in the evaluation. Finally, the National Reports prepared on Socrates and LdV were synthesised and the results of this are presented under separate cover.

The key findings presented below are based on the full range of data sources including programme and financial data, the responses from the surveys, and in-depth interviews.

1 In addition a separate survey was undertaken among schools participating in Action 3 of the eLearning programme, eTwinning. 2 And National Support Services in the case of the eLearning Action 3 survey for eTwinning. 3 Throughout this report 'stakeholders' are taken to refer to ‘people with a good knowledge of the education field and the three programmes. They included Commission staff, Executive Agency staff, members of Working Groups, experts and evaluators.

ECOTEC

5

ECOTEC

6

Selected Headline Outputs

Comenius

Over 74,000 schools in partnerships

Over 5 million pupils engaged

eTwinning

Nearly 24,000 schools involved

Over 310,000 pupils took part

Leonardo da Vinci II

21,000 projects

367,000 mobility placements for VET students

Erasmus1

Over 783,000 university students

Over 100,000 university teachers

Over 2,500 institutions held an Erasmus University Charter

Grundtvig

Over 1,900 adult education institutions participating in partnerships by 2006

1 Figures to the end of 2005

ECOTEC

7

Selected Headline Outputs

455 multilateral projects, networks and thematic seminars

2.0 Impacts

2.1 Impact in the schools sector

The key programmes potentially impacting on the schools sector were the Comenius action of Socrates II and eTwinning, Action 3 of the eLearning programme. A crucial factor underpinning all activity in the sector is that the Commission is not responsible for school education, this being a Member State responsibility. Schools policy during the period 2000-6 was an area where the EU had limited competence, primarily being there to support Member States to develop and modernise their own systems. This therefore limited the impact interventions were designed to achieve, particularly at the policy level.

Comenius aimed to involve a target of 10% of schools in activities supporting: enhancing the quality and reinforcing the European dimension of school education; promoting the learning of languages and promoting intercultural awareness through €599.5 million (27% of the overall Socrates budget) over 2000-06. These objectives were operationalised through three activities: school partnership projects (C1), training of school staff via transnational co-operation projects (C2.1) and individual teacher mobility (C2.2) and networks of teacher training institutions and other stakeholders (C3). In Comenius, over 74,000 schools participated in partnerships and over 5 million pupils were engaged1.

The intervention logic (IL) analysis for the Action highlighted that a key focus of decentralised actions was the co-operation process itself – the carrying out of a project with a number of partners from other European countries. In this case, tangible outputs such as tools/methods were seen as additional, 'induced' outputs. Centrally managed cooperation and networking actions in contrast were designed to develop new approaches and share good practice.

1 European Commission (2007) Comenius success Stories. Europe Creates Opportunities. DG Education and Culture, Belgium, p.4.

ECOTEC

8

In addition the IL analysis suggested Comenius partnerships probably faced barriers in transferring lessons learned to other practitioners and to national policy makers, given the relative inflexibility of national education policies and the EU’s limited competence in this area.

eTwinning was launched in 2005 as the main Action (Action 3) of the eLearning programme (2004-6), comprising 45% of the programme's €44 million budget. It focussed solely on activity in schools: i.e. twinning schools through the internet. eTwinning contributed to eLearning's global objective through encouraging the effective use of e-learning and to its specific objectives through strengthening and developing European level networking among schools, teachers and pupils, thereby enhancing the European dimension in school education. The scheme differed from the European Commission's other programmes/actions in that it did not give grants direct to participants. Rather, it provided free access to an ICT infrastructure, the eTwinning Portal (at central and national levels), which facilitated the establishment of partnerships between schools across Europe; schools could then choose what type of project they wanted to undertake. It thus constituted a wholly unique mode of intervention in the school sector. eTwinning involved nearly 24,000 schools across Europe, with an estimated 310,600 pupils taking part1. Our analysis of the IL indicated that eTwinning’s impacts were most likely to be qualitative rather than quantitative and when considering impacts, the relatively small size of the e-Learning budget should be used to contextualise impacts.

Relevance (To what extent were the intervention's objectives pertinent to needs, problems and issues to be addressed?)

Comenius and eTwinning were in line with the identified needs in terms of enhancing the European dimension and improving skills and competences in schools, particularly around ICT and languages. There was no formal overall rationale for intervention in the 'schools sector' through Community action – meaning that the various activities in schools were not conceived or designed to fit together – and their contribution to the Education 2010 work programmes (and Lisbon) was never fully considered, despite schools having the potential to contribute to both2. Programmes for schools were designed to impact from the bottom-upwards through impact on individual teachers/pupils/schools; not to bring about systemic changes. Regarding the Lisbon agenda, the programmes' objectives did not directly reflect this; and reducing early school leaving was an aspect of the Lisbon priorities which was not prioritised in the programme objectives and therefore did not translate into impact.

1 European Commission (2007) Comenius success Stories. Europe Creates Opportunities. DG Education and Culture, Belgium, p.4. 2 OJC 142, 14.6.2002, Detailed work programme on the follow-up of the objectives of education and training systems in Europe, p.1-22.

ECOTEC

9

Given the EU’s competence in schools, the scope for wider policy influence of the programmes was also limited to sharing of good practices and facilitating networking.

The programmes focussed on clear activities and had a clear sense of purpose (they were internally coherent), and activity between eTwinning and Comenius was clearly laid out to avoid duplication (they were complementary). The evaluators concluded that activity that would not otherwise have taken place without EU intervention had been funded (additionality).

Effectiveness (To what extent were the programmes successful in attaining the objectives set and achieving the intended results?)

Very broad objectives were set for programmes in the schools sector and as these were not translated into verifiable outputs/results and impacts this made the task of evaluation difficult. As a result, the evaluators imputed a set of impacts against which to evaluate the programmes to identify impacts in key areas. One target that was identified in the Socrates Decision was to attain a participation rate of around 10% of schools1. This was difficult to measure since a formal count of the number of schools in Europe was not available2, but qualitative feedback from stakeholders confirmed that the target was not met. The evaluation evidence demonstrated that eTwinning in particular appeared to have been very effective in achieving its objective of supporting European co-operation, by creating a network of interest and enthusiasm for co-operation among schools, which led to positive impacts, through the scale of activity supported through its innovative approach. Both programmes produced positive outputs such as tools, resources, DVDs and websites.

The key results and impacts in schools concerned: enhancing the schools’ European dimension and outlook, and for the individual: improvements in skills and experience of languages and ICT. An important area where the programmes impacted was in creating a clearer sense of European identity and through schools benefiting from co-operating with other schools. These areas clearly aligned with the needs and intended impacts the programmes were designed to address. Another important area of impact was on teacher training, for example teachers trained under Comenius’s decentralised mobility action reported having improved skills and competences to utilise in their teaching when they returned. Impact on curricula was more limited (not least because this was not a major intended effect) and tended to be localised; whole school impacts were comparatively rare (again, these were not intended). Policy impacts were also more limited; teachers could make small curriculum improvements locally, but as predicted, wider national changes to curricula were beyond the remit of the programmes and the EU’s competence. There were

1 Decision No. 253/2000/EC of the European Parliament and the Council of 24 January 2000 2 Not available in Eurostat Education Statistics, 2007.

ECOTEC

10

also some small scale impacts on language learning, with teachers reporting that pupils had improved language skills and were more open to language learning in future. Little impact was seen on less widely used/less taught languages as most projects were undertaken in one of the main EU languages. Overall though the programmes funded activity which led to impact in areas which met needs around skills levels and improving European co-operation and thus, in evaluation terms, were of good utility. Both Comenius and eTwinning provided good value for money with regard to the scale and extent of activity being funded (i.e. the small scale of many Comenius funding grants provided for teacher mobility and the fact that eTwinning schools did not receive direct grants at all). In both programmes, the added value of significant additional person-hours spent on projects by teachers has never been quantified. The innovative funding model of intervention used in eTwinning deserves recognition as an excellent value for money method of supporting transnational activity in the school sector. The evaluator’s opinion is that the value of both programmes could be further improved through more effective support for dissemination, sustainability and mainstreaming.

2.2 Impact in the VET sector

The evaluation looked at the impact of the programmes on the vocational education and training (VET) sector primarily through the impact of the Leonardo da Vinci II programme.

Leonardo da Vinci II was a key part of the Commission's intervention in the field of VET. The programme's objectives concerned improving the skills and competencies of people, especially young people, in initial VET at all levels, to facilitate their integration and reintegration into the labour market; improving the quality of, and access to, continuing VET; and promoting the contribution of vocational training to the processes of innovation, competitiveness and entrepreneurship. The key areas of activity (measures) were L1 mobility (including transnational placements and exchanges, and study visits); L2 transnational pilot projects including thematic actions; L3 projects concerning language competencies; L4 transnational networks; and support for reference materials, joint actions and accompanying measures. The total available budget for the programme increased from 2000 to 2006 from €171 million to €251 million, an increase of 47%. Of that, funding provided to mobility (L1) formed at least 39%; transnational pilot projects not less than 36% and funds for thematic action projects not more than 5 % (L2). In addition, language competence projects were not less than 5% of the annual budget (L3). During the course of the programme, the focus shifted from a balance between mobility and pilot projects to an emphasis on the former. As with other programmes indicators of impact were not specified at the outset so they were imputed from background documentation and interviews with stakeholders. In terms of outputs, placements of young people undergoing

ECOTEC

11

initial vocational training were key (L1). In the period 2000-2006 approximately 21,000 projects were established, the vast majority being mobility projects (19,307) and the total amount of mobility placements amounted to approximately 367,000. 70% of pilot projects (L2) focussed on creating training courses, particularly in ICT. A further 30% created vocational guidance products. The majority of language projects (L3) also produced training courses (57%), whereas transnational networks focussed on exchange of experience and dissemination activities.


In VET there was an identified need, and strong momentum, for example via the Copenhagen Declaration, for strengthening the European dimension, improving transparency, information and guidance systems, recognition of competences and qualifications and promoting quality assurance. It is clear that looking at the intended and actual outputs, results and impacts that these needs have been addressed in various ways, something that would not have happened without the Programme (in that sense the programme was promoting additional activity to what would have occurred anyway). There was also a clear alignment between the design of the Leonardo programme and the Lisbon guidelines through aspects such as improving employability, prioritising lifelong learning and developing the knowledge economy. Similarly, the Leonardo programme objectives did match the Integrated Guidelines1 during the period 2000-2006, with their focus on: comprehensive and coherent strategies for Lifelong Learning, helping people acquire and update the skills needed to cope with economic and social changes throughout the entire life cycle; improving the quality and efficiency of education and training systems; and equipping all individuals with the skills required for a modern workforce in a knowledge-based society, to permit their career development and to reduce skills mismatch and bottlenecks in the labour market. The budget increase awarded to Leonardo over the course of the programme indicates the increased political priority accorded to human capital development at EU-level.

The programme has proved to be coherent in terms of contributing towards the creation of a European education area through the promotion of lifelong learning and continued Community level cooperation between actors in the field of VET, and successive calls for proposals have reinforced this. Furthermore, half of the survey respondents (46%) said their project had helped to bring about convergence between Member States in policy and practice in the field of VET.

1 Commission of the European Communities (2005) Integrated Guidelines for Jobs and Growth (2005-2008) Brussels 12.4.2005, COM (2005) 141 final, 2005/0057 (CNS)

ECOTEC

12


Overall the evaluation found that the programme and its underlying measures contributed to the development of a European area of co-operation in the field of education and training and in this respect contributed to the promotion of a Europe of knowledge. The specific objectives of the programme were felt to be broad and flexible enough to address the needs in VET, taking into account local, regional and national diversity. The main benefit for the participating organisations was the development of a greater European outlook. In particular mobility and language projects contributed to this, whereas the other measures mainly produced networks among institutions from different European countries. The benefits for the participating VET students and the young workers consisted mainly of improved knowledge, skills and competencies and an improved quality of VET, especially through their participation in mobility projects.

The strongest area of impact for the programme was socio-economic, in relation to young people in VET. In particular, the projects contributed to improvements in: the knowledge, skills and competencies of young people in initial VET; capacities for lifelong acquisition of skills and competencies; and VET quality. In particular, the acquisition of foreign language skills was an important socio-economic benefit for young people participating in the programme. Strong socio-economic impacts were reported in relation to the employability and adaptability (to labour market developments) of participants in mobility, although most of these impacts can only be demonstrated in the long run when the participants have been active on the labour market.

The programme also impacted considerably on the curriculum in participating institutions, especially via activities in mobility and pilot projects. Leonardo substantially increased the quality of learning and teaching in the VET sector, for instance by improving and introducing new teaching methods and curricula. Also in many respects it contributed to the opening up of VET systems by organising transnational cooperation and placements, resulting in greater transparency of curricula and qualifications.

Additional to the organisational/institutional impacts, Leonardo also impacted on VET policy for example by developing standards, methods and tools that were integrated into national or regional policy and practice. Policy impacts appeared strongest at local and regional levels which was logical due to the limited geographical scope of most projects.

The impact of the Leonardo programme on increased transnational cooperation in the field of VET should not be underestimated, since without the programme little cooperation would have taken place. Overall, the programme contributed to the development of a European VET area which otherwise would not have developed at all or at best at a much

ECOTEC

13

slower pace. In addition, the most significant contribution of the programme towards the Lisbon goals was around delivering a better skilled labour force and in improving the labour market opportunities of young people.

In terms of sustainability, the programme was strong: the majority (73%) of survey respondents1 stated that (all or some of) the project activities had continued after the end of funding. In particular, the respondents of pilot projects indicated sustainability of project activities (84%) while 91% of respondents indicated that their project outputs were still in use in their own organisation and 64% in partner organisations. Furthermore, since the majority of the Leonardo projects has achieved their objectives, and since only a minority of the respondents stated that larger budgets would have contributed to higher quality outputs/results, and since the evidence pointed to a considerable impact on a large range of aspects of VET systems and its participants, the programme has probably produced value for money.

2.3 Impact in the higher education sector

Of the various actions of the Socrates II programme, Erasmus was the only one that was intended specifically for higher education at university and post-university level2. It supported the mobility of EU students and teaching staff within the EU, as well as co-operation between European universities in the areas of teaching and research. However, higher education institutions were also actively involved in other actions/programmes3, therefore when appropriate we draw on examples from those in the chapter.

Erasmus supported the mobility of EU students and teaching staff within the EU, as well as co-operation between European universities in the areas of teaching and research. Its objectives were to enhance the European dimension of higher education, encourage trans-national co-operation, promote mobility for students and staff and improve transparency and recognition of qualifications across Europe. These objectives were implemented via three main sub-actions: European Inter-university co-operation through multi-lateral projects (centralised), mobility opportunities for staff and students (which were managed in a decentralised manner by NAs) and the centralised set of thematic networks which involved HE and wider actors. The Erasmus budget for 2000-2006 amounted to 950

1 Of finished projects. 2 We took the view that the focus of Erasmus is on non-vocational education. 3 For instance, in eLearning, the virtual campus action aimed to add a virtual dimension to European cooperation in higher education by encouraging the development of new organisational models for European universities (virtual campuses) and for European exchange and sharing schemes (virtual mobility). This action line aimed at building on existing co-operation frameworks created by Erasmus, giving them an e-learning component.

ECOTEC

14

million euros (of which approximately 750 million euros was for student grants)1. Student mobility lay at the heart of the action, and during the implementation period there was further impetus to increase the numbers of participants and to integrate such mobility into a wider framework of co-operation activities that aimed at developing a "European Dimension" within the entire range of a university's academic programme.

According to EC figures2, Erasmus mobilised 783,104 students and 109,399 teachers; 3,192 students took part in Intensive Language Courses and 2,520 institutions held an Erasmus University Charter. The Action also implemented 479 curriculum projects, developed 1,271 intensive programmes and supported 166 networks.


The programme fitted well the identified needs in higher education and it stimulated co-operation in the higher education sector across Europe. Socrates, through the Erasmus action in particular, offered a wide range of measures designed to support the 'European dimension' of higher education, by encouraging co-operation between higher education institutions, promoting the mobility and exchange of their teaching staff and students, and improving transparency and academic recognition of studies and qualifications. Furthermore, by facilitating mobility, the programme potentially contributed to the achievement of the aims of the Bologna process and the Europe Education and Training 2010 Strategy and, through them, to Lisbon. There is an underlying assumption that given the number of people benefiting from Erasmus the programme will have had an impact on the skills and knowledge of people arriving on the labour market and their mobility and, consequently, a potential impact on the performance of the European economy in general and the objectives of the Lisbon strategy. Evidence collated in previous evaluations that focused on the professional value of Erasmus mobility indicated that a high percentage of Erasmus students saw their first job search as having benefited from participation, that there was far higher mobility in jobs and a stronger international dimension to the work found by Erasmus students as result of their participation.

The programme demonstrated strong internal and external fit (coherence). It had clear linkages between the global objectives (Socrates II) and the specific objectives (Erasmus) and the sub-actions supported. There was also a strong external fit with the Bologna process (most of the action lines were directly relevant to the action) and with Education and Training 2010 which emphasises the need to increase the quality and effectiveness of

1 Additional funds were provided in each country by public authorities, by the universities themselves and by other organisations. 2 Data only available for the 2000-2005 period

ECOTEC

15

education in Europe and to open up the systems to the wider world. The focus of the programme on the European dimension could be seen as complementary to much national policy since internationalisation of the higher education sector is prioritised in many Member States. Furthermore the evidence indicated that activity supported was additional and most would not have taken place without EU support. Where activities would have occurred without Socrates II funding, qualitative feedback from Erasmus university co-ordinators and project co-ordinators gave an insight into the important dimensions which additional EC funding had brought.

Overall it is important that higher education is not subject to a ‘common European policy’ because the authority over (and competence for) the content and the organisation of studies remains at national level. However the autonomy and flexibility of higher education institutions played an important role in facilitating the adoption of a European approach to tackling the common problems faced by the sector. That 90% of European universities participated in Erasmus1shows how instrumental it was in encouraging higher education institutions to take part in exchange programmes and exchanging information and best practice, showing the potential to influence the introduction of measures across the board.


Overall it was concluded that Erasmus activities fitted well with the objectives of the Action, which in turn fitted the perceived needs in the sector.

Increasing the capacity for mobility was the largest positive impact of Erasmus (89% agreed with the statement that their activity had achieved this). The short-term impacts on participants in terms of their personal and professional development were strong. Key impacts were an increased capacity for mobility in the future (within and outside Europe), a more open attitude and a clearer and better informed perspective for their subsequent studies or professional life; greater understanding of Europe and ‘belonging to a European family’, improved knowledge of EU language(s) and better contacts with European colleagues. A further area of significant impact was in the improvement of professional skills, and knowledge of other education systems and practices. As an ‘indirect’ effect of the mobility period, participants became more ‘employable’ (due to international experience and improved foreign languages skills). Erasmus's level of resourcing meant that it was able to support a large number of individuals. However while it had strong impacts on individuals, the 'real' effect on the entire student population was small at only 4%.

1 European Commission (2007), Erasmus Success Stories. DG Education and Culture, p. 3.

ECOTEC

16

Evidence of longer-term impacts was more difficult to identify. But there was some clear evidence of impact on policy and practice, particularly in terms of the response to the challenges of internationalisation in the sector and providing support to the inter-governmentally steered Bologna process. The Erasmus University Charter provided the general framework for all European cooperation activities undertaken by higher education institutions and helped to internationalise and to increase the European awareness and cooperation of institutions (thus potentially impacting on all students and not just those who took mobility periods). And overall Erasmus has acted as a driver for change in European higher education by providing support to the objectives set out in the Bologna process. The Bologna Process has led to a convergence in course structures, while existing programmes including Erasmus have sought to provide the tools (e.g. the European Credit Transfer System), stimulate the development of joint degrees and develop collaborative teaching approaches. Sustainability was also strong in Erasmus with 50% of projects stating that outputs from the centralised sub-actions were still in use in their own organisations. 72% of respondents also stated that all/most of the activities were continuing and/or would continue beyond the funding period.

Erasmus demonstrated good utility and good value for money. The majority of Erasmus funding went into the mobility aspect and data provided shows that the action has been clearly cost effective in this respect, given the high number of exchanges and despite the relatively low1 financial support. Outputs were appropriate and their generation has clearly benefited participant organisations and individuals.

2.4 Impact in the adult education sector

The Grundtvig action within the Socrates programme was the key policy instrument designed to target this sector. It aimed to raise demand for, and participation in lifelong learning activities, improve basic education competences, enhance information and support services and improve teacher training and recognition of competences. The funding of €130 million was modest compared to other programmes/Actions, representing some 7% of the entire Socrates II budget for 2000-2006. The objectives were implemented through four measures: G1 European Co-operation Projects and Grundtvig Training Courses (46% of the total budget), G2 Learning Partnerships: Exchange of experience/practice/methods (40% total budget), G3 Individual Training Grants for Adult Education Staff (6% total budget) and G4 Networks and Thematic Seminars (8% total budget). G1 and G4 were centralised, the others de-centralised (managed by National

1 The Erasmus grant, on average, covered the additional expense of studying abroad. As a recent survey on the economic background of Erasmus students confirmed, about half of the students who participated in the survey reported the Erasmus grant as being insufficient.

ECOTEC

17

Agencies). The evaluation took account of the less advanced nature of the adult education sector: more dispersed, less institutionalised, less well funded and with a shorter history of support from EU programmes. The action was not designed primarily to impact directly on systems for adult education, but rather emphasised the role that greater collaboration can play in building capacity and subsequently promoting changes in structures and systems. The main focus of the Grundtvig action was thus to promote the process of transnational co-operation and collaboration among relevant bodies within the sector. Few, if any, forums existed for the exploration of matters of common policy interest at European level. It was therefore recognised from the beginning that the application of this objective should focus on creating the capacity to explore such matters – particularly through the G4 networks.

Overall, in terms of participation, the number of adult education institutions participating in G2 Learning Partnerships rose year-on-year from 478 in 2001 to 1,980 in 20061 and a total of 455 multilateral projects, networks and thematic seminars (G1 and G4) were funded.2 The most common outputs were exchanges of experience and good practice (84%); partnerships or networks (54%); websites (50%); seminars, workshops, conferences and exhibitions (48%); transnational meetings (47%); and new teaching/training materials (41%)3. Mobility was also a significant output of the Action, including mobilisation of partners for meetings and workshops, many of whom experienced their first opportunity for mobility.


Overall, this evaluation supported the conclusions of the Interim Evaluation of Grundtvig4, which reported that it was seen as relevant to the needs of adult education in Europe, since its objectives were broadly defined and thus provided flexibility to meet institutional or national priorities, and for the Action to respond to shifts in needs over time. Nationally, given the rather broad sweep of the global objectives, there was very little need to tailor these at national level to achieve consistency with national priorities. While the Grundtvig Action was not designed primarily to have a direct impact on whole adult education systems as such, it did have the potential to contribute to major European policy agendas. Its contribution to the Lisbon agenda was mainly in terms of internationalising organisations in the sector, so that they are better positioned to work on common challenges, including those particularly germane to the Lisbon goals – equipping people

1 European Commission (2007): Grundtvig Success Stories, Europe creates opportunities. 2 ibid. 3 Based on survey of Grundtvig participants, base size = 627. 4ECOTEC (2004) The Interim Evaluation of the Grundtvig Action of SOCRATES programme.

ECOTEC

18

with the skills for a knowledge society and increasing access to lifelong learning opportunities, especially for those without a basic education. The strategic objectives of the Education and Training 2010 work programme also applied to the adult education sector, given their emphasis on increasing the quality and effectiveness of education and training systems, facilitating access for all and opening up education and training to the wider world, notably their emphasis on learning at all stages of life.

Coherence within the Grundtvig Action appears to have been strong, with an appropriate mix of measures selected to reflect the need for the sector to develop more European platforms for joint working, allied with individual mobility and professional development. It is also likely that much of the activity funded was additional, given the fragmented nature of the sector, and since the majority of Grundtvig project co-ordinators reported that their project would not have taken place without EU funding. Furthermore, lack of additionality was certainly not an issue raised by National Agencies or other stakeholders.


At programme level, Grundtvig's funded activities and results made a strong contribution to meeting the operational objectives of the Action – in particular in terms of supporting European co-operation projects, exchange of experience through transnational learning partnerships, facilitating the training of adult education staff abroad and, to a lesser extent, the development of European platforms in adult education and the exchange of information to shape policy and research. However our analysis of the intervention logic identified some weak linkages between operational objectives and the global objectives of the action; and we thus concluded that if a more focused set of specific objectives, taking into account intended impacts and recognising the specific contextual conditions in the adult education sector had been developed, this may have resulted in a more robust set of objectives, against which success could be measured. For example, the operationalisation of the specific objective to boost 'demand for participation in lifelong learning activities' was weak. This outcome might be expected to flow from the creation of a climate more conducive to achieving increased participation rates, but the Grundtvig activities in themselves were unlikely to have been able to contribute directly to the achievement of this.

The evidence suggests that the strongest emerging impacts in adult learning were in terms of increased and sustained co-operation between institutions/organisations, increased capacity for mobility and increased European outlook for individuals and institutions. Self-sustaining networks/ communities of interest in lifelong learning at EU level were

ECOTEC

19

established and have been successful1 although the extent to which these were sustained was more limited. There appears to have been a particularly significant impact on European outlook for professionals and students, particularly through emphasis on intercultural methods and issues. This was particularly important for first time overseas travellers, e.g. adult learners mobilised by these projects, as well as for participants from European Member States with relatively under-developed adult education sectors. While activity has resulted in transnational co-operation and has benefited a number of participating organisations and individuals, it cannot be said that this has been translated into a stronger "European dimension" to the adult education sector. Such an impact will require the sector and its main players to be able to exercise a much greater influence on policy development, an outcome which appears to remain elusive. The evidence also suggests a solid, if small-scale impact on the everyday lives and careers of professionals in the adult learning sector through enhancing their skills and improvements to the quality of curricula and teacher training. There were only limited impacts in terms of language learning; this was consistent with the Commission's decision not to set aside a specific funding pot for this objective within the Action. There was limited evidence of impact on transparency/recognition of qualifications and limited impact on national policy influence.

In terms of utility, the action could not be said to have contributed greatly to the identified needs in adult education, given the limits of its scale and scope. However it did lead to some strong shorter term impacts at the individual level which were of high utility. In terms of value for money, the budget allocated to Grundtvig was by no means extravagant. Given the under-developed state of the sector, and if the desired effect was to sow the seeds of a European dimension in adult learning, then the cost appears reasonable, however limited the evidence of long term impacts may be. G2 learning partnerships were particularly found to be cost-effective in terms of bringing new organisations into European co-operation. That said, the amount of funding allocated cannot therefore be said to be commensurate with the state of the sector, nor the scale of the challenges posed to the sector by the Lisbon strategy.

1 Indeed, this impact may have been greater than our evidence suggests since our sample did not include any action 4 projects 'Grundtvig networks'.

ECOTEC

20

3.0 Efficiency of the three programmes

Separate evaluations of the efficiency of the three programmes were carried out, as well as a common analysis of the committee system. The key evaluation question regarding efficiency was: To what extent have the desired effects been achieved at a reasonable cost?

3.1 Efficiency of Socrates II

Overall, the management and monitoring of Socrates II facilitated effective and efficient operation; in line with the identified needs; the programme’s objectives; and the expected outputs and results. About two thirds of the Socrates II respondents considered the overall application process to be 'good' or 'very good'. However there were some areas for improvement, which should be taken into account in the delivery of future programmes; e.g. co-ordinators expressed some dissatisfaction with the amount of administration involved in application and selection, and monitoring and reporting requirements, which they found to be complex and time-consuming. Difficulties were apparent in the availability and quality of monitoring information, specifically basic information such as application forms and interim and final reports. Evidence suggested there had been delays in payments to the projects – and the evaluators inferred that improved ICT tools would improve this aspect of the programme management.

3.2 Efficiency of Leonardo da Vinci II

The overall management and monitoring of the Leonardo programme generally facilitated effective and efficient operation. Improvements made to the programme over the years were generally welcomed; and the percentage of satisfied participants rose from 13% in the period of the first call for proposal (2000-2002) to 29% in the last call period (2005-2006). Changes in procedures A and B were generally welcomed by respondents, although some stated the division of labour between the National Agencies (NAs) and the Commission within procedure B may have been causing some disparity in the quality of accepted proposals. Most participants in Leonardo were satisfied with the application procedures. Only the quality and usability of the application form was received less positively. On a positive note, the delays in payment that were heavily criticised in the first phase of the Leonardo Programme (1995-1999) were largely resolved in the second phase (2000-2006) and generally budgets were felt to be adequate. At programme level both the NAs and the Commission indicated that the total available budget was not

ECOTEC

21

sufficient compared with demand, especially for mobility – this indicates rising demand for this type of activity. The main criticism of the programme management was the lack of a comprehensive IT system/tool to manage programme data although some NAs had developed management information systems, including one developed by the Norwegian NA which was also used by others, and ten NAs had – following the encouragement of the Commission – obtained ISO 9001. There was also some ex-ante and ex-post measurement of impact (though Rap4leo). It was the evaluator's view that having a well-structured and coordinated management information system would have improved the Commission’s understanding of the programme at EC and national levels.

3.3 Efficiency of eLearning

The management and monitoring of the all actions of the eLearning programme facilitated effective and efficient operation on the whole, and applicants in actions 1, 2 and 4 were generally satisfied with the efficiency of the management processes: 96% of eLearning project co-ordinators in our survey considered the overall application process to be adequate or better, with more than half (55%) rating it 'good' or 'very good'. In particular the qualitative evidence highlighted the proficiency of the Executive Agency (EACEA) in responding to all queries promptly by email and circulating email newsletters to all funded projects. Good practices like this could be extended to other programmes going forwards. The innovative model of intervention used in eTwinning was welcomed by schools as meeting a key need in the sector and feedback indicated schools found the tools very useful and well-administered. Schools made some small suggestions for improvement of the eTwinning services, including refinements to the partner search tool, which could be taken into account in the delivery of future programmes.

3.4 Efficiency of the Committee system

Committees represented a forum to allow Member States to participate in the political and strategic management of the three programmes and to comment on and approve arrangements for programme management as well as to ensure some flexibility in delivery, allowing them to adapt to changing needs and political contexts. Their key strength - as identified in interviews with Committee secretaries - was their consideration of 'points for information', since this allowed a more wide-ranging discussion (of issues not specified in the programme Decisions), enabling Member State representatives to gain a deeper understanding of programme implementation.

ECOTEC

22

Weaknesses respondents identified related mainly to the administration of the Committees; there were challenges associated with the time pressures involved – documentation was often distributed very close to the meeting date, as a result of tight selection timetables. This reduced the amount of time and attention members could pay. There was also no evidence of linkages between the three Committees, which the evaluators concluded could have been beneficial.

ECOTEC

23

4.0 The common impacts of the programmes and the contribution to the Lisbon goals

4.1 Introduction

To conclude the evaluation we returned to the evaluation framework presented in Chapter 2 to examine the programmes collectively and in relation to the Lisbon goals. Each of the common high level impacts was assessed.

Key Impacts of the Programmes

• Most significant impacts were on the creation of a European education area

• Strong impacts on the development of ‘professional social capital’ with significant effects on both individuals and organisations through mobility activities, networks, and partnerships

• Transnational cooperation firmly embedded in higher education where almost all institutions participated and a platform established in VET for the first time

• Sustainable cooperation mechanisms still patchy in the schools and adult sectors

• Improvements in teaching, learning and management formed the second most common group of impacts

• Widespread curriculum impacts: professionals brought back and applied their learning quite extensively within the curriculum, alongside the significant contributions made by 'strategic' interventions such as dissemination networks

• Impacts on management were also registered such that there was quite a deep and probably enduring impact within institutions in certain areas

• Impacts on policy and practice were not intended goals of the programmes and came mainly through the efforts of individual projects to disseminate their results

ECOTEC

24

• Most policy impacts were at local level but Leonardo and in particular Erasmus registered significant impacts at national and European levels

• There were widespread impacts on language learning not just through the development and dissemination of new methods but also through the act of project participation itself, although this tended to involve the more common languages, especially English

• The main socio-economic impacts took place through mobility, this being an especially prominent part of Erasmus and Leonardo. Erasmus was instrumental in institutionalising and firmly embedding mobility – and its benefits – in university life. In Leonardo mobility projects were the most successful and cost-effective of all measures within the programme

• Despite being a prominent horizontal theme, tackling social disadvantage was not a prominent feature of the programmes either in terms of activity or impact, though it was perhaps most prominent in Grundtvig and eLearning

4.2 Creation of a European education area

It was clear from multiple sources of evidence that in general the creation of a European education area was the single most important common goal for the programmes evaluated and that cooperation and increased European outlook were considered to be important impacts on both organisations and individuals across all programmes (85% to 93%). Furthermore, the added value of European cooperation was universally rated highly (79% to 94%). All programmes contributed to the development of what we have termed 'professional social capital' through mobility activities, co-operation amongst professionals, networks, partnerships.

Success in this area depends on developing a culture of European co-operation which is sustainable and where co-operation becomes commonplace rather than unusual; and on putting in place on-going mechanisms in the form of self-sustaining communities of interests through which co-operation can take place. Such an infrastructure has arguably been put in place through Erasmus, where almost all European universities participate, and in VET a platform of transnational cooperation has been established for the first time, but in other areas it is still patchy and this can be developed further.

ECOTEC

25

There is also an issue regarding the extent to which establishing a European area depends on a large volume or 'critical mass' of mobile students or co-operation activity. The most successful and best-resourced action – Erasmus – achieved a reach (or 'market penetration') of only 4% of the student body. However, large numbers of organisations were involved in the programmes and this therefore extended the programmes’ reach – and their indirect impacts – much further. In the case of the programme that was most successful in this respect – Erasmus – this meant a reach to almost all relevant institutions. Given the nature of the programmes, it would be wrong to equate ‘critical mass’ with volume. Rather, what becomes important is ensuring that individuals become 'change agents' who can take the benefits of European co-operation to others through effective dissemination and exploitation.

In relation to the more specific issues of convergence and transparency, the programmes have played the role of developing tools that can support and enhance ‘structural’ developments such as the development of the European Qualifications Framework (EQF), European Credits in VET (ECVET) and Europass. However, outside of HE and VET there are few structures to prevent divergence in future practice.

4.3 Improvements in teaching practice and approaches to learning and management

Improvements in teaching, learning and management formed the most common group of impacts after those associated with the European education area. Like the latter, they were seen as a positive impact by large proportions of participants and stakeholders across all the programmes (62% to 78%). Evidence demonstrated that professionals participating in the programmes brought back and applied their learning quite extensively within the curriculum, alongside the significant contributions made by 'strategic' interventions such as dissemination networks. Impacts on the curriculum, in the main, tended to be in parts of the curriculum rather than all of it. Nonetheless, we regard it as significant that, depending on the programme, some 8% to 21% of respondents indicated that the activities in which they had been involved had impacted in all areas of the curriculum, a significant achievement given the programmes’ design. Impacts on management were also registered to an extent that had not really been anticipated in the process of reconstructing the intervention logics for the programmes; this finding speaks of quite a deep and probably enduring impact within institutions in certain areas.

ECOTEC

26

4.4 Impacts on policy and practice at EU and Member State levels

The extent to which activities were intended to have impacts on policy and practice beyond the partners involved varied significantly depending on the programmes concerned, the nature of the activity and the EU's competence in each sector. The fact that some types of activity were decentralised whilst others were centralised is also highly pertinent here. Of these two types it is decentralised activities that were more likely to have had an effect on national policy and practice through dissemination. Impacts on policy and practice at national levels within countries came through the efforts of individual projects to disseminate their results; in the main national authorities did not play an extensive active role in gathering together and disseminating the results from projects. Thus with the exception of Erasmus, programmes/actions had the greatest impact on policy at local level. There is also evidence that Leonardo registered good levels of impact at European levels, which we regard as particularly impressive in light of the comparatively weak baseline of international cooperation in the VET sector in 2000.

4.5 Increased proficiency in EU languages

All the programmes sought to cover language learning in some way and impacts were widespread. Language proficiency developed not just through the development and dissemination of new methods but also through the act of project participation itself, although this naturally tended to involve the more common languages, especially English. Indeed, the continuing trend towards globalisation and EU enlargement both provided strong contextual factors that have almost certainly had more of an effect on language proficiency than the programmes. Lingua in particular impacted in this area. While being a small programme with a focus on devising new approaches to language learning, by no means all projects saw their role to increase the volume of language learning and teaching, but rather to improve the way in which language learning takes place. Activity in the schools sector through Comenius and eTwinning also demonstrates positive impacts on early language learning.

4.6 Socio-economic impacts

The main way in which programmes achieved impact in relation to society and the economy was through mobility. At the same time, an underlying theme within most of the programmes and their component parts was the tackling of socio-economic disadvantage. In relation to mobility the main contribution was through Erasmus and Leonardo, although such activity was also part of Comenius and Grundtvig. Erasmus was instrumental in

ECOTEC

27

institutionalising and firmly embedding mobility – and its benefits – in university life. In Leonardo mobility projects were the most successful and cost-effective of all measures within the programme, and there was a strong contribution to the improvement of skills and competencies of the people involved, which distinguishes it to a certain extent from Erasmus with its stronger emphasis on social skills development. Staff trained abroad under Comenius mobility reported good results in relation to the extent to which they could use their skills when they returned home, particularly for language teachers.

Tackling social disadvantage was not a prominent feature of the programmes either in terms of activity or impact. It was perhaps most prominent in Grundtvig, where some 42% of projects said it was an important objective, compared to 9-19% amongst the other Actions of Socrates II. eLearning also made a contribution here: 67-75% of respondents agreed that their project had achieved a group of effects linked to socio-economic disadvantage including: more individuals being able to use digital literacy skills in the knowledge society context; improved digital literacy of specific target groups; improved awareness of using e-learning to improve digital literacy. In Erasmus there was evidence that the small grants available and the concomitant need for self-financing meant that students from lower socio-economic backgrounds were less likely to participate than people from higher socio-economic backgrounds and in Leonardo NAs had to strike a balance between giving as many students as possible the opportunity to participate by distributing more, but smaller, grants or giving enhanced financial support to a smaller number of students.

4.7 The contribution of the programmes to Lisbon-related objectives

The programmes we have evaluated were not designed to achieve the Lisbon goals. Indeed, our analysis of policy found that the links from the programmes, through the body of education and training policy to Lisbon were complex, lacked clarity and were hard to unravel. That said, the greatest contribution of the programmes appears to have been in terms of 'opening up education and training systems to the wider world', following by 'increasing the quality and effectiveness of education and training systems'. Effects in terms of 'facilitating the access of all' were found to be less widespread, although clearly individual projects made important contributions in their own right. Unpacking this further, we can say that the programmes' strongest, most consistent and widespread impact was in terms of strengthening European co-operation and increasing mobility and exchanges, to which all the programmes and actions made a contribution.

Looking in more detail at quality and effectiveness, all the programmes contributed in this area, but as defined in Education and Training 2010 the objective of developing skills for

ECOTEC

28

the knowledge economy was not a strong feature of the programmes collectively, though Leonardo registered important impacts in this regard, and improving recruitment to scientific and technical occupations and making the best use of resources did not feature. Ensuring access to ICTs was especially important in eLearning and Minerva.

The contribution of the programmes/actions to quality and effectiveness varied between the programmes/Actions and the different modes of intervention. Whilst it was the task of networks and cooperation projects to develop and disseminate new methods, tools and approaches at a pan-European level, quality and effectiveness impacts were also registered in the decentralised actions but in highly localised contexts, and in general not across whole institutions. In terms of the extent to which the programme impacted on entire education and training systems both Erasmus and Leonardo were influential in respect of supporting developments which have and continue to influence the shape of systems in their respective systems. In other sectors, systems impacts were not intended. However, some countries used the opportunities presented by the programmes to bring about change (as in the case of Grundtvig to enhance under-developed adult learning systems) although these instances were rare.

ECOTEC

29

5.0 Strategic Conclusions

5.1 The relevance of the programmes

Overall it is clear that broadly speaking the programmes were relevant to the needs in the sectors to which they were applied. Erasmus continued to build on the success it had already established and has embedded a sustainable culture and platform for on-going cooperation. A key question for the new Lifelong Learning Programme is how to maintain the vitality of the Erasmus ‘product’ to ensure its future success: improvements in the quality of the mobility period, support provided and the recognition of the study period should assist in this regard. Leonardo has built capacity within the VET sector and helped to place the sector on a sounder footing with regard to transnational cooperation and the Copenhagen process and needs to continue to contribute to progress in the areas of quality, attractiveness and recognition and transparency. In the schools and adult learning sectors, the programmes were designed to stimulate cooperation and the development and sharing of new approaches and good practice as well as to meet highly localised needs. We consider this combination to have been appropriate to the task in this period and to have developed both capacity and capability within the sector. For these sectors, we believe the question is now how to develop activities so that they can make a stronger contribution to Lisbon – if this is desired – rather than delivering more of the same activity. To this end we recommend that firm priorities should be set for programmes in the schools sector in relation to the reduction of early school leaving, and in the adult education sector for raising participation in lifelong learning to assist in the achievement of the relevant EU benchmarks, both of which are behind target.

5.2 The effectiveness of the programmes: objectives, impacts and sustainability

The programmes' global objectives concerned both social and economic impacts and effects on practitioners, practice and to a lesser degree policy within the various educational and training systems; in budgetary terms there was roughly a two thirds/one third split in favour of the latter. This has been reflected in the pattern of their achieved impacts. Practitioners benefited from a stronger European outlook, being brought together in various co-operation activities and being provided with a means to learn new practices which they have carried back into their teaching practice. Impacts on management were also registered to a significant degree that had not been anticipated. Most impacts were small scale, localised and within parts of institutions. The programmes thus had an important impact on the development of professional social capital. Without the

ECOTEC

30

programmes much of the activity would not have taken place (additionality). Sustainability remained an issue in the schools and adult education sectors which have less developed capacity for cooperation: most projects will or have already fragmented and will not maintain all their partners and activities. It is therefore likely that a high percentage of the impacts will not just be localised but will also turn out to be short-term. The question that arises however is whether more could have been done to ensure greater, more lasting impact.

5.3 Balancing grass-roots needs and top-down direction

By design the programmes had very broad objectives, virtually no targets and a voluntaristic modus operandi. This meant the programmes were highly flexible, that learning ‘from the field’ could take place and that objectives could be fine-tuned to local and sectoral needs. At the same time, two factors made it difficult for managing authorities to balance this ‘bottom-up’ element with ‘top-down’ direction. First, although there was a process for setting priorities this was found to suffer from a number of deficiencies: it was not clear what status the priorities had alongside the objectives and the priorities could vary from the specific to the highly generalised (looking in some case like general objectives rather than priorities). Furthermore, the evidence points to the fact that in many cases in practice they did not have a major effect on the direction of the programmes. Secondly, there were deficiencies in management information systems such that it was very difficult for managing authorities to access robust, accurate and timely information about the outputs and results – in aggregate terms – that had been achieved at particular points in the programmes. Naturally this inhibits the ability to make effective decisions about the future direction of programmes.

5.4 Modes of intervention

There were two main modes of intervention: centrally managed transnational networks and cooperation projects; and decentralised individual mobility programmes and inter-institution partnerships. We conclude that this division of types of activity between different levels of management was appropriate.

Additionally, eTwinning adopted a unique mode of intervention whereby funding was directed to a system of Central and National Support Services and an IT infrastructure (via the eTwinning portal) instead of giving funding directly to applicants. In this mode, schools (the target audience) benefited from using the tools and resources to find partners to twin

ECOTEC

31

with, and undertake school-level projects. This was a mode of intervention that provided to be popular with the target audience and highly cost-effective.

5.5 Decentralisation and the role of Member States

Member States have a key role to play in modernising and reforming their systems in the Lisbon context. However, the evaluation in the main found little systematic evidence of Member State’s making use of the programmes strategically to adjust their policies and systems, although some countries – particular the new EU members – did so in some sectors. We consider this to have been a missed opportunity for most (though not all) Member States.

5.6 Overall contribution to the Lisbon agenda

The programmes' effects were of two types: on practitioners, practice and policy where their link to Lisbon was indirect and tenuous; and on individuals through the mobility actions where the contribution was more direct in terms of delivering both social and economic impacts. Any assessment must take into account the resources employed, which though significant were small in comparison to education and training budgets. One would not therefore anticipate major effects. The programmes served the function of building underpinning capacity and capability within education and training to support efforts to achieve the Lisbon goals. However, the evidence suggests that much of that capacity will wither and dissipate now that funding has ceased. Many effects were intangible and dispersed. In order to have a more significant effect on Lisbon such effects would need to be brought together and focused. Furthermore, most Member States were not actively engaged in the programmes to the extent of systematically seeking synergies with their policies. Greater consideration also needs to be given to the role that the programmes can play alongside and preferably in conjunction with other sources of funding, reflecting the move towards greater integration of actions that is being promulgated through the New Integrated Guidelines and associated National Reform Programmes.

Lisbon accords Member States a central role and thus it is important in the new generation of programmes that they seize the opportunities presented and also adopted a holistic viewpoint of their potential.

ECOTEC

32

6.0 Recommendations

6.1 Strategic recommendations

6.1.1 Building on the achievements of 2000-06

The programmes/actions were designed and made different contributions to the varied sectors at which they were directed which were appropriate to the diverse baseline conditions they faced. After seven years of operations, it is important that the Lifelong Learning Programme capitalises on the achievements made as much as possible.

In higher education, Erasmus needs to revitalise its mobility 'product': the Commission and partners involved in the Bologna process should take steps to improve the quality of the mobility period, the support provided, the range of partners involved from business and civil society and the recognition of the study period.

In vocational education and training, Leonardo da Vinci II has developed capacity and contributed to the establishment of transnational cooperation. The Copenhagen process has acknowledged the need for greater progress in the areas of quality, attractiveness and recognition and transparency and the Commission should ensure that Leonardo gives priority to these areas.

In the school and adult learning sectors the priority should be on developing measures that make a stronger contribution to Lisbon rather than delivering more of the same types of cooperative capacity building and exchange activities, valuable though these clearly were. The Commission should therefore give priority to measures that contribute to achieving relevant benchmarks: the reduction of early school leaving in the case of schools and in raising participation in lifelong learning.

6.1.2 Achieving the right balance between grass-roots needs and top-down direction

It is important that the right balance is struck between being responsive to grass-roots needs and ensuring there is adequate top-down direction. Priorities that are set during the course of the programme should be strengthened in order to have an effect on activities. The Commission should clarify the relationship between priorities and objectives, setting out the weight to be given to priorities, to ensure priority is given to proposals that reflect them, and to ensure that priorities are SMART (i.e. Specific, Measurable, Achievable, Realistic and Timebound).

ECOTEC

33

6.1.3 Realising the potential for synergy between actions

The evaluation found that few linkages or exchanges had been made between the actions and programmes considered, which in the case of Socrates meant that little added value appears to have flowed from bringing the diverse actions together in one programme. The integration of the actions in the new Lifelong Learning Programme is expected to facilitate better exchange across actions. However, the Commission and Member States should implement a number of measures including the facilitation of network meetings and the proactive exchange of information between EC staff/EACEA during the different stages of the life of a project (selection, implementation).

6.1.4 Improving synergies within countries

The evaluation found little evidence of Member States having an holistic approach to the programmes/actions or of making strategic use of them to adjust policies and systems. Given the central role of national governments in education and training policy it is important that Member States are more actively engaged with the Lifelong Learning Programme. The Commission should stress to Member States the importance of taking an holistic approach to decentralised actions and of forging links with national policy and strengthen the collection of monitoring data to inform Member State action.

Member States should take a systematic look at the role that the LLP can play in taking forward the EU agenda. They should also examine the potential for synergy between EU programmes and national policy and take stock of the linkages between the LLP and other EU programmes and funding streams such as ESF.

6.1.5 Giving greater priority to social disadvantage

Issues of social disadvantage had not been a prominent feature within most of the programmes and actions examined, despite being emphasised as an important horizontal issue. It would be beneficial therefore if greater emphasis were to be given to social disadvantage in annual priority setting by the Commission. Additionally people from lower socio-economic backgrounds tend to be under-represented within Erasmus mobility periods, probably on account of the fact that they are predominantly self-financed. The Commission should give consideration to ways of increasing grants for students from disadvantaged backgrounds, as is already happening in some Member States.

ECOTEC

34

6.2 Operational recommendations

6.2.1 Developing indicators and targets

Almost all of the programmes had very broad objectives which did not perform well when measured against the SMART yardstick (i.e. being Specific, Measurable, Achievable, Realistic and Timebound). They were also deficient in terms of indicators and targets. The Commission should ensure the development of coherent banks of indicators and targets and that the any priorities set are SMART (i.e. Specific, Measurable, Achievable, Realistic and Timebound).

6.2.2 Improving monitoring

It was very difficult for managing authorities to obtain robust, accurate and timely data on outputs and results. The Commission should seek to develop current management information systems so that they are able to collect systematically information on programme outputs and results. Beneficial steps would include ensuring applicants effectively identify at application stage their intended results and impacts as well as outputs and to describe the evaluation methods they intend to use; to use typologies like those used in the evaluation (e.g. mobility, methods, networks etc) to classify results and impacts; and to identify a sample cohort of projects to supply data on results and impacts achieved.

6.2.3 Improving efficiency

There are a number of recommendations directed towards the Commission and EACEA that would help to improve the efficiency of programme management:

• more and better use should be made of ICT to strengthen and improve the efficiency of all processes, for example through the use of common programme management and monitoring databases of projects and the wider use of online application forms.

• practice can vary between National Agencies, not least in relation to quality assurance. Good practice should be better shared, e.g. through workshops, audits, peer reviews, evaluations, and consideration should be given to following the example of Leonardo where NAs were encouraged to obtain ISO accreditation

• more comprehensive guidance and support from the Commission and EACEA would improve the mainstreaming of impact and sustainability of the programme achievements.

ECOTEC

35

6.2.4 Improving dissemination and sustainability

Much of the capacity and many of the capabilities that were developed are likely to dissipate now that the programmes have ended, and many of the effects were – as intended - not tangible, and were dispersed widely within the education and training infrastructures of Europe.

To improve the collection of good practice and dissemination the Commission and EACEA should require projects to allocate specific resources to dissemination and give consideration to the possibility of including a 'dissemination phase' similar to Action 3 within the ESF Equal Programme. They should also facilitate the sharing at EU level of good practice in dissemination and exploitation. National authorities should be required to adopt effective strategies to disseminate and exploit results and impacts within their countries which complement those of projects.

To improve sustainability, the Commission and EACEA should consider additional support measures to try to ensure the continuance and mainstreaming of activity. Consideration should be given to introducing measures such as the SALTOs in the YOUTH programme1. Support should also be provided to encourage participants to understand and plan for sustainability. All applicants should be required to have a sustainability plan built into their project design, which enables them to plan for their future more effectively

Additionally Member States need to take steps to ensure that European co-operation becomes embedded within curricula so that it becomes the norm rather than the exception. Additional funding sources should be sought either in other EU programmes where synergies might be made or in the form of public-private partnerships.

6.2.5 Structuring cooperation

Sharing and exchanging information amongst institutions across Europe was a central aspect of the programmes. However it was observed that in multiple cases co-operation was not developed in a clear framework with defined joint concrete objectives. To improve this situation it is recommended that the Commission ensures the provision of guidance and support, especially for schools and adult education institutions that do not have the same level of transnational experience as in other sectors.

1 SALTOs are resource centres that provide support such as training courses and contact-making activities to support organisations and National Agencies on various themes related to the programme. There are eight for YOUTH. Each SALTO is hosted by a National Agency with funding provided by the Commission.

ECOTEC

36

6.2.6 Improving how equality and accessibility issues are handled

The evaluation concluded that equality and accessibility was one of the areas where the programmes could be improved. A smaller than expected number of projects reported a focus on this transversal issue and some projects have a very limited understanding of inclusion issues. In the future participating institutions should demonstrate that they have considered how equal opportunities and empowerment will be integrated into all aspects of their activities. Steps to be taken forward by the Commission and EACEA could include: setting a comprehensive equality and empowerment policy that outlines the rights and responsibilities of all participants; promoting and tracking progress in equality and empowerment policy and practice; providing support, materials and processes that take the needs of target groups into consideration.

joint report on the final evaluation of socrates ii ... · separate programme-level reports on...

Documents