final evaluation report - cepal.org · pdf filethis report was prepared by elisabetta micaro,...

FINAL EVALUATION REPORT

November 2012

EVALUATION OF THE ECLAC - SIDA COOPERATION PROGRAMME 2010-2011:

Social protection and social inclusion in Latin America and the Caribbean

2012-1044

This report was prepared by Elisabetta Micaro, external consultant, who led this

evaluation. She worked under the guidance of Alejandro Torres Lépori, Chief of the Programme Planning and Evaluation Unit of the Programme Planning and Operations Division of ECLAC, and Irene Barquero, Associate Programme Officer in the same Unit, who also provided technical coordination and methodological and logistical support. This evaluation also benefited from assistance and inputs from Laetitia Montero, Associate Programme Officer of the Project Management Unit, and María Victoria Labra, Research Assistant in the Programme Planning and Evaluation Unit, both of the Programme Planning and Operations Division, ECLAC. The evaluation team wishes to thank programme managers of ECLAC for their contributions to this evaluation, in particular Sandra Huenchuan, Research Assistant of the Latin American and Caribbean Demographic Centre (CELADE) - Population Division of ECLAC, Simone Cecchini, Social Affairs Officer of the Commission‘s Social Development Division, Sonia Montaño, Chief of the Division for Gender Affairs, and Daniel Titelman, Chief of the Financing for Development Division. All comments and suggestions made by senior staff were duly reflected in the text of the report. Financial support for this study was provided by the Swedish International Development Cooperation Agency (SIDA). The views expressed in this report are those of the author and do not necessarily reflect the views of the organization.

Final Evaluation Report

i

CONTENTS EXECUTIVE SUMMARY .................................................................................................................. 1 A. INTRODUCTION ....................................................................................................................... 8 B. PURPOSE AND OBJECTIVES OF THE EVALUATION .................................................................. 8 C. METHODOLOGY ....................................................................................................................... 9

1. Limitations of the evaluation ..................................................................................................................... 10 D. BRIEF DESCRIPTION OF THE COOPERATION PROGRAMME 2010-2011 ............................... 12

1. The cooperation programme in Costa Rica ........................................................................................... 14 2. The cooperation programme in Ecuador ............................................................................................... 15 3. The cooperation programme in Paraguay ............................................................................................ 16

E. FINDINGS ................................................................................................................................ 18 1. Relevance ..................................................................................................................................................... 18 2. Effectiveness ................................................................................................................................................. 28 3. Sustainability ............................................................................................................................................... 41 4. Efficiency ...................................................................................................................................................... 44

F. CONCLUSIONS ........................................................................................................................ 47 G. RECOMMENDATIONS ............................................................................................................. 48

1. Recommendations for ECLAC ................................................................................................................... 48 2. Recommendation for SIDA and ECLAC ................................................................................................... 50 3. Recommendation for ECLAC and its donors .......................................................................................... 51

H. LESSONS LEARNED ................................................................................................................. 52 Annexes Annex I Findings ................................................................................................................................................. 56 Annex II Recommendations ............................................................................................................................... 58 Annex III Evaluation matrix ................................................................................................................................ 59 Annex IV List of persons interviewed ............................................................................................................... 65 Annex V List of recommendations arising from the evaluation of the cooperation programme

between ECLAC and the Swedish International Development Cooperation Agency (SIDA) 2007-2009 .......................................................................................................................................... 67

Annex VI Profile of persons surveyed .............................................................................................................. 71 Table Table 1 Consistency between the objectives of the cooperation programme 2010-2011 and the

draft programme of work of the ECLAC system, 2010-2011 ................................................ 24 Figures Figure 1 Classification of activities under the cooperation programme 2010-2011 .......................... 13 Figure 2 Survey respondent‘s level of satisfaction with the technical assistance received .................. 20 Figure 3 Relevance of the studies and seminars........................................................................................... 20 Figure 4 Usefulness of the seminars, workshops and studies according to the survey respondents . 30 Figure 5 Usefulness of technical assistance for respondents ...................................................................... 31 Figure 6 Usefulness of technical assistance for respondents‘ organizations ........................................... 32 Figure 7 Programme study usefulness ............................................................................................................ 36 Figure 8 Usefulness of technical assistance for mainstreaming gender and human rights issues ...... 38


ii

ABBREVIATIONS AND ACRONYMS ACABQ Advisory Committee on Administrative and Budgetary Questions

BOA United Nations Board of Auditors

CCSS Costa Rican Social Security Fund

CELADE Latin American and Caribbean Demographic Centre - Population Division of

ECLAC

CIDES Inter-American Committee on Social Development

CIPPEC Center for the Implementation of Public Policies Promoting Equity and Growth

CONAPAM National Council for Older Persons

ECLAC Economic Commission for Latin America and the Caribbean

FAO Food and Agriculture Organization of the United Nations

FLACSO Latin American Faculty of Social Sciences

IASPN Inter-American Social Protection Network

IDB Inter-American Development Bank

IEPS National Institute of Popular and Solidary Economics

ILO International Labour Organization

ILPES Latin American and Caribbean Institute for Economic and Social Planning

IMAS Joint Institute for Social Aid

INAMU National Institute of Women

INEGI National Institute of Statistics and Geography

JIU Joint Inspection Unit

MIES Ministry for Economic and Social Inclusion

OAS Organization of American States

OIOS Office of Internal Oversight Services

PPOD Programme Planning and Operations Division

SENPLADES National Secretariat of Planning and Development

SIDA Swedish International Development Cooperation Agency

UNICEF United Nations Children‘s Fund

UNRISD United Nations Research Institute for Social Development

UN-Women United Nations Entity for Gender Equality and the Empowerment of Women


1

EXECUTIVE SUMMARY

The ECLAC-SIDA cooperation programme, entitled Social Protection and Inclusion in Latin America and the Caribbean 2010-2011 was implemented by ECLAC between December 2009 and December 2011 and had a total budget of US$ 675,000. Its aim was to boost strategic cooperation between ECLAC and SIDA and thereby facilitate the analysis of social protection in Latin America and the Caribbean. This programme represents phase two of the programme implemented in 2007-2009. Its specific objectives were to foster policy debate on equality and social protection; to build capacity through dialogue and by sharing experiences among the countries in the region; to promote the use of best practices; and to assist and advise in policy formulation and reform. Within the framework of the four programme components (which are detailed in section D), ECLAC conducted studies and prepared technical material on issues relating to social protection (such as population ageing and pension systems), offered training to professional staff working at ministries and social security institutions, organized seminars and provided technical assistance to ministries and social security institutions across Latin America.1

The programme came to an end in December 2011 and the Executive Secretary of ECLAC commissioned a final evaluation report, which was completed with technical support from the Programme Planning and Operations Division (PPOD). The evaluation was designed to analyse the relevance, effectiveness, efficiency and sustainability of the results, and to provide recommendations and identify the lessons learned. The report fulfils the accountability requirement of SIDA, and also serves as a means for ECLAC to reflect on the programme‘s achievements and shortcomings.

It is important to note that owing to a change of priorities in the SIDA development cooperation policy, there are no plans to renew the cooperation agreement.

The programme was evaluated between September and November 2012. This consisted of the following stages: designing the evaluation; compiling and analysing the data; presenting the preliminary findings to ECLAC by videoconference; and preparing a first draft and the final evaluation report. The methodology used for the evaluation included interviews with key respondents, document analysis and an online survey. More detailed data on programme performance was compiled and analysed for three countries: Costa Rica, Ecuador and Paraguay. The evaluation was limited by, among other factors, the representativeness of the sample, the scant information gathered regarding the performance of the 2010-2011 cooperation programme at the regional level, the possible distortion of the survey results and the limited information available on the programme‘s performance in Paraguay.

1 The programme was implemented in 12 Latin American countries: Argentina, Brazil, Chile, Colombia, Costa

Rica, Ecuador, El Salvador, Mexico, Nicaragua, Panama, Paraguay and Uruguay.


2

1. Findings

Relevance: The programme was highly relevant to the countries in the region, taking into consideration their needs and priorities. The survey results, document analysis and the interviews all concur that the programme covered key subjects linked to human, social and economic development in the countries of Latin America and that it supplied important inputs (in the form of knowledge, tools and data) to guide decision-making towards the improvement of social protection systems. Two factors which made the programme particularly relevant to ECLAC were the fact that the issues addressed under the programme were closely aligned with the lines of work of the ECLAC divisions that implemented the programme; and, in addition, the flexibility of the programme meant that divisions were able to work in more countries than originally planned. Although the geographical priorities of Swedish cooperation activities have changed, social protection remains a relevant issue for SIDA both as a topic in its own right and as a means of working on its other priorities.

Effectiveness: The programme was effective in building capacity among the different stakeholders in the region who work in the area of social protection, providing inputs including knowledge, data and tools for policy analysis. The data gathered in Costa Rica, Ecuador and Paraguay made it possible to identify instances where the knowledge and tools related to the programme had informed the decision-making process. The programme also helped foster policy debate in the region on ageing, women‘s unpaid work and comprehensive social protection. Furthermore, although few examples were found of the implementation of best practices in relation to social protection, the promotion of those practices was one of the aspects most appreciated by the respondents. The evaluation concluded that gender considerations and the human rights-based approach had been incorporated to some extent. However, it is worth noting that the ECLAC-SIDA cooperation activities carried out in 2010-2011 placed a greater emphasis on gender considerations in the area of social protection than the previous phase of the programme by including a component on unpaid work carried out by women. Lastly, according to the evaluation respondents, the programme helped ECLAC to improve its position in the countries of Latin America and to develop the capacity of division staff.

Although many short- and long-term outcomes have been identified, measuring the programme‘s effectiveness was made more difficult by a number of factors related to the theory of change, logical frameworks, the information presented in the reports submitted to SIDA and the monitoring system.

Sustainability: The sustainability of the programme was assessed on two fronts: the sustainability of the impact of the outputs produced and services rendered in the countries; and sustainability within ECLAC of the topics covered in the programme. In both cases, the data collected indicate a high level of sustainability, although in the case of technical assistance, the divisions do not appear to have identified or considered how to use the experience and lessons learned through this activity in their future work.

Efficiency: Although ECLAC has ensured efficient programme execution by taking advantage of its existing expertise and relationships with governments in the region and, in general, by adhering to the schedule of planned activities, any collaboration between the divisions during programme implementation was on a limited and ad hoc basis. ECLAC also seems to have made little progress towards implementing the recommendations from the previous evaluation, owing, among other reasons, to a lack of response to the evaluation from management and the absence of a plan for implementing the recommendations.


3

2. Recommendations for ECLAC

Recommendation 1: ECLAC should continue to improve its programme monitoring system

The evaluation identified the monitoring system as an area of improvement for ECLAC. More specifically, it found that the outcomes, indicators and the baselines set out in the logical frameworks were not sufficient for monitoring and evaluating the cooperation programme. In recent years, ECLAC has taken several steps to improve in this area. Developing effective and efficient monitoring systems is a long-term project that requires not only the development of systems, but also a change in organizational practices and culture. It is recommended, therefore, that ECLAC continue its efforts to improve its monitoring system, stressing both the development of systems and the promotion of an organizational culture that values results-based management. Some of the recommended actions for ECLAC to ensure effective monitoring and evaluation of its projects and programmes (operations) are as follows:

Identify short-, medium- and long-term expected results that are specific, measurable, attainable and relevant;

Develop specific, measurable, achievable, relevant and time-bound indicators, that allow ECLAC to collect more meaningful data and that reflect the results, so that they may be used to guide decision-making;

Complement the indicators on document downloads with additional information about the main user profiles for ECLAC;

Ensure that reports systematically provide information on the indicators selected during the design phase and that those reports reflect any changes that are made to the indicators at a later stage along with an explanation of the reason for the change;

Extend the monitoring period beyond the end of the programme or initiative in order to increase the chances of finding short- and medium-term results.

Recommendation 2: For future programmes and projects, ECLAC should explicitly state the theory of change underpinning them

The evaluation highlighted the difficulty of analysing the long-term effectiveness of the programme because of the absence of an explicit theory of change stating the underlying foundations and assumptions linking the initiative to the desired effects. Short- and medium-term results were identified in terms of capacity-building and the promotion of policy debate. However, it was more difficult to establish to what extent and under what conditions such results represent progress towards the goal of improving the coverage and quality of social protection systems. It is therefore recommended that ECLAC develop a theory of change for its future programmes and projects that establishes more clearly the steps needed to move from one result to another, the conditions in which this can take place and the priority stakeholders targeted by each intervention. The theory of change should include specific short-, medium- and long-term results. A well-articulated theory of change would provide the Commission with elements to draw upon when deciding which projects and activities to carry out and where to invest its resources. This also requires some reflection by


4

ECLAC on its comparative advantage in the region, the areas in which it can contribute and how it can achieve this.

Recommendation 3: ECLAC should continue to strengthen coordination and collaboration between divisions

The evaluation demonstrated that cooperation and coordination between the divisions responsible for implementation of the cooperation programme 2010-2011 was generally on an ad hoc basis. ECLAC has been strengthening coordination and collaboration between the divisions in recent years, but further improvements are needed.

For coordinated and collaborative programme work, it is recommended that:

Programmes be designed jointly by the divisions responsible for their implementation and this be reflected both in the development of a common logical framework for the programme and in the preparation of the outputs to be produced together;

Systems be set up so information can be shared between divisions, within the programme framework, for example, regular meetings between the project coordinators;

Formal spaces be created for exchange, such as regional meetings, as part of programme implementation, so that lessons learned can be identified and disseminated among ECLAC divisions, as well as outside the Commission; and

Communication be facilitated among researchers and consultants from the countries targeted by ECLAC, in order to increase the opportunities for synergy and coordination between countries and reduce duplication.

Recommendation 4: The Executive Secretary of ECLAC, via the Project Management Unit, should ensure that all evaluations are accompanied by a response from management and an action plan

As previously mentioned, it was somewhat difficult to establish whether the recommendations from the previous evaluation had been implemented by ECLAC, and to what extent. It is thus recommended that the Executive Secretary of ECLAC:

Clarify the roles and responsibilities of the Project Management Unit and the Programme Planning and Evaluation Unit in terms of following up on the implementation of any recommendations arising from evaluations of completed programmes;

Ensure that all evaluations are followed by a response from management and an action plan. If any recommendations emanating from the evaluation of specific programmes are discussed within the broader context of ECLAC reform, the Project Management Unit, in collaboration with the Programme Planning and Evaluation Unit, should ensure that this information is reflected in the plan to follow up the recommendations.


5

Recommendation 5: ECLAC should continue its efforts to incorporate a human rights-based approach and gender considerations into its programmes and projects

A human rights-based approach and gender considerations were incorporated into the cooperation programme 2010-2011 to some extent. While some elements of these principles or perspectives (such as strengthening the capacities of duty bearers, access to information and transparency) were fully incorporated, other elements (such as strengthening the capacities of rights holders, participation and gender analysis) would have required greater attention from ECLAC.

More specifically, regarding the integration of the human rights-based approach, it is recommended that the Executive Secretary of ECLAC ensure that the divisions design, implement and monitor programmes and projects in accordance with the principles of capacity-building of duty bearers and rights holders, participation, access to information and transparency. Other United Nations agencies, for example, UNICEF and UN-Women, have developed guidelines for mainstreaming these principles, which the ECLAC divisions could use as a reference.

ECLAC is already taking action to mainstream the gender perspective. Therefore, it is recommended that the Executive Secretary, with the support of the Division for Gender Affairs, continue these efforts, ensuring that, among other things, a gender analysis is conducted for all ECLAC programmes and projects, that their logical frameworks include gender-specific indicators and that data disaggregated by gender is gathered, used and presented in the Commission‘s publications.

3. Recommendation for SIDA and ECLAC

Recommendation 6: SIDA and ECLAC should capitalize on their long experience of cooperation on social protection matters to disseminate the lessons learned in other Latin American and Caribbean countries, and in the case of SIDA, in other regions where it operates

Social protection is a multidimensional issue, encompassing several fundamental human rights in the economic and social sphere. Experience in international development cooperation has shown that the fulfilment of each right is linked to that of other human rights. Therefore, social protection lends itself to being used as a gateway to work on different aspects of social and economic development.

ECLAC and SIDA should capitalize on their long experience of cooperation on social protection to extract lessons learned and disseminate them, in the case of SIDA, to other actors in different regions and, in the case of ECLAC, to other actors in Latin America and the Caribbean where it works on the theme of social protection.

4. Recommendation for ECLAC and its donors

Recommendation 7: To ensure that the task is performed effectively, ECLAC donors should play a greater role in monitoring the programmes they fund, supporting ECLAC in developing logical frameworks and providing feedback on the progress reports and final reports


6

Responsibility for monitoring a programme lies with both the donor and the executing organization. Therefore, donors should support ECLAC to ensure that the programmes they fund are monitored effectively. The evaluation showed that the design of logical frameworks and the reporting of results are still areas for improvement for ECLAC. Donors should provide more support to the Commission in order to develop logical frameworks that allow the correct identification of the different levels of outcomes and indicators for each level. They should also make it a requirement for reports to contain data on short- and possibly medium-term results. We also suggest that donors make sure that reports deal with more than the activities and outcomes of funded programmes. Moreover, they should allow for a delay between implementation of a programme and presentation of the report, so that short- and medium-term results can be identified.

5. Lessons learned

Lesson 1: In order to identify significant results, the period of time should be sufficiently long to consider the cumulative effect of the interventions designed to impact public policy

The evaluation of the 2010-2011 cooperation programme identified several short-term results in terms of capacity-building and to a lesser extent, medium-term results in terms of policy changes. The evaluation also considered the importance of taking into account factors related to time and the type of relationship (contributory rather than attributive) between activities concerning research and public policy change to determine the programme‘s effectiveness. As is widely recognized in the literature, policy change is the result of a long process and multiple actions by different actors, as well as being influenced by contextual factors. Therefore, the results of research activities and the relationship between these and changes in public policy could be identified and evaluated within a period of time longer than one biennium.

Lesson 2: The effectiveness of research activities in promoting policy debate is greater if joint efforts are undertaken with other stakeholders

The evaluation found that programme activities have been most effective in terms of contributing to policy debates when conducted as part of joint efforts with other actors. Studies and other products presented in forums, conferences and other events organized jointly with other actors also trying to influence public policy will not only reach a wider audience, but also potentially important actors and, above all, decision-makers who happen to be present.

Lesson 3: The effectiveness and sustainability of research products are heightened when the issues raised are part of the regular work of the organization

The evaluation demonstrated the importance of working on issues that are part of the regular duties of ECLAC. This does not mean that ECLAC should not work on new or innovative topics, but this must be done within a well-defined framework, with clear expectations and different strategies. Working on issues that are part of the main line of work helps ensure that products and reflections can be reused in other contexts and in other projects and can serve as a basis for other activities.


7

Lesson 4: The implementation of social protection initiatives should reflect the multidimensional nature of the issue

Social protection is a multidimensional issue that can be addressed from different angles, as in the case of the 2010-2011 cooperation programme, this, however, failed to have a multidimensional nature in its implementation. Although the multidimensional nature of social protection is an advantage because it can act on different human rights (health, education, gender equality, food and others), for which respect has not been guaranteed, it can become a disadvantage because it favours situations in which human rights are treated separately, which prevents a holistic view of the political action necessary to remedy the situation of infringement. Therefore, social protection initiatives require a holistic view of the problem. This has to be reflected also in the design and implementation of interrelated activities that are consistent with the various aspects of social protection.

Lesson 5: Advocacy initiatives are more effective and sustainable when integrated in the activities of rights-holders and duty-bearers alike

The 2010-2011 cooperation programme focused on the influence of public policy, mainly through strengthening the capacity of duty bearers. As recognized by the same agencies of the United Nations, development cooperation interventions must build the capacities of duty-bearers and rights-holders so that the balance of power between these two groups can change in a sustainable way. Consequently, interventions related to advocacy work must recognize the respective roles of duty-bearers and rights-holders in the formulation, implementation, monitoring and evaluation of public policies and to evaluate the respective needs and develop strategies so that groups can participate, directly or indirectly, in the processes of change proposed by the interventions.


8

A. INTRODUCTION

1. This document contains the final evaluation report on the cooperation programme between ECLAC and SIDA 2010-2011, entitled Social Protection and Social Inclusion in Latin America and the Caribbean. The Executive Secretary of ECLAC, with technical assistance from PPOD, commissioned a final evaluation of the programme at its conclusion in December 2011. 2. The evaluation was conducted by Elisabetta Micaro, a consultant specializing in programme evaluation, with inputs from the Programme Planning and Evaluation Unit of PPOD. 3. The report is organized as follows:

A. Introduction B. Purpose and objectives of the evaluation C. Methodology D. Brief description of the cooperation programme 2010-2011 E. Findings F. Conclusions G. Recommendations H. Lessons learned

4. The lists of people interviewed and documents consulted and the evaluation matrix are included in the annexes.

This image is used throughout the report to draw attention to certain observations which are worth taking into account, but were not singled out as specific recommendations, either because a detailed exploration of certain issues was beyond the scope of the evaluation or because the implications of the particular findings were of a practical nature.

B. PURPOSE AND OBJECTIVES OF THE EVALUATION

5. This evaluation was conducted further to the accountability requirements established by SIDA and provided an opportunity for ECLAC to study the achievements and shortfalls of the 2010-2011 cooperation programme. It is worth mentioning that, due to a shift in the priorities of SIDA development cooperation policy, there are no plans to renew the agreement.

Specific objectives of the evaluation

Evaluate the expected and unexpected results of the programme. Evaluate the relevance of the programme for the countries in which it was implemented and for ECLAC. Evaluate the level of cooperation between ECLAC divisions during programme implementation. Evaluate the extent to which gender and human rights considerations were mainstreamed into the programme. Make recommendations and identify lessons learned

about the implementation of the programme.


9

6. The evaluation analysed the relevance, efficacy, efficiency and sustainability of the results of the 2010-2011 cooperation programme.

7. The report was prepared on the basis of responses to questions included in the evaluation matrix, which was developed in collaboration with the Programme Planning and Evaluation Unit of ECLAC, and interviews with the ECLAC divisions responsible for the programme‘s implementation. The evaluation matrix is attached as annex III.

C. METHODOLOGY

8. The methodology was based on questions formulated according to the terms of reference and mutual agreement between ECLAC and the specialized consultant. The report covers mainly qualitative —but also some quantitative— dimensions, and was prepared between September and November 2012. The work was carried out in stages: design, data collection and analysis, presentation of preliminary findings to ECLAC by videoconference, creation of an evaluation draft report and preparation of the final report. 9. Budget constraints precluded conducting field missions at ECLAC headquarters or in the countries in which the 2010-2011 cooperation programme was implemented. Instead, mini-case studies were conducted in Costa Rica, Ecuador and Paraguay, with a view to gaining a better appreciation of the programme‘s results. These countries were selected based on criteria and considerations agreed upon with the ECLAC Programme Planning and Evaluation Unit, namely: (i) to cover the programme‘s four components in order to provide a more comprehensive vision; (ii) to address two or more components in each country, in order to follow up on the implementation of various components in different contexts; (iii) to reflect the region‘s socioeconomic heterogeneity, including lower middle-income countries (Paraguay) and upper middle-income countries (Costa Rica and Ecuador); and (iv) to capture the heterogeneity of the region‘s social protection policies by covering countries with a more advanced social protection system (Costa Rica), countries whose social protection is at an intermediate level (Ecuador) and countries in which it is still weak (Paraguay), in order to evaluate the programme‘s adaptability and responsiveness to different challenges and needs. 10. The data collection methods used included:

(a) Interviews: Interviews were conducted mainly in the three countries selected with government officials and representatives from public institutions; with academics specializing in social protection; with some of the authors of the studies conducted in the framework of the programme, especially in the three selected countries; with ECLAC staff who participated in the programme; and with officials from peer organizations in Latin America and the Caribbean. Details on the interviewees are provided in annex IV.

(b) Document analysis: In order to achieve a more thorough overview of the activities undertaken by ECLAC, the Commission‘s reports to SIDA were analysed, as was the evaluation of the programme‘s previous phase. A content review of


10

the publications, workshops and seminars undertaken in the framework of the programme was also conducted. The national development plans of the selected countries were examined, too, in order to identify their respective priorities.

(c) Survey: A survey was distributed to ECLAC counterparts involved in the programme in order to triangulate the results arising from the interviews and the document analysis. The survey was designed and analysed by the specialized consultant and administered by ECLAC; it was successfully sent to 421 individuals and elicited a response rate of 24% (more details on survey respondent profiles are included in annex VI).

1. Limitations of the evaluation

11. A number of factors related to the budget and time available for the evaluation, the content of programme reports for SIDA, the availability of contact information for the participants in various seminars that ECLAC either organized or participated in, the non-existence of participant assessments for these seminars, and the recent change of government in Paraguay had a bearing on the evaluation and led to certain limitations:

(a) Sampling representativeness: Given the time frame and budget available for the evaluation, sampling of the selected countries for the case studies is not statistically representative; instead it reflects different extremes of development and progress in social protection. Although the selected countries are not directly representative of the programme, the sampling was undertaken with the goal of drawing attention to areas that may merit further research in other countries.

(b) Limits of information collected on the performance of the 2010-2011 cooperation programme at the regional level: ECLAC has two main sources for assessing the programme‘s results at the regional level: the progress reports and the final report presented to SIDA, and participant assessments relating to various seminars which ECLAC either organized or participated in. In the case of the first source, most of the reported results refer to products (publications, studies, training material) and services (technical assistance to governments). Several programme components have reports containing data on total publication downloads and on requests made to ECLAC for technical assistance by governments or other public institutions associated with social protection. But limited information is available concerning the impacts of these products and services on capacity-building or the use of products or acquired capacities for creating, implementing, monitoring or evaluating policies. In the case of the surveys, these were administered at the end of four different seminars; therefore, the information they provide is the participants‘ assessment of the seminars‘ usefulness and of the publications presented at these events. As a result, the data contained in the reports and the surveys sheds little light on the short- and medium-term results of the programme. The methodology for the present evaluation attempted to compensate for this shortcoming with a survey to compile data on the results obtained in the various countries where the programme was implemented. However, time and budget constraints prevented the triangulation of this information, to the detriment of its consistency.


11

(c) Possible distortion of survey results and of the independence of the evaluation: The survey was administered by ECLAC, which was responsible both for identifying counterparts and participants in programme activities and for disseminating the survey. The fact that ECLAC was responsible for both aspects introduces the possibility of distortion in the survey results. This possibility has been counteracted, in part, by the document analysis and third-party interviews. It should also be noted that the number of respondents identified by each division varied considerably, with a maximum of 309 and a minimum of 7. The number of responses received ranged from 64 to 6. The main implications of this are that the survey results do not represent a proportional assessment of the programme‘s different components, nor do they allow for a comparative analysis of the components (although this was not, in any case, the purpose of the survey).

(d) Limited information on the performance of the 2010-2011 cooperation programme in Paraguay: Only two respondents could be interviewed in Paraguay: one had participated in a seminar, the other was a government official who had received technical assistance from ECLAC on population ageing. Although efforts were made to track down officials who had worked with ECLAC or participated in seminars delivered by ECLAC, as well as to locate current officials, these were not successful: the individuals in question either no longer worked in the government (owing to the sudden change of administration in June 2012) or did not answer the e-mails sent to them. Information on the 2010-2011 cooperation programme‘s performance in Paraguay is therefore limited, especially in terms of efficacy and sustainability. Nonetheless, interviews with experts and representatives of peer organizations did yield information on the programme‘s relevance.


12

D. BRIEF DESCRIPTION OF THE COOPERATION PROGRAMME 2010-2011

12. The cooperation agreement between ECLAC and SIDA, signed in December 2009, and the project proposal dated 19 November 2009, are the key documents of the cooperation programme, 2010-2011, entitled Social Protection and Inclusion in Latin America and the Caribbean. The programme, implemented by ECLAC between December 2009 and December 2011 with a total budget of US$ 675,000, was aimed at strengthening the strategic partnership between ECLAC and SIDA with a view to facilitating the analysis of social protection in Latin America and the Caribbean by:

Fostering policy debate on equality and social protection;

Strengthening capacities through dialogue and exchange of experiences among countries of the region; and

Promoting best practices and providing assistance and advice on the formulation of policies and the implementation of reforms.

13. This programme represents phase two of the programme implemented in 2007-2009. Among the most important changes introduced in the second phase are the inclusion of ageing-related issues, social protection and the specific issue of unpaid work and the social protection of women. 14. The activities carried out included:

Studies on issues related to social protection (for example, the ageing population and pension systems);

Technical assistance to ministries and social security institutions;

Training professional staff working at ministries and social security institutions;

Organization of seminars to share experiences concerning the design and implementation of social protection systems;

Development of technical materials, including the creation of a database and learning modules.

2 The Financing for Development Division was established in 2011 under a new subprogramme on financing for

development, approved by the member States during the session of the General Assembly of the United Nations that same year. Previously, the Financing for Development Division had been part of a unit of another division of ECLAC.

The four components of the cooperation programme 2010-2011 Component 1: Financing for social protection towards universal coverage, executed by the Financing for Development Division.2 Component 2: Social assistance: poverty reduction and income redistribution co-responsibility transfer programmes, executed by the Social Development Division. Component 3: Social protection strategies for older persons, executed by the Latin American and Caribbean Demographic Centre (CELADE) - Population Division of ECLAC. Component 4: Unpaid work and social protection of women, executed by the Division for Gender Affairs.


13

15. A breakdown of the different types of activities carried out under the programme can be seen in figure 1. As can be seen, the majority of the activities carried out were studies, most of which were conducted in order to develop specific expertise on social protection issues. The second most common activity was the technical assistance offered to government ministries and institutions, followed by the organization of seminars, preparation of technical materials and conducting courses for civil servants.

Figure 1 CLASSIFICATION OF ACTIVITIES UNDER THE COOPERATION PROGRAMME 2010-2011

(Percentages) Source: Prepared by the authors.

16. The programme was implemented in 12 countries in Latin America: Argentina, Brazil, Chile, Colombia, Costa Rica, Ecuador, El Salvador, Mexico, Nicaragua, Panama, Paraguay and Uruguay. 17. It is important to note that this initiative is part of the efforts taken over several years by ECLAC, with the support of the Swedish cooperation to integrate the best practices of the Nordic model of social protection in Latin America and the Caribbean. These activities have been described in detail in past evaluations, the reports of which can be referred to in order to better understand the context in which the cooperation programme was implemented.3

3 J.C. Peña, R. Zivy and F. Amariles, ―Evaluación del Proyecto Inclusión del ‗Modelo nórdico‘ europeo en el

debate sobre la reforma de los sistemas de protección social en América Latina y El Caribe. Informe final‖, 15 February 2010; see also Vladislav Guerassev, Evaluation of the ECLAC-SIDA Technical Cooperation Programme 2007-2009: Enhancing economic and social conditions and opportunities of vulnerable groups in Latin America. Final Evaluation Report, December 2010.


14

1. The cooperation programme in Costa Rica

18. Costa Rica has one of the four lowest poverty rates in Latin America: while in the 1990s 30.5% of the population lived in poverty and 10% in extreme poverty, in 2008 those percentages had fallen to 20% and 6%, respectively.4 At the same time, inequality, measured by the Gini coefficient, has increased since the 1990s, to an average of 0.421 between 2000 and 2008.

19. The social protection system in Costa Rica is recognized as one of the best in Latin America in terms of coverage, access and quality of services.5 The system is universal in nature, with full health coverage through a combination of health insurance and free access to public health services6 and a social security system that is accessed by 86% of the population (data from the 2011 census).7 The legal and institutional social protection apparatus is robust and benefits the population in a fair and equal manner. The country, which has had social insurance since the 1940s, also has a network of institutions, specialized programs and public policies that guarantee social protection to the population. The Ministry of Labour and Social Security is the leading body responsible for social protection programmes, including the main national social protection programme, the Fund for Social Development and Family Allowances, created in 1974 for ―Costa Ricans with limited economic resources‖. Other institutions responsible for social protection issues are the Joint Institute for Social Aid (IMAS), created in 1971 to ―solve the problem of extreme poverty in the country‖, the Costa Rican Social Security Fund, which administers the health and pensions systems, the Housing Mortgage Bank, which ensures the financing of housing for low-income sectors and the middle class, and the National Council for Older Persons (CONAPAM), created in 1999 as the lead agency on ageing and old age. 20. In Costa Rica, two studies were conducted within the framework of the cooperation programme 2010-2011:

―Equidad y solvencia del sistema de salud de cara al envejecimiento: el caso de Costa Rica‖8, carried out by the Financing for Development Division in partnership with the Costa Rican Social Security Fund; and

―Protección social y trabajo no remunerado: redistribución de las responsabilidades y tareas del cuidado. Estudio de caso Costa Rica‖9, carried out by the Division for

4 Marco V. Sanchez and Pablo Sauma ―Choques externos y políticas de protección social en Costa Rica‖

[online] http://www.un.org/en/development/desa/policy/cdp/cdp_publications/choques_externos.pdf, 2010.

5 Carmelo Mesa-Lago, ―The extension of healthcare coverage and protection in relation with the labour market: problems and policies in Latin America‖, International Social Security Review, vol. 60, No. 1, January-March 2007.

6 International Labour Organization (ILO), ―Facts on Social Security‖ [online] http://www.ilo.org/public/english/ protection/secsoc/downloads/events/factsheet.pdf.

7 Patricia Leitón, ―Mayor cobertura en seguridad social pone en apuros a la caja‖, Periódico La Nación, 18 June 2012.

8 Adolfo Rodríguez and Jacqueline Castillo, ―Equidad y solvencia del sistema de salud de cara al envejecimiento: el caso de Costa Rica‖, Financiamiento del Desarrollo series, No. 239 (LC/L.3370), Santiago, Chile, Economic Commission for Latin America and the Caribbean (ECLAC).

9 Pablo Sauma, ―Protección social y trabajo no remunerado: redistribución de las responsabilidades y tareas del cuidado. Estudio de caso Costa Rica‖, Mujer y Desarrollo series, No. 116 (LC/L.3519), Santiago, Chile, Economic Commission for Latin America and the Caribbean (ECLAC).


15

Gender Affairs in conjunction with the National Women‘s Institute (INAMU) of Costa Rica, the Ministry of Social Welfare and Family and the National Network for Child Care and Development. The study was presented at the International Seminar ―Unpaid Work and Social Protection of Women‖, held in Costa Rica in February 2012. The event, organized by the Division for Gender Affairs of ECLAC and INAMU, was attended by representatives of the Ministry of Labour and Social Security of Costa Rica and representatives of the National Women‘s Institute (INMUJERES) of Uruguay.

21. In addition, the third Regional Intergovernmental Conference on Ageing in Latin America and the Caribbean was held in Costa Rica. While this conference was not part of the cooperation program 2010-2011, it was at this event that countries, including Costa Rica, presented their reports drafted using the methodology developed under the framework of the cooperation programme and contained in the document ―From commitment to action. Guide to the review and evaluation of the Brasilia Declaration in Latin America and the Caribbean‖.

2. The cooperation programme in Ecuador

22. While Ecuador‘s economy has grown steadily since the economic recession of 2000 and the poverty rate has declined compared to previous years, it remains at 32.8%.10 Moreover, inequality remains a problem in Ecuador, with a Gini coefficient of 0.493, much higher than that of Costa Rica. 23. Considered by the World Bank as an upper middle income country, Ecuador has an intermediate level system of social protection and relatively strong institutional apparatus. In recent years there have been several advances in the field of social protection, including the expansion of social protection systems. For example, the health and pension systems now cover 40% of the population, a substantial increase from five years ago, when that figure stood at only 16%.11 24. However, according to the ILO,12 the informal sector accounted for 57.6% of non-agricultural employment in 2004 and a larger proportion of female workers were employed in the informal market than male workers (63.9% and 52.9%, respectively). 25. Ecuador is one of the Latin American countries that is revamping its approach to social spending and implementing a specific strategy to improve social protection and eradicate poverty.13 The changes in strategy can be seen in the national development plan, known as the National Plan for Good Living 2009-2013, which divides the responsibilities for social protection issues among different government bodies. the National Secretariat for Planning and Development organizes the government‘s objectives; the Ministry of Economic and Social Inclusion develops programmes and policies in line with the development objectives, such as the Social Protection Programme; the Coordinating Ministry of Social Development

10 See World Bank [online] http://datos.bancomundial.org/pais/ecuador. 11 See Coordinating Ministy for Social Development, Social Agenda 2012-2013. 12 Fabio Durán Valverde, Diagnóstico del sistema de seguridad social del Ecuador, International Labour

Organization (ILO), June 2008 [online] www.ilo.org/gimi/gess/RessFileDownload.do?ressourceId=6093 13 Ibid.

http://www.ilo.org/gimi/gess/RessFileDownload.do?ressourceId=6093


16

coordinates and oversees the implementation of social policies and coordinates activities with other government agencies; and lastly, the National Institute of Popular and Solidary Economics (IEPS). 26. In Ecuador, the cooperation program has involved:

Technical assistance provided by the Financing for Development Division of ECLAC to the Coordinating Ministry for Social Development on the subject of health and pensions;

Technical assistance provided by the Social Development Division of ECLAC to the Ministry of Economic and Social Inclusion on conditional cash transfer programmes;

Participation of the Social Development Division and the Division for Gender Affairs of ECLAC in the International Seminar ―Comprehensive Social Protection Systems: experiences and perspectives‖, organized by the Ministry of Economic and Social Inclusion. During the seminar, the ECLAC representatives gave the following presentations: ―comprehensive social protection systems: experiences and perspectives‖; ―Ecuador in the context of Latin America and the Caribbean from the perspective of the Gender Equality Observatory for Latin America and the Caribbean‖; ―Inclusive social protection: a comprehensive look, a rights perspective; and ―Social protection and non-contributory programmes: family, life cycle and social protection‖;

The case study conducted by the Division for Gender Affairs ―Social protection and unpaid work: redistribution of responsibilities and care tasks. Ecuador: a case study‖,14 whose main counterpart was the Transition Committee of the Council of Women and Gender Equality.

3. The cooperation programme in Paraguay

27. Paraguay has seen little human development, slow economic growth and limited institutional development. This is reflected in the current situation of the country, where 34.7%15 of the population lives in poverty, inequality reached a Gini coefficient of 0.524,16 and political changes have affected government institutions. 28. The social protection system in Paraguay is at the development stage and is characterized by low coverage and few non-contributory financing programmes. Social security consists of the pension system and the National Health System, which are oriented towards the formal economy. In 2008, only 13%17 of employed persons were making social security contributions, the lowest percentage in the region. Furthermore, Paraguay has the second lowest direct tax revenues in the region. The large size of the informal sector means that 8 in 10 citizens have no protection for old age and 7 in 10 workers have no health

14 Alison Vásconez, ―Protección social y trabajo no remunerado: Redistribución de las responsabilidades y

tareas del cuidado. Estudio de caso Ecuador‖, Mujer y Desarrollo series, No. 115 (LC/L.3518), Santiago, Chile, Economic Commission for Latin America and the Caribbean (ECLAC).

15 Government of the Republic of Paraguay, Plan Estratégico Económico y Social – PEES- 2008-2013. Propuestas para un crecimiento económico con inclusión social en Paraguay, Asunción, National Economic Team, 2008.

16 See [online] http://datos.bancomundial.org/indicador/SI.POV.GINI. 17 Diana Ardila-Fajardo and others, ―Políticas públicas para la igualdad: Paraguay en el contexto

latinoamericano‖, Economic Commission for Latin America and the Caribbean (ECLAC), May 2011.


17

insurance.18 Currently, Paraguay is facing the challenge to create, expand and implement a social protection system that ensures that these services are provided on an uninterrupted basis. 29. Government institutions responsible for social protection in the current system are the Social Security Institute, which administers the social insurance in the country, the Department of Social Welfare and the Ministry of Public Health and Social Welfare. 30. In Paraguay, a study of public policy pertaining to component 2, ―Public policy for equality: Paraguay in the Latin American context‖, was prepared for the government, and technical assistance was provided by the CELADE - Population Division of ECLAC to help the Ministry of Health and Social Welfare to develop a national action plan for older persons.

18 See [online] http://www.socialsecurityextension.org/gimi/gess/ShowCountryProfile.do?cid=395.


18

E. FINDINGS

1. Relevance

31. According to the OECD, the relevance of a programme is defined as the extent to which the objectives of a development intervention are consistent with beneficiaries‘ requirements, country needs, global priorities and partners‘ and donors‘ policies. Retrospectively, the question of relevance often becomes a question as to whether the objectives of an intervention or its design are still appropriate given changed circumstances.19

32. The relevance of the cooperation programme 2010-2011 was evaluated in terms of the interests of current and potential beneficiaries or users of the social protection systems in the countries selected, the governments of the countries selected, ECLAC and SIDA.

Finding 1: The data collected suggest that the cooperation programme was relevant or very relevant in the countries where it was implemented as it was consistent with national needs and priorities and the comparative advantage of ECLAC in the region.

33. The data collected show that the programme was relevant or very relevant in the countries where it was implemented.

34. Its relevance can be attributed to the following factors:

Consistency between the objectives of each component implemented and the long-term needs of the population targeted by the programme interventions. While several social protection programmes have been carried out in the different countries of the region with positive effects in terms of coverage of the population in need, the data available on poverty in Latin America show that it still affects approximately one third of the population and that 13.3% of Latin Americans live in extreme poverty or

19 Organisation for Economic Cooperation and Development (OECD), Glossary of Key Terms in Evaluation and

Results Based Management, Paris.

Programme objectives:

Component 1: Strengthen the capacity of governments in the region to formulate and implement policies and financial mechanisms in order to improve social protection services, including pensions and health services, in terms of expanding and deepening both the coverage and quality of services.

Component 2: Improve the capacity of governments with respect to analysis and policymaking through political debate and technical assistance, in relation to the policies on poverty reduction and fostering social cohesion, with a particular focus on conditional cash transfer programmes in the context of the present international financial crisis.

Component 3: Better integrate the interests and needs of older adults into all aspects of development and policymaking in the countries of the region, in accordance with the recommendations of the Madrid International Plan of Action on Ageing and its main regional adaptations, from the perspective of inclusion and social protection.

Component 4: Improve the national capacities of governments to

formulate policies that promote gender equality through the documentation and dissemination of information and in-depth analysis and the provision of technical assistance with respect to the effects of the international economic crisis in selected Latin

American countries from a gender perspective.


19

indigence (data from 2009).20 This situation, in conjunction with informal labour conditions, which make access to social protection systems more difficult (especially for women), the low level of public spending on social protection programmes,21 the gradual ageing of the population in Latin America22 and its impact in terms of poverty in this segment of the population, and the limited effects of the current conditional transfer programmes on human development23 demonstrate not only that social protection systems are still needed in the region, but that reforms are necessary to ensure greater access and coverage in order to foster human, social and economic development.

Consistency between the type of support provided by the programme and the capacity-building needs of partners. ECLAC reports to SIDA, interviews with staff of the ECLAC divisions and the results of the survey indicate that there is significant demand among government actors and public authorities with regard to capacity-building in the area of social protection. Indeed, the fact that the technical assistance provided by ECLAC under the cooperation programme 2010-2011 was requested by those same governments and public institutions is proof of this. Furthermore, nearly all of those surveyed who had received technical assistance (30 out of a total of 34) said they would repeat their request to ECLAC for assistance on social protection. This can be seen not only as an indicator of satisfaction with the technical assistance received, but also as an indicator of the demand for training. The figures below, prepared on the basis of the survey results, confirm the relevance of the technical assistance provided by ECLAC and of the studies conducted under the cooperation programme 2010-2011.

The comparative advantage that ECLAC has in the region in terms of expertise in social protection, as recognized by the experts interviewed, a representative of a similar organization and those interviewed in the three countries selected.

35. Data collected (remotely) in the three selected countries support these findings and, in the case of Costa Rica and Ecuador, confirm the relevance of the programme with the objectives of the programme‘s components found to be consistent with the national priorities established by the governments. The following paragraphs provide further detail on the relevance of the programme in each of the three countries under consideration.

20 Economic Commission for Latin America and the Caribbean (ECLAC), Social Panorama of Latin America

2010 (LC/G.2481-P), Santiago, Chile, 2011. United Nations publication, Sales No.:E.11.II.G.6. 21 Francisco Ferreira and David Robalino, Social Protection in Latin America: Achievements and Limitations,

World Bank, May 2010. 22 Economic Commission for Latin America and the Caribbean (ECLAC), Ageing, Solidarity and Social

Protection: Time to Move Towards Equality (LC/L.3451(CRE.3/3)), Santiago, Chile, 2012. 23 Francisco Ferreira and David Robalino, Social Protection in Latin America: Achievements and Limitations,

World Bank, May 2010; Enrique Valencia Lomelí, David Foust Rodríguez and Darcy Tetreault Weber, ―Sistema de protección social en México a inicios del siglo XXI”, Project Documents, No. 0474 (LC/W.0474), Santiago, Chile, Economic Commission for Latin America and the Caribbean (ECLAC), April 2012.


20

Figure 2 SURVEY RESPONDENT’S LEVEL OF SATISFACTION WITH THE TECHNICAL

ASSISTANCE RECEIVED (Percentages)

Source: Prepared by the authors.

Figure 3 RELEVANCE OF THE STUDIES AND SEMINARS

(Percentages) Source: Prepared by the authors.


21

(a) Costa Rica

36. While social protection systems in Costa Rica are at an advanced stage of development, certain challenges persist with regard to ensuring universal access, including the difficulties that undocumented or indigenous people face in accessing services, poor coordination between the various existing programs, the problems of information management, and population ageing and its impact on national social protection systems. In Costa Rica, which is classified by ECLAC as a country experiencing moderate population ageing,24 an estimated 11.6% of the population will be over 65 years old in 2025, compared with 6% in 2010.25 The academic experts interviewed recognized that the ageing population and the consequent need to adapt national social protection systems to this new demographic configuration are priority areas for the country in terms of social protection. Although support for older persons has been on the public authorities‘ agenda for several years26 (as evidenced by the establishment of the National Council for Older Persons (CONAPAM)27 in 1999 and the National Policy on Ageing and Old Age 2011-2021), ensuring access for this segment of the population to social protection services continues to pose some difficulties. Those interviewed identified the following as the biggest challenges:

Inadequate management of available resources for social protection, as evidenced by the solvency crisis affecting the Costa Rican Social Security Fund in 2011;

Public officials‘ limited capacity to provide appropriate services to this segment of the population as befits their condition;

Weak State capacity to mitigate the consequences of the ageing process from the point of view of traditional caregivers (usually women);

The low level of coordination between the different institutions and social protection programmes; and

High poverty rates among the elderly population, which prevents or hinders access to the social benefits available.

37. In this context, the work of the Financing for Development Division, the Division for Gender Affairs and the Population Division of ECLAC was rated as relevant or very relevant by all those interviewed as it has provided guidance in relation to solving the problems mentioned. Specifically:

The study entitled ―Equity and solvency of the health system in the face of population ageing: the case of Costa Rica‖ (component 1 of the programme), which projects the impact that population ageing will have on the solvency of the Costa Rican Social Security Fund, is relevant, first and in the long term, for the Costa Rican Social Security Fund and second, indirectly, for the general population. In effect, this study

24 Economic Commission for Latin America and the Caribbean (ECLAC), Population, ageing and development

(LC/G.2235(SES.30/16)), Santiago, Chile, 2004 cited by Fabio Betranou, ―Restricciones, problemas y dilemas de la protección social en América Latina: enfrentando los desafíos del envejecimiento y la seguridad de los ingresos‖, Bienestar y política social, vol. 1, Nº 1, 2005.

25 Fuentes-Bolaños, Carlos ―El envejecimiento de la población y la política pública en Costa Rica‖, Revista Seguridad Social, No. 260, September 2010.

26 For example, the 2011-2014 national development plan recognizes care for the elderly as a strategic social well-being goal.

27 CONAPAM reports to the Office of the President of the Republic and provides guidance on matters of ageing and old age.


22

may be useful to the authorities of the Fund in redefining their policies and management practices in order to achieve its strategic objectives, which are to ―ensure the safety and quality of all health services and other social benefits provided by the institution‖ and to ―extend social security protection to groups who are excluded from coverage on the basis of specific conditions‖.28

The study entitled ―Social protection and unpaid work: redistribution of responsibilities and care tasks. Costa Rica: a case study‖ (component 4 of the programme), which aims to identify the policies needed to support the redistribution of responsibilities and care work, is relevant to the National Women‘s Institute (INAMU),29 in connection with its efforts to formulate an action plan for implementation of the National Policy for Gender Equality and Equity (PIEG) 2007. The study examines one of the reasons for the low participation of women in the labour market, which subsequently restricts their autonomy and ability to participate fully in the economic and social development of society;

The document ―From commitment to action. Guide to the review and evaluation of the Brasilia Declaration in Latin America and the Caribbean‖ (component 3 of the programme), which provides a methodological framework for the evaluation of the Brasilia Declaration, was relevant to the National Council for Older Persons (CONAPAM), which is responsible for implementing and monitoring, on behalf of the Government of Costa Rica, the commitments of the Brasilia Declaration on Ageing and the Madrid International Plan of Action on Ageing.

(b) Ecuador

38. Social protection programmes in Ecuador are going through a transitional phase, moving from a welfare model towards a comprehensive rights-based model.30 According to the officials interviewed, although this transition is complete from a conceptual point of view, there remain difficulties to overcome in relation to the implementation of this model. One of the areas of reform is the social security system and the low coverage of the pension system is one of the main challenges.31 Increasing the coverage of the pension system was identified in 2009 as one of the priorities of the Coordinating Ministry for Social Development.32 It was in that context that the Ecuadorian authorities —namely, the Ministry of Economic and Social Inclusion and the Coordinating Ministry for Social Development— requested the advice of the Social Development Division and the Financing for Development Division of

28 Costa Rican Social Security Fund (CCSS), Una Caja Costarricense de Seguro Social renovada hacia el 2025

[online] http://www.cendeisss.sa.cr/seguridadsocial/politicas2007-25.pdf. 29 No representatives of INAMU were available for interview and it was therefore not possible to confirm this

finding, which is based on the feedback given by persons from outside INAMU and on information gleaned in documents.

30 See also [online] http://www.rlc.fao.org/es/prioridades/seguridad/ingreso6/documentos/Presentaciones/Pa%C3% ADses/ECUADOR.pdf.

31 Fabio Durán Valverde, Diagnóstico del sistema de seguridad social del Ecuador, International Labour Organization (ILO), June 2008 [online] www.ilo.org/gimi/gess/RessFileDownload.do?ressourceId=6093.

32 Seminar entitled ―Social security and comprehensive protection for older adults: challenges and strategies in Latin America and the Caribbean and international experiences‖, which was held on 30 November and 1 December 2009 in Lima.

Comment made by one survey respondent on the relevance of the programme: ―What I value is having a qualified interlocutor ... ECLAC provides sound technical support.‖

http://www.ilo.org/gimi/gess/RessFileDownload.do?ressourceId=6093


23

ECLAC. Officials from the Ministry of Economic and Social Inclusion considered the advice received to be very relevant for two main reasons: lack of national experts in the field of social protection, especially in connection with this kind of approach; and the expertise of the staff of the Social Development Division and their ability to share that specialist knowledge. 39. Another aspect of the social protection system which requires closer attention in the government‘s reform agenda is the situation and needs of women. The overrepresentation of women in unpaid domestic work, which is recognized as productive work in the 2008 Constitution (as it was in the 1998 Constitution), has attracted the attention of government authorities. In the last five years, the Government of Ecuador has tried to ensure the effective recognition of domestic work as productive work worthy of remuneration.33 This objective is also included in the National Plan for Good Living 2009-2013, policy 6.2 which is to ―foster the acknowledgement of autonomous work, human and family care, self-consumption, as well as the integral transformation of their conditions‖.34 40. Consequently, the conceptual framework and the case study on care responsibilities and tasks in Ecuador prepared by the Division for Gender Affairs was highly relevant to the Government, as they include policy guidelines aimed at extending the coverage of the right to care and social protection.

(c) Paraguay

41. Only a limited assessment of the relevance of the programme in Paraguay could be conducted owing to the difficulty in obtaining interviews with government officials. For that reason, it was only possible to assess, and then only partially, the relevance of the programme with respect to the technical assistance provided by the Population Division to the Department for Older Persons. As confirmed in several studies, one of the country‘s most vulnerable groups consists of older adults, particularly women, who often live in poverty or extreme poverty.35 While the percentage of older persons in the population is still relatively low (6%), it is expected to increase to around 20% in 2050, resulting in a considerable demographic change in the country, which will have a significant impact on the national social protection system. According to the interviews conducted, although the Constitution of Paraguay recognizes the right of older persons to full protection, the issue of old age and ageing has not, so far, received the attention it requires by the Government. In addition, the

33 Jeanette Sánchez, ―Gobierno Constitucional del Ecuador. Protección social y trabajo doméstico no

remunerado en Ecuador‖ [online] http://www.eclac.cl/mujer/noticias/paginas/0/29390/PresentacionJeanetteSanchez.pdf.

34 Republic of Ecuador, National Development Plan. National Plan for Good Living 2009-2013. Building a Plurinational and Intercultural State. Summarized version, Quito.

35 ―Informe del Estado Paraguayo sobre el Cumplimiento de la Resolución 8/11 sobre derechos humanos y extrema pobreza‖ [online] http://www.ohchr.org/Documents/Issues/EPoverty/socialprotection/Paraguay.pdf. See also: Observatory on Ageing and Old Age, Interdisciplinary Centre for the Study of Ageing and Old Age (NIEVE) of the University of the Republic. United Nations Population Fund (UNFPA). Regional Symposium of Experts on Ageing, Gender and Public Policy organized by the Interdisciplinary Centre for the Study of Ageing and Old Age (NIEVE) of the University of the Republic with the support of the United Nations Population Fund (UNFPA), Montevideo, 9 and 10 September 2010; Jens Holst, La protección social en Paraguay. Condiciones, problemas, desafíos y perspectivas de los sistemas provisionales, Technical Secretariat for Planning, Office of the President of the Republic/German Agency for Technical Cooperation (GTZ), 2003; Andalusian Agency for International Development Cooperation, Prioritary Country Operational Programme: Paraguay, 2009-2011, Asunción.

http://www.eclac.cl/mujer/noticias/paginas/0/29390/PresentacionJeanetteSanchez.pdf

http://www.ohchr.org/Documents/Issues/EPoverty/socialprotection/Paraguay.pdf


24

entity set up to guide the development of policies and plans on old age and ageing has limited operational capacity, as it was created relatively recently (2008) and has few resources at its disposal. In the light of this situation, with the aim of strengthening this entity and extending and improving the protection of older persons, CELADE provided technical assistance to the Department for Older Persons in relation to the development of a national policy plan on ageing. This support was deemed very relevant by those interviewed because it aimed to meet an immediate need of the Department, which is, in the long run, on of the country‘s needs.

Finding 2: The objectives of the cooperation programme 2010-2011 are relevant to the work of the ECLAC divisions.

42. The relevance of the programme to ECLAC was evaluated by analysing the draft programme of work of the ECLAC system, 2010-2011,36 and conducting interviews with the divisions responsible for programme implementation and the PPOD management team. As shown in table 1, this analysis showed that three of the programme‘s objectives are consistent with the objectives of ECLAC. Although the fourth objective (on financing of social protection and universal coverage) is not stated explicitly in the draft programme of work of the ECLAC system, 2010-2011, its relevance to ECLAC can be seen in the various references in the draft programme of work to the connection between the need to ensure universal coverage of social protection, financial constraints and the balance between contributory and non-contributory mechanisms.

Table 1 CONSISTENCY BETWEEN THE OBJECTIVES OF THE COOPERATION

PROGRAMME 2010-2011 AND THE DRAFT PROGRAMME OF WORK OF THE ECLAC SYSTEM, 2010-2011

Objectives under each component of the cooperation

programme 2010-2011

Relevant extracts of the draft programme of work of the ECLAC system, 2010-2011

Comments

Component 1: Strengthen the capacity of governments in the region to formulate and implement policies and financial mechanisms in order to improve social protection services, including pensions and health services, in terms of expanding and deepening both the coverage and quality of

services.

6. ―In other words, the region is facing the imperative need to underpin a social-rights-based covenant reconciled with the existing financial constraints‖

Component 1 was implemented by the Financing for Development Division, established in 2011. As this subprogramme was formally added to the programme of work for the ECLAC system in the biennium 2012-2013, there are no specific references to this division in the programme of work, 2010-2011. These activities were previously the responsibility of a unit under another subprogramme (it is worth recalling that the programme of work, 2012-2013, was developed in 2010).

36 ECLAC, ―Draft programme of work of the ECLAC system, 2010-2011. LG/G.2373(SES.32/9), 16 May

2008.


25


programme 2010-2011


Comments

The creation of the new subprogramme was approved by the member States at the ordinary session of the United Nations General Assembly in 2011. It is precisely because this topic is relevant to the work of ECLAC and to member States that a proposal was put forward to create a specific subprogramme to address this issue.

Consistency between the objectives of the cooperation programme 2010-2011 and subprogramme 4 of the draft programme of work of the eclac system 2010-2011

Objective of the Organization: To foster an efficient generation and allocation of financial resources to support development in Latin America and the Caribbean

Component 2: Improve the capacity of governments with respect to analysis and policymaking through political debate and technical assistance, in relation to the policies on poverty reduction and fostering social cohesion, with special emphasis on conditional cash transfer programmes in the context of the present international financial crisis.

Subprogramme 4: Social development and equity

Objective: To accelerate poverty reduction and the achievement of social equity and cohesion in the region.

Strategy: The strategy will consist in developing applied research, generating analyses, disseminating the results and formulating policy recommendations to strengthen the capacity of stakeholders to make sound diagnoses of social issues for application into policies and programmes.

High level of consistency

Component 3: Better integrate the interests and needs of older adults into all aspects of development and policymaking in the countries of the region, in accordance with the recommendations of the Madrid

Subprogramme 6: Population and development

Objective: To improve the incorporation of population issues into development policies and programmes in countries of



26


programme 2010-2011


Comments

International Plan of Action on Ageing and its main regional adaptations, from the perspective of inclusion and social protection.

Latin America and the Caribbean.

Strategy: To provide support to the ECLAC sessional Ad Hoc Committee on Population and Development and to the countries of the region, in the follow-up to the Programme of Action of the International Conference on Population and Development (ICPD), in the fulfilment of the Regional Strategy for the Implementation in Latin America and the Caribbean of the Madrid International Plan of Action on Ageing, and in monitoring the goals and objectives of the Millennium Declaration and other relevant international conferences

Component 4: Improve national capacities of governments to formulate policies that promote gender equality through the documentation and dissemination of information and in-depth analysis and provision of technical assistance with respect to the effects of the international economic crisis in selected Latin America countries from a gender perspective.

Subprograme 5: Mainstreaming the gender perspective in regional development

Objective: To achieve the mainstreaming of the gender equality agenda into the principal policies of the Governments of the Latin America and Caribbean countries.

Strategy: To support the collection of basic data on the situation of women and men and the development of gender indicators, as well as to provide technical assistance for capacity-building to sectoral ministries and selected entities, like national statistical offices. The subprogramme will also support the sharing of experiences and good practices among member States of the Regional Conference on Women in Latin America and the Caribbean.


Source: Prepared by the authors.


27

43. The interviews with the officials from the different ECLAC divisions and the PPOD management team revealed other factors explaining the relevance of the program, such as:

The continuity in the work that the divisions have carried out on the issues covered by the programme, particularly in the case of CELADE and Division for Gender Affairs;

The flexibility of the programme, which allowed the divisions to work in more countries than originally agreed. The divisions‘ work was thus regional in scope (or almost regional as will be explained below); and

The type of activities funded, that is, ECLAC conducted applied research enabling it to provide advice and, thus, better position itself in the countries to have more impact on policymaking.

44. The key documents for the cooperation programme 2010-2011 were unspecific regarding the geographical scope of the initiative. According to the cooperation agreement and project proposal, the objective of the programme was to facilitate the analysis of development issues in Latin America and the Caribbean. However, the project proposal states that the programme should focus on the Latin American countries that are a priority for SIDA. This discrepancy was not identified as a problem by either ECLAC or SIDA. It is also important to recognize that the ECLAC divisions partially integrated Caribbean into the programme by, for example, translating some publications into English and inviting officials from Caribbean countries to participate in the courses and seminars that were organized. However, from an organizational standpoint, the support provided by the programme to ECLAC fulfils only part of its mandate. The Commission‘s limited presence in the Caribbean has been raised in other assessments as a problem that calls into question the relevance of the programmes implemented by ECLAC in the subregion.37 The 2010-2011 cooperation programme further reinforces the Commission‘s focus on Latin America.

Given that the ECLAC mandate covers both Latin America and the Caribbean, ECLAC and donors should promote the application of a more regional approach in relation to programmes. While this does not mean that cooperation agreements cannot concentrate on a single subregion, it would be advisable to identify opportunities that allow the transmission of the lessons learned throughout the region.

Finding 3: Although the geographical priorities of Swedish cooperation have

changed, social protection remains an important issue for the ASDI and also a means to work on other priorities. The cooperation programme 2010-2011 and, more generally, the experience that ECLAC has accumulated on social protection issues are therefore very relevant to the Swedish agency’s cooperation priorities.

45. At the time of this evaluation, the priorities of the Swedish International Development Cooperation Agency had changed with regard to the priorities it set when signing the cooperation agreement with the Commission for the programme subject to evaluation here.

37 Office of Internal Oversight Services (OIOS), ―Comprehensive Audit of ECLAC‖, 31 January 2011 [online]

http://usun.state.gov/documents/organization/165457.pdf.


28

In line with its new priorities, the SIDA will focus its future efforts on African countries, but will continue working on the issue of social protection. 46. The 2010-2011 cooperation programme offers interesting insights into addressing specific priorities for SIDA. For example, the Ministry for Foreign Affairs document entitled Policy for Economic Growth in Swedish Development Cooperation, 2010–201438 establishes that the general objective of this line of work is to improve the conditions for sustainable economic growth in developing countries and recognizes the development of individuals as one of the fundamental conditions for this. In this sense, the work carried out under the framework of the cooperation programme is highly relevant since it was based on the premise that social protection is a key element for ensuring individual development and facilitating the individual‘s participation in social and economic development. 47. Another example of the programme‘s relevance to the SIDA priorities is the issue of gender equality, which is addressed in the document On Equal Footing,39 whose lines of action seek to promote equal access of women and men to decent employment, including the possibility of reconciling family life and paid employment. The other issues addressed by ECLAC under the cooperation programme are also relevant to the SIDA priorities (namely, democracy, equality and human rights; economic development; knowledge, health and social development; sustainable development; and human security), because these are issues with implications not only in terms of social development, but also sustainable development (development will not be sustainable unless it ensures the participation of all people in the production process), human security (which is impossible without guaranteeing people‘s basic rights to education, health and nutrition) and democracy (which is closely linked to individuals‘ ability to take part in decision-making). 48. Also, while Latin America and the Caribbean is no longer a priority region for SIDA, the Swedish agency continues to work in the following countries: Colombia, Guatemala and the Plurinational State of Bolivia. While the work of SIDA focuses on the issues of peace and security in Colombia, in Guatemala and the Plurinational State of Bolivia it focuses on the fight against poverty, which is an issue closely related to social protection. 49. In conclusion, the cooperation programme 2010-2011 and, more generally, the experience that ECLAC has accumulated on social protection issues are therefore very relevant to the Swedish agency‘s cooperation priorities.

2. Effectiveness 50. According to the OECD definition, the effectiveness of a development intervention depends on the extent to which its objectives are achieved, or are expected to be achieved, taking into account their relative importance.40

38 Ministry for Foreign Affairs, Policy for Economic Growth in Swedish Development Cooperation, 2010–2014

(UF/2010/6949/UP), Stockholm, 11 February 2010. 39 Government Offices of Sweden, On Equal Footing. Policy for Gender Equality and the Rights and Role of

Women in Sweden’s International Development Cooperation 2010-2015. 40 Organisation for Economic Cooperation and Development (OECD), Glossary of Key Terms in Evaluation and

Results Based Management, Paris..


29

51. The evaluation sought to reveal the extent to which the expected outcomes of the 2010-2011 cooperation programme were achieved in terms of:

Capacity-building;

Fostering policy debate;

Promoting the application of best practices. 52. The programme was not evaluated using the logical frameworks developed for each of its components for the following reasons:

As a final evaluation, even though this section refers to the components, the aim was to assess the overall effectiveness of the programme as a whole, rather than reviewing the effectiveness of each separate component;

The statements of results and the proposed indicators in the logical framework of the components did not constitute a sufficient assessment of the programme (see finding 8 for more details).

53. The evaluation also analysed the extent to which gender considerations and a human rights-based approach were incorporated into the cooperation programme. 54. Before presenting the findings of the programme, it is important to mention two elements in order to understand the context in which ECLAC operates and, thus, be able to judge its effectiveness. The first is related to the principle of non-interference stated in Article 2 of the Charter of the United Nations, from which it follows that ECLAC, as part of the United Nations system, can advise governments and provide inputs to the discussion, formulation and implementation of public policies, but it cannot directly influence those policies. 55. The second element is that one of the difficulties in understanding how research contributes to policy change is that its impact is neither linear nor predictable. This has two consequences: first, in terms of the relationship between research and policy change, the former can be seen as helping to change the attitudes, skills and knowledge needed for policy change; second, the time between the intervention and policy change is usually long. Carol Weiss41 acknowledges that research influences public policymaking, but that this contribution is generally incremental and starts with a first step to sensitize politicians. Over time, the ideas generated by research can surface in policy discourse and on the policy agenda. 56. Carol Weiss also provides an analytical framework for understanding and evaluating the impact of research on policymaking. The author cites data, arguments and ideas as ways in which research can have an influence. Therefore, this evaluation has sought to

41 Carol H. Weiss, ―Policy research as advocacy: pro and con‖, Knowledge and Policy, vol. 4, Nº1-2, 1991. 42 La Nación, ―Jubilados, salarios y empleos. La verificación de los mensajes políticos de la semana‖,

30 September 2012 [online] http://www.lanacion.com.ar/1513037-jubilados-salarios-y-empleo.

Although Argentina was not one of the case studies for this evaluation, the evaluator noted a newspaper article42 which cites data from the study ―Social protection in Argentina‖, funded by the cooperation programme 2010-2011, to verify the claims made by the President of Argentina on the rate of social security coverage at the national level. It was not possible to verify whether the President was aware of the study data, but this highlights the varied and unpredictable use of the data and reasoning in research and their link with the policy.


30

determine to what extent the cooperation programme provided policymakers with knowledge, data, arguments and ideas. Consequently, the analysis focused on the results of the cooperation programme 2010-2011, not in terms of outcomes (which have already been presented in the final programme report for SIDA), but in terms of the short- and medium-term results that would otherwise be invisible or little known owing to issues (mentioned later in this report) related to the ECLAC monitoring system. Finding 4: The data collected suggest that the programme has been effective in

strengthening the capacities of the different actors in the region whose work is related to social protection, providing, among others, knowledge, data and tools for policy analysis.

57. Based on data collected using different methods, it can be concluded that the products and services provided by the cooperation programme 2010-2011 have been highly useful to the beneficiary countries. The results of the survey conducted as part of this evaluation are consistent with those obtained by ECLAC in the surveys conducted by those responsible for components 1 and 2 following the seminars that were held, which most participants considered ―useful‖ or ―very useful‖. As shown in the following figures, studies and seminars were considered useful or very useful by not less than 87% of respondents. No respondents said that the technical assistance had not been useful. With respect to the objectives, 74% of respondents (25 persons) felt that these were fully achieved, while 15% of the sample said that they were partially achieved.43

Figure 4

USEFULNESS OF THE SEMINARS, WORKSHOPS AND STUDIES ACCORDING TO THE SURVEY RESPONDENTS a

(Percentages) Source: Prepared by the authors. a The survey asked about the usefulness of each study, workshop and seminar held. Total number of

respondents: question on studies: 117; questions on seminars or workshops: 67.

43 The other four recipients of technical assistance (12%) did not answer the question.


31

58. The survey findings suggest that the outputs and services of the programme have been useful in strengthening capacities (knowledge and skills) of the various actors in the region whose work is related to social protection. 59. As figure 5 shows, the technical assistance has enabled individuals to:

acquire an understanding of specific topics relating to social protection, such as the work of women, population ageing and the challenges and opportunities of social protection systems, among others;

build capacities with a view to promoting the improvement of the social protection system, seeking sources of information relevant to ongoing efforts in the area of social protection and estimating the costs of implementing social protection policies.

60. The survey findings suggest, above all, that the organizations for which the respondents work have indeed seen a strengthening of capacities to advance the discussion on social protection at the national and regional level, design, monitor and evaluate public policies for social protection and implement social protection programmes, focusing on the most vulnerable groups and ensuring that they are taken into account during the programme development stage. Technical assistance has been less useful for organizations in regard to securing and allocating resources for work in the area of social protection.

Figure 5 USEFULNESS OF TECHNICAL ASSISTANCE FOR RESPONDENTS a

(Percentages) Source: Prepared by the author. a More than one response per person is possible. Total number of responses: 69. Total number of respondents to

the question: 32.


32

Figure 6 USEFULNESS OF TECHNICAL ASSISTANCE FOR RESPONDENTS’ ORGANIZATIONS a

(Percentages) Source: Prepared by the author. a More than one response per person is possible. Total number of responses: 62. Total number of respondents to

the question: 31.

61. Interviews with respondents in Costa Rica, Ecuador and Paraguay revealed that, in addition to building expertise, the programme has raised awareness of specific social protection issues, provided a rationale, data and tools for analysis, strengthened capacities for gender analysis and boosted respondent confidence. These contributions are useful for promoting, developing, implementing, monitoring and evaluating public policies.

Respondent feedback ―The information provided by the study is useful for developing fund policies, and it is helpful for the finance division as well, which has to set and monitor the budgetary ceiling to see what demand can be met. The study enabled us to answer some of the questions that we had on, for example, whether women have more coverage than men.‖ ―ECLAC gave us the guidance we needed for outlining the national plan on ageing while ensuring that the document reflected our country‘s needs … they steered us along the way and provided the frameworks we were lacking.‖ ―No one is a prophet in his own land. It is very useful to see our region‘s stock of experience.‖ ―Support from ECLAC helped us put our ideas on sounder footing.‖ ―Something that I wanted to highlight is that job training

programmes for persons living in poverty should operate alongside psychosocial development programmes for individuals and families, as we have done in several of our programmes targeting those sectors. I think that very few countries are doing this...‖.


33

62. The interviews also showed how the knowledge, rationale, data, skills and instruments have been used in the three countries. 63. In Costa Rica, the Office of the Financial Manager of the Costa Rican Social Security Fund is using the data and analyses in the study on ageing and health systems in its budgetary ceiling projections. Thanks to the study, which examines how social protection services (particularly, health services) are used by different population groups (on the basis of sex and income), the fund‘s authorities have understood the need to shift the supply of services towards a more preventive approach and consider the impacts on the existing insurance contribution system. Another aspect that has to be taken into account, in terms of strengthening capacities, is that the study provided a framework for analysing health data from the census. 64. By showing the importance of the cultural factor in determining care responsibilities, the study of care responsibilities and tasks led to the revamping of Costa Rica‘s National Policy for Gender Equality and Equity (PIEG) to make this factor an important component of the plan and translate this recognition into appropriate action to raise awareness among the population.44 The study also served as input to improve how household surveys measure time use. 65. The international course on ageing and social protection strategies in Latin America conducted in the framework of the programme provided material for staff training and sensitization at public institutions in Ecuador (such as the Ministry of Public Health and the Ministry of Economic and Social Inclusion) on the rights of older persons. The technical assistance provided by the Social Development Division of the Ministry of Economic and Social Inclusion helped to give the latter a better basis for the ideas that should guide the implementation of social protection policies. 66. In Paraguay, the technical assistance provided by CELADE to the Bureau for Older Persons has been seen as key for developing the basic pillars of the National Policy Plan on Older Adults and for better integrating the needs of older adults and, especially, women in this document. Finding 5: The data suggest that the programme has helped to foster political debate

at the regional level on the issues of ageing, women’s unpaid work and comprehensive social protection. No evidence was found that the programme contributed to policy discussions on the funding of health systems and non-contributory pension schemes.

67. As for advancing the political debate on equity and social protection, an analysis of documents and interviews suggests that while the programme has made a significant contribution in some areas, including issues related to ageing, women‘s unpaid work and comprehensive social protection, its contribution in other spheres (funding of health systems and non-contributory pension regimes) has been marginal and unsystematic. In the first case, the outputs funded by the cooperation programme 2010-2011 served as a basis for reflection in regional forums and, to a lesser extent, at the national level. The most relevant examples of this contribution are set out below.

44 This information could not be confirmed with the National Institute for Women because representatives

were not available.


34

68. The contribution of the study ―Inclusive social protection in Latin America: A comprehensive, rights-based approach‖45 to the 29 May 2012 joint declaration on ―Food security: A priority for the Inter-American Social Protection Network‖ of the Food and Agriculture Organization of the United Nations (FAO) and the Organization of American States (OAS). The declaration stresses the significant link between food security and social protection and proposes a social protection strategy for food security. The strategy was developed further on the basis of the above-mentioned study.46 In Argentina, the study also contributed to an initiative for dialogue among government officials, civil society leaders, business leaders, academics and representatives of international agencies on social protection in Argentina.47

The analysis of social protection systems in the study ―Conditional cash transfer programmes. The recent experience in Latin America and the Caribbean‖,48 funded by the cooperation programme 2010-2011 was used by Ecuador‘s Ministry of Economic and Social Inclusion (MIES) for a compendium of technical documents in preparation for the international meeting ―De pobres a ciudadanos: experiencias latinoamericanas de equidad y movilidad social‖, held in Ecuador in August 2012. MIES defined the meeting as ―part of the process of reflection‖ on economic and social policies in the region.49

The indirect contribution50 of the document ―From commitment to action. Guide to the review and evaluation of the Brasilia Declaration in Latin America and the Caribbean‖ to adoption of the San José Charter on the rights of older persons in Latin America and the Caribbean and, as a result, the reaffirmation of the commitment to an international convention to protect the rights of older persons.

The case study on care responsibilities and tasks in Costa Rica, presented at the international seminar ―Unpaid labour and the social protection of women‖ (with the participation of INAMU, the Ministry of Labour and Social Security and the Ministry of Social Welfare) advanced the dialogue among these ministries on the need for social and economic policy coordination.51

45 S. Cecchini and R. Martínez, ―Inclusive social protection in Latin America: a comprehensive, rights-based

approach‖, Libros de la CEPAL, No. 111 (LC/G.2488-P), Santiago, Chile, Economic Commission for Latin America and the Caribbean (ECLAC), 2011. United Nations publication, Sales No. S.11.II.G.23.

46 Food security: A priority for the Inter-American Social Protection Network [online] http://www.oas.org/ es/centro_noticias/comunicado_prensa.asp?sCodigo=C-193/12.

47 Centro de Implementación de Políticas Públicas para la Equidad y el Crecimiento (CIPPEC) and others, Diálogos de Protección Social. Reporte final 2011.

48 S. Cecchini y A. Madariaga,, ―Conditional cash transfer programmes. The recent experience in Latin America and the Caribbean‖, Cuadernos de la CEPAL, No. 95 (LC/G.2497-P), Santiago, Chile, Economic Commission for Latin America and the Caribbean (ECLAC), 2011. United Nations publication, Sales No. E.11.II.G.55.

49 Overall Knowledge Management Coordinator, ―De pobres a ciudadanos. Experiencias latinoamericanas de equidad y movilidad social. Documentos técnicos‖, Quito, Ministry for Economic and Social Inclusion (MIES).

50 The contribution is referred to here as indirect because the document served as a guide for drafting national reports on follow-up to the Madrid International Plan of Action. National reports were submitted at the Third Regional Intergovernmental Conference on Ageing in Latin America and the Caribbean, which ended with adoption of the San José Charter reaffirming the need for an international convention on the rights of older persons.

51 It has not been possible to confirm how or to what extent the study facilitated the dialogue between these ministries because it was not possible to talk with representatives of the agencies involved.


35

69. Three conclusions may be drawn from these examples:

The programme is more effective in advancing the political debate where the outputs were developed in the framework of a set of initiatives of ECLAC and other actors than in cases where there was no such connection;

The outputs of the programme were among the factors that helped advance the political debate on equity and social protection in the region; and

The programme‘s contribution to the discussion on equity and social protection is more readily assessed when looking at a longer period and taking into account the cumulative impact of ECLAC efforts over the past ten years.

Finding 6: The promotion of best practices related to the development,

implementation, follow-up and evaluation of social protection policies is one of the aspects of the programme that the respondents valued the most. During the data collection process, two examples were found of how such practices are being implemented in Costa Rica and Ecuador. And there is evidence of greater awareness at the individual level.

70. Within the framework of the programme, the use of best practices related to the development, implementation, monitoring and evaluation of social protection policies in the countries of the region was promoted by means of studies, seminars, conferences and technical assistance. Among the region-wide studies and publications on regional trends and best practices are the following: ―Conditional cash transfer programmes. The recent experience in Latin America and the Caribbean,‖ ―Inclusive social protection in Latin America: A comprehensive, rights-based approach‖, and ―Ageing and the protection of human rights: current situation and outlook.‖52 Best practices have also been disseminated through studies on specific countries where social protection system reforms are at a more advanced stage. The experience accumulated by these countries could guide reforms in others. Examples of this research include ―La protección social en Chile. El Plan AUGE: avances y desafíos‖53, ―La reforma del sistema de pensiones en Chile: desafíos pendientes,‖54 ―Protección social y trabajo no remunerado: redistribución de las responsabilidades y tareas del cuidado. Estudio de caso Costa Rica‖ and ―Protección social y trabajo no remunerado: Redistribución de las resonsabilidades y tareas del cuidado. Estudio de caso Ecuador‖. 71. Survey data confirm that comparisons with other countries are key for social protection policy development and implementation because they show that the studies funded by the cooperation programme 2010-2011 helped to identify effective social protection policies and strategies.

52 Sandra Huenchuan and Luís Rodríguez-Piñero, ―Ageing and the protection of human rights: current situation

and outlook‖, Project Document Nº 353 (LC/W.353) Santiago, Chile, Economic Commission for Latin America and the Caribbean (ECLAC), 2010.

53 Álvaro Erazo,―La protección social en Chile. El Plan AUGE: avances y desafíos‖, serie Financiamiento del Desarrollo, Nº 238 (LC/L.3348), Santiago, Chile, Economic Commission for Latin America and the Caribbean (ECLAC).

54 Andras Uthoff, ―La reforma del sistema de pensiones en Chile: desafíos pendientes‖, serie Financiamiento del Desarrollo, Nº 112 (LC/L. 1575-P), Santiago, Chile, Economic Commission for Latin America and the Caribbean (ECLAC).

http://biblioteca.cepal.org/search~S0*spi?/h%5bE%2F%5dLC%2FW.353/h%5be%2F%5dlc%2Fw.353/-3,-1,0,B/browse





36

Figure 7 PROGRAMME STUDY USEFULNESS a

(Percentages) Source: Prepared by the author. a More than one response per person is possible. Total number of responses: 148. Total number of respondents

to the question: 73.

72. Findings from interviews conducted in Costa Rica, Ecuador and Paraguay are consistent with the survey findings: according to the respondents, it is very useful to compare social protection systems across countries and present best practices at seminars, workshops and courses in the framework of the programme. 73. However, the data collected in the three countries did not point to systematic use of shared best practices in designing, implementing, monitoring or evaluating social protection policies. During the preparation of this report, two cases of best practices came to light: one in Costa Rica, relating to the implementation of measures to raise awareness among persons in charge of services for older adults, and one in Ecuador, on the development and implementation of time use surveys. A possible explanation for this finding is that a certain amount of political and institutional time needs to pass before best practices are implemented in other contexts: decision-makers probably use information on best practices as inputs for modifying existing policies and not so much as measures to be implemented in a mechanical fashion. In the short term, changes are more likely in awareness and sensitization at the individual level. As respondent feedback in the box shows, this explanation is supported by interviews.

Respondent feedback

―I was impressed by a presentation on Brazil, where the State and employers did a joint study. I think that employers should participate in decision-making on projects.‖

―Comparisons with other countries are important … we realized that access to public transport is a problem for older persons, so we have focused on training bus drivers.‖

―We are not behind; we are creating strategies and programmes, but we can also learn from other countries‘ experiences.‖


37

Finding 7: The level of integration of gender issues and the human rights-based approach in the programme has been modest. However, it should be noted that ECLAC cooperation with SIDA during 2010-2011 attached greater importance to gender issues linked to the issue of social protection during the programme development stage, when a component on women’s unpaid work was included.

74. Mainstreaming gender issues55 and the human rights-based approach56 in programming is a requirement that applies to all the agencies of the United Nations. Mainstreaming these two principles or approaches requires not only that the actions of the agencies of the United Nations promote human rights and equality between women and men as a final outcome, but also that these two principles or approaches guide the operations of the agencies in the design, implementation, monitoring and evaluation of initiatives. 75. The cooperation programme has clearly taken a human rights-based approach to the second component, promoting improvement of social protection systems for guaranteeing such fundamental human rights as the right to adequate food, education and health and freedom from discrimination based on gender. 76. A key element of the rights-based approach involves strengthening both the capacities of responsible parties to fulfil their obligations and the capacities of rights holders to hold them accountable. The programme focused on the former and addressed the latter marginally. This is a major gap in programme design, especially in view of the key role played by civil society in tracking implementation of public policies. 77. Implementing a human rights-based approach also promotes the use of participatory methods in the different phases of development, implementation, monitoring and evaluation, as well as transparency and access to information. The data collected show that civil society participation is limited at the study stage but more significant in seminars and workshops. In terms of transparency and access to information, most ECLAC studies and information on conferences organized in the framework of the programme are open to the public. 78. With regard to the mainstreaming of gender issues, the documents analysed and the interviews conducted indicate that ECLAC divisions have included this principle or approach in the different phases of programme implementation to a modest degree. With the exception of component 4 on unpaid work and social protection of women (where studies and activities are based on gender equality issues), only some of the studies and activities undertaken in the framework of the other components (between two and three per component) include gender issues (such as, for example, data disaggregated by sex). Some focus on a number of causes of discrimination against women. The cooperation programme proposal for 2010-2011 includes

55 United Nations, Mainstreaming the gender perspective into all policies and programmes in the United

Nations system, 1997. 56 United Nations Development Programme (UNDP), ―The Human Rights Based Approach to Development

Cooperation Towards a Common Understanding Among UN Agencies‖ [online] http://www.undg.org/archive_docs/ 6959-The_Human_Rights_Based_Approach_to_Development_Cooperation_Towards_a_Common_Understanding_ among_UN.pdf.


38

references to the mainstreaming of gender issues in programme implementation, but the logical framework developed for each component does not include specific indicators for this. 79. Nevertheless, joint ECLAC/SIDA efforts during 2010-2011 attached greater importance to social protection-related gender issues than in the previous stage of the programme (2007-2009). During the first phase, gender issues were integrated by involving the Women and Development Unit (now the Division for Gender Affairs) in specific activities. By contrast, a specific component on unpaid work and social protection of women was developed within the framework of the cooperation programme 2010-2011. 80. Similarly, the findings of the online survey show that technical assistance has been somewhat useful for mainstreaming gender and human rights issues, according to 54% of the responses concerning the usefulness of technical assistance for adapting and implementing social protection policies targeting the needs of the most vulnerable groups.

Figure 8 USEFULNESS OF TECHNICAL ASSISTANCE FOR MAINSTREAMING GENDER

AND HUMAN RIGHTS ISSUES a (Percentages)

Source: Prepared by the author. a The survey included the following question: How has technical assistance from ECLAC been useful for your

department or organization?


39

Finding 8: Assessing programme effectiveness in terms of short-, medium- and long- term results was hampered by several factors related to logical frameworks, the information reported to SIDA, the follow-up system and the programme’s theory of change.

81. The following sections examine the factors that hampered assessment of the programme‘s short-, medium- and long-term outcomes. (a) Logical frameworks 82. The logical frameworks of the four components include a set of results in terms of outcomes and direct impacts, as well as indicators for both. In some cases, the baseline is also established. According to the evaluation of the cooperation programme 2007-2009, ECLAC had improved the development of logical frameworks compared with the first phase of the programme, but there is room for improvement in some areas. These are:

Indicators: Among the indicators used in the components‘ logical frameworks are the number of downloads of publications and other documents and the capacities that were strengthened. With regard to the first type of indicator, while it was certainly a helpful instrument for measuring the usefulness of documents produced, if the absolute number of downloads was not accompanied by any type of user profile information it does not, in and of itself, provide key information for programme follow-up and evaluation. With regard to the second kind of indicator, lists are not specific enough because they neither define the types of capacities to be built nor identify the targets.

Results: The lists of results are generic. For example, they do not spell out the capacities that are to be built or why. Although governments are identified as the subject of the results, the ministries or agencies that need to be strengthened are not specified. In addition, in some cases the outcomes identified relate to actions.

Baselines: In some cases, it is not clear how the baseline values were calculated; in others the baselines did not seem appropriate to the type of outcome identified.

(b) Reporting to SIDA and the follow-up system 83. Another factor that has hampered assessment of programme results is the practice whereby divisions prepare reports almost exclusively in terms of products and services provided. This practice shows, on the one hand, that division management has focused mainly on products and services and not on development impact57 and, on the other hand, that follow-up ends with releasing a study or holding a seminar. Some divisions have monitored the use of publications according to the number of downloads from the ECLAC website, the messages received by event participants and end-of-workshop surveys. While these mechanisms can provide some information on the usefulness of studies and seminars, they are not enough for collecting

57 This consideration is supported by recent evaluations of ECLAC citing the difficulties, both at ECLAC and in

the United Nations in general, in moving from product-based management to results-based management. See, for example, D Miller, A. Khan and J.C. Peña, In-depth Evaluation of the Role of the Economic Commission for Latin America and the Caribbean (ECLAC) in the Caribbean, 2010 [online] http://www.cepal.org/dppo/noticias/ paginas/4/37534/EvaluacionPOS-V1.pdf.


40

meaningful, useful data for guiding decision-making with a view to boosting the use of outputs among their target population. 84. ECLAC and SIDA are jointly responsible for programme follow-up. According to the interviews, internal restructuring has kept the latter from ensuring regular programme follow-up. In addition, it is not clear from the reports that SIDA authorities exerted enough pressure for the reports (particularly, the final one) to provide an exhaustive accounting of the results achieved. (c) The programme’s theory of change 85. One last factor that makes it difficult to assess programme effectiveness, especially over the long run, is that it is not clearly tied to a theory of change that would make it possible to understand the rationale and underlying assumptions that link the strategy behind the initiative to the desired impacts. As shown above, the programme has built capacities and contributed to the discussion at the regional level on some specific issues. However, it is difficult to assess to what extent and in what conditions these results represent progress towards achievement of the ultimate goal (improving the coverage and quality of social protection systems in the region) because these elements are not spelled out in the programme. As already indicated in the preceding sections, there are a number of obstacles to access to social protection. These obstacles vary from country to country and include the following: underinvestment in social spending; poor coordination among programmes and institutions involved in social protection; and issues relating to the technical capacities of these institutions. 86. It is not expected that ECLAC will resolve all of these obstacles, but it is important to clearly identify them and explain why priority is given to one area of intervention over another. Once the target area has been identified, it is necessary to establish the steps needed to move from one result to another and how to do this. The programme strengthened the capacities of several officials involved in decision-making. How, or under what conditions, will this translate into better social protection system coverage and quality? To what extent was the programme pivotal in building the capacities needed for developing and implementing social protection policies with a rights-based approach? 87. Moreover, neither the actors that the divisions were seeking to influence, nor the means for doing so, were specifically identified. According to the interviews, the programme‘s sphere of influence includes the following groups of actors: ministries of social development, ministries of women‘s affairs, universities, the media, civil society organizations, regional and international organizations with a similar mandate, national institutes of statistics, social security funds and central banks. However, in a number of cases the relationship of these actors with the programme has not been spelled out, since it was not clear whether the divisions were regarded as primary or secondary users of the studies (especially in the case of the media, civil society and academia), how ECLAC expected them to access and use the studies, and for what purpose, and how influencing these actors would contribute to the development of public policies for social protection and, in the final analysis, improve access to social protection systems (in terms of coverage and quality). As explained in the preceding section, the logical frameworks developed for the four components do not provide enough information for answering these questions.


41

Finding 9: The cooperation programme 2010-2011 helped to better position ECLAC in the countries of Latin America and build staff capacities at the division level.

88. Interviews with ECLAC staff made it possible to identify the following programme results, which seem to be important because they broaden the opportunities for ECLAC to help shape public policies for social protection:

capacity building or furthering the specialization of some ECLAC staff in specific topics relating to social protection;

positioning ECLAC better in the countries of Latin America and, to a lesser extent, of the Caribbean, as evidenced by the requests for technical assistance received as a result of seminars and other activities undertaken in the framework of the programme (see box).

These requests for technical assistance suggest that there is significant demand for training in social protection policies. As noted by a government representative, this puts another type of pressure on ECLAC: to respond in a timely manner to requests for training.

3. Sustainability 89. According to the OECD definition, sustainability is the continuation of benefits from a development intervention after completion; the probability of continued long-term benefits; the resilience to risk of the net benefit flows over time.58 90. In terms of sustainability, the evaluation sought to answer these two questions:

1. To what extent are the results achieved in the selected countries sustainable beyond the programme of cooperation?

2. To what extent will the results achieved thanks to the programme further the work being done by ECLAC?

58 Organization for Economic Cooperation and Development (OECD), Glossary of Key Terms in Evaluation and

Results Based Management, Paris

Staff from ECLAC divisions cited a number of examples to show how cooperation programme 2010-2011 helped spur demand for assistance in building the capacities of governmental actors. Some examples are set out below.

As a result of the technical assistance provided to Ecuador‘s Ministry of Social Development Coordination, ECLAC was once again asked to support the ministry regarding some key elements of the social security reform process.

Also in Ecuador, the Ministry of Economic and Social Inclusion sought Social Development Division support

for implementation of existing social protection policies.

In Colombia, Financing for Development Division participation in the national congress GestarSalud led the Colombian authorities to request division guidance for improving health sector policies.

The government of Haiti invited the Social Development Division to provide technical assistance for implementing its national conditional transfer programme following release of the regional study

on such programmes.


42

Finding 10: The data provide evidence of the continuity of the impacts of some of the outputs and services provided in a number of countries under the cooperation programme 2010-2011.

91. Not only do the data highlight that the end of the cooperation programme 2010-2011 did not mean the end of efforts in the area of social protection; they also show that the products generated and the knowledge transferred have indeed been used, and that other actors have supported the effort (see examples in the box). This has been particularly the case when:

the divisions have joined forces with other organizations at the national, regional or international level; and

social protection system reforms have been initiated in the countries separately from the programme, as in Costa Rica and Ecuador.

92. The data gathered suggest that in some cases the use of studies by other actors has not been an intended result but rather an initiative on the part of the authors of the studies and their counterparts, who shared documents with their colleagues. Interviews with respondents in Costa Rica, Ecuador and Paraguay brought the following cases to light:

The study on care responsibilities and tasks in Costa Rica has been used by another researcher for a study on pensions commissioned by the World Bank. The study will look at non-contributory pensions with a view to moving away from the welfare approach towards a range of services that ensure the realization of human rights;

In Costa Rica, the study on ageing and the health system was found to be very useful by the Costa Rican Social Security Fund, which has used it for projecting budgetary ceilings and eventual policy changes. In addition, the author presented the study‘s findings during a seminar organized by the University of Costa Rica and shared them with the members of the PAHO mission in the framework of the PAHO assessment of the fund;

Examples of the sustainability of the outputs of the cooperation programme 2010-2011

In Ecuador, SENPLADES has signed an agreement with ECLAC to, among other things, design a comprehensive social protection system. The road map defined with ECLAC includes the four topics addressed in the framework of the cooperation programme 2010-2011.

In Ecuador, UN-Women will continue to support, through a scholarship programme, participants in the course on time use surveys.

The book Inclusive social protection in Latin America: a comprehensive, rights-based approach is being used by IASPN for a diploma course and webinar, as well as by UNICEF.

In Ecuador, the modules on the rights of older persons prepared for the international course ―Ageing and Social Protection Strategies in Latin America‖ will be used by the National Institute for Gerontological Research (INIGER) for training civil servants from the Ministry of Public Health and the Ministry of Economic and Social Inclusion.

The work on time use surveys and satellite accounts for unpaid work was submitted to the Andean Community, which is considering the implementation of such surveys at the regional level.

In Ecuador, the authorities of the Institute of Statistics have expressed interest in developing a face-to-face workshop on time use surveys.

In Uruguay, the Social Security Bank (BPS) not only continues to offer a diploma in social protection with the support of the ECLAC Financing for Development Division but is also planning to extend the diploma to other levels.

In Argentina, the teaching materials for the course on ageing are being used for training civil servants.


43

In the case of Paraguay, it has not been possible to verify whether the work with the Bureau for Older Persons (DAM) has been recovered by the current management. Nonetheless, a civil society organization is planning a publication on Paraguay‘s experience in the area of ageing, including the development of a plan of action, which will be used to get the candidates for the 2013 elections to commit to carrying out reforms to guarantee the rights of the elderly.

Finding 11: The data indicate a high level of sustainability of outputs and themes

developed in the framework of the cooperation programme 2010-2011. However, it does not appear that the divisions have reflected on or decided how to use, moving forward, the experience and the lessons learned from the technical assistance provided to the countries.

93. With regard to programme sustainability at ECLAC, two aspects have been considered. The first relates to how the divisions use the outputs, services and knowledge gained through implementation of the cooperation programme 2010-2011 in their regular work or specific programmes or projects. The second aspect has to do with the continuity of the themes themselves in the regular work of ECLAC. 94. In regard to the use of outputs, services and knowledge, interviews with staff from the divisions charged with implementing the cooperation programme 2010-2011 have made it possible to identify a number of cases where studies, books, databases and other outputs funded by the programme have been or will be recovered by the divisions for working in other contexts (such as conferences and working groups) and projects. For example, staff members of the Division for Gender Affairs and CELADE mentioned that the experience and the materials produced on the issue of care will be reused in the ECLAC publication Social Panorama of Latin America. The Division for Gender Affairs is also using the material produced for the distance course on time use surveys in order to continue offering courses in conjunction with the Latin American and Caribbean Institute for Economic and Social Planning (ILPES) of ECLAC. The Social Development Division has used the knowledge and materials on conditional transfer programmes prepared with SIDA support in a postgraduate course for public officials in the region organized by OAS. The Financing for Development Division signed an agreement with the Centre for Social Security Studies of Uruguay‘s Social Security Bank to continue offering a diploma in social protection for Uruguayan civil servants. 95. During the evaluation it was not possible to determine how the divisions have used the experiences and lessons learned with the technical assistance provided to the countries or how they will do so in the future. 96. Looking ahead, the results of the evaluation suggest a high level of sustainability of programme themes within ECLAC, as they are still part of its programme of work for the bienniums 2012-2013 and 2014-2015. For example, ―supporting country efforts to implement time-use surveys to enhance policy-making relating to the care economy‖ is included in the work programme for 2012-2013 as one of the areas of focus of the Division for Gender Affairs, and one of the issues of the Women and Development series will be devoted to the ―value of women‘s unpaid work‖. The work programme for 2012-2013 also refers to systems for funding social protection as one of the thematic areas of priority for the Financing for Development Division. Similarly, the issue of population ageing and its


44

impact on social protection systems, as well as the need to implement a rights-based approach to conditional transfer programmes, are explicitly identified in the aforementioned document. 97. As pointed out by the Chief of the Programme Planning and Operations Division, social protection remains a key issue for ECLAC. The establishment, in 2010, of the Community of Latin American and Caribbean States (CELAC ) as a mechanism for regional integration not only in economic terms but also at the social, political and cultural level based on the promotion of equity and social justice, as well as the interest in social protection shown by the Chinese authorities, represent opportunities for further work in this field.

4. Efficiency 98. According to the OECD definition, the efficiency of an intervention is a measure of how economically resources/inputs (funds, expertise, time, etc.) are converted to results.59 99. The evaluation focused on the following:

the extent to which the thematic divisions took advantage of their complementarity and synergies in implementing the programme, and

the extent to which ECLAC ensured efficient implementation of the programme. Finding 12: The data indicate that collaboration among ECLAC divisions during

programme implementation has been limited and ad hoc. 100. The cooperation programme for 2010-2011 was seen as a way to address the issue of social protection from different, complementary angles. Conditional transfers, population ageing, women‘s unpaid work and non-contributory pensions are closely interlinked with social protection and (as mentioned in the programme proposal) held opportunities for collaboration and synergies among ECLAC divisions.60

But interviews with division staff members and their counterparts in the reference countries revealed that collaboration across divisions was, for the most part, limited and ad-hoc, and that coordination on issues was wanting. The following factors explain the limited collaboration across divisions and the lack of synergies on themes.

59 Organization for Economic Cooperation and Development (OECD), Glossary of Key Terms in Evaluation and

Results Based Management, Paris. 60 Economic Commission for Latin America and the Caribbean (ECLAC)/ Swedish International Development

Cooperation Agency (SIDA), ―Social protection and social inclusion in Latin America and the Caribbean. ECLAC-SIDA Cooperation Programme 2010-2011. Project Proposal‖, 19 November 2009.

Collaboration among divisions:

The Social Development Division requested and received comments from the Division for Gender Affairs for its publication on conditional transfer programmes.

The Financing for Development Division participated in one of the sessions of the international course on ageing and social protection strategies in Latin America, organized by CELADE.

The Social Development Division and the Division for Gender Affairs have exchanged reports on activities throughout the programme.


45

The limited number of joint regional workshops: The proposed programme included regional seminars as the best way to create synergies among the themes, obtain an overview of the programme and submit conclusions and joint recommendations. However, only one joint regional workshop was held (in Ecuador), and its purpose was not to share conclusions and recommendations on the programme.

The design of the programme as four different, isolated projects: As mentioned above, despite the fact that the four topics could be addressed jointly, the very structure of the programme led to tackling each component in isolation because its separate logical frameworks, products and activities did not, at any time, make for common elements.

The dearth of exchange systems between focal points: There was only one meeting with SIDA where representatives from the four divisions had an opportunity to explain and discuss the work done. Except for this case, no evidence was found of other meetings or mechanisms which might have enabled focal points to share ideas on a regular basis.

Finding 13: The data suggest that the ECLAC divisions ensured efficient programme

implementation, drawing on the expertise of and existing relationships with the governments of the region and, in general, keeping on schedule.

101. The data suggest that the ECLAC divisions ensured efficient programme implementation. As for the use of resources available in the selected countries, the divisions turned to local experts to conduct the national studies. The academics recruited worked closely with public authorities and institutions, which contributed to the dissemination of studies at the governmental level. The regional and international seminars in Costa Rica and Ecuador were organized with government support; the seminars in Brazil and Chile, where the Social Development Division studies were presented, were organized jointly with other participating organizations. With respect to the implementation efficiency, a number of the programme outputs were also translated into English and thus made accessible to a wider audience. 102. As regards the schedule of activities, the respondents interviewed mentioned only two instances in which there were implementation delays: seminars for presenting case studies. Delays were attributed to problems with the availability of government authorities to participate in activities. Finding 14: Progress in implementing the recommendations set out in the previous

evaluation appears to be limited, owing to, among other things, the lack of ECLAC management response to the evaluation and the lack of a plan for putting the recommendations into practice.

103. The terms of reference for the current evaluation required a status report on implementation of the recommendations made in the framework of the evaluation of the previous phase of the SIDA cooperation programme (see the full list in annex VI). Gathering this information proved to be very difficult because ECLAC management did not respond to the evaluation and no plan of action was drawn up for putting the recommendations in practice. According to the ECLAC officers interviewed, there was no management response owing to staff changes and the limited resources available at the PPOD Programme


46

Planning and Evaluation Unit). In addition, given the general nature and applicability of the recommendations, it was felt that they could not be implemented separately but rather should be considered in the broader context of follow-up actions taken by ECLAC to implement the relevant recommendations of the oversight bodies (ACABQ, OIOS, JIU, BOA). 104. The report on the activities of ECLAC in 2010-2011 provides information on the improvements with regard to one of the areas of recommendations covered by the previous evaluation: managing for results. According to the report of activities: (i) training workshops on results-based management were held for ECLAC staff; (ii) the monitoring process and reporting procedures through the Integrated Monitoring and Documentation Information System (IMDIS) were improved; (iii) the semi-annual reviews of indicators and results through IMDIS were institutionalized; (iv) the use of follow-up tools such as assessment surveys and focal points was systematized; and (v) advice was provided to improve follow-up work in the substantive divisions. The interviews also revealed that, in the past two years, a working group was established to improve the dissemination of documents and that ECLAC is working to improve its website, communications and distance education. 105. On the basis of the above considerations, progress in implementing the recommendations made in the previous evaluation appears to be limited.


47

F. CONCLUSIONS

106. Most of the countries of Latin America and the Caribbean are classified as middle-income countries, but a significant proportion of the population continues to live in a situation of poverty or extreme poverty, discrimination and social exclusion. As a result (which is also a cause), the fundamental rights of these persons are systematically violated. The structural problems underlying this situation call for an approach that differs from the development model that dominated until the 1990s. Several actors in the region have put forward a redefinition of this model such that economic development, instead of being the axis of government action, is integrated into broader political action that combines economic situations with social ones in order to achieve sustainable development of society and citizen fulfilment. 107. Among these actors, ECLAC has stood out, among other reasons, because it promotes a rights-based approach to social and economic development. The work carried out by ECLAC, with the support of SIDA, in the area of social protection has been instrumental in promoting the principles and best practices of the Nordic model of social protection.61 108. The cooperation programme 2010-2011 has enabled ECLAC to better position itself in several countries of the region, particularly in those where reforming social protection systems were included in the government‘s agenda. As noted herein, the programme was effective in building decision-maker capacities and furthered the political debate on social protection in the region, but it was not as effective in promoting the use of best practices. Factors both internal and external to ECLAC seem to favour the sustainability of the outcomes achieved at the country level, although this area requires more attention on the part of ECLAC to ensure continuity in all actions undertaken. 109. Because of its regional scope, ECLAC will face a number of future challenges in ensuring that the outcomes of the cooperation programme 2010-2011 and, more generally, of its social protection agenda, are disseminated throughout its sphere of influence. The scarcity of development cooperation resources, which is particularly pronounced in Latin America and the Caribbean due to its relatively high income level compared with other regions of the world, will exert further pressure on ECLAC to ensure the strategic use of the limited resources available. To this end, results-based management and knowledge management are areas that will require particular attention.

The recommendations and lessons learned that are set out in the following sections are intended to support ECLAC activities in the area of social protection.

61 J.C. Peña, R. Zivy and F. Amariles, ―Evaluación del Proyecto Inclusión del ‗Modelo nórdico‘ europeo en el

debate sobre la reforma de los sistemas de protección social en América Latina y El Caribe. Informe final‖, 15 February 2010.


48

G. RECOMMENDATIONS

110. The following recommendations were developed on the basis of the data collected and analysed during the evaluation and have received feedback from ECLAC.

1. Recommendations for ECLAC Recommendation 1: ECLAC should continue to improve its programme monitoring

system 111. The evaluation identified the monitoring system as an area of improvement for ECLAC. More specifically, it found that the outcomes, indicators and the baselines set out in the logical frameworks were not sufficient for monitoring and evaluating the cooperation programme. In recent years, ECLAC has taken several steps to improve in this area. Developing effective and efficient monitoring systems is a long-term project that requires not only the development of systems, but also a change in organizational practices and culture. It is recommended, therefore, that ECLAC continue its efforts to improve its monitoring system, stressing both the development of systems and the promotion of an organizational culture that values results-based management. Some of the recommended actions for ECLAC to ensure effective monitoring and evaluation of its projects and programmes (operations) are as follows:

Identify short-, medium- and long-term expected results that are specific, measurable, attainable and relevant;

Develop specific, measurable, achievable, relevant and time-bound indicators, that allow ECLAC to collect more meaningful data and that reflect the results, so that they may be used to guide decision-making;

Complement the indicators on document downloads with additional information about the main user profiles for ECLAC;

Ensure that reports systematically provide information on the indicators selected during the design phase and that those reports reflect any changes that are made to the indicators at a later stage along with an explanation of the reason for the change;

Extend the monitoring period beyond the end of the programme or initiative in order to increase the chances of finding short- and medium-term results.

Recommendation 2: For future programmes and projects, ECLAC should explicitly

state the theory of change underpinning them 112. The evaluation highlighted the difficulty of analysing the long-term effectiveness of the programme because of the absence of an explicit theory of change stating the underlying foundations and assumptions linking the initiative to the desired effects. Short- and medium-term results were identified in terms of capacity-building and the promotion of policy debate. However, it was more difficult to establish to what extent and under what conditions these results represent progress towards the goal of improving the coverage and quality of social protection systems. It is therefore recommended that ECLAC develop a theory of


49

change for its future programmes and projects that establishes more clearly the steps needed to move from one result to another, the conditions in which this can take place and the priority stakeholders targeted by each intervention. The theory of change should include specific short-, medium- and long-term results. A well-articulated theory of change would provide the Commission with elements to draw upon when deciding which projects and activities to carry out and where to invest its resources. This also requires some reflection by ECLAC on its comparative advantage in the region, the areas in which it can contribute and how it can achieve this. Recommendation 3: ECLAC should continue to strengthen coordination and

collaboration between divisions 113. The evaluation demonstrated that cooperation and coordination between the divisions responsible for the implementation of the cooperation programme 2010-2011 was generally on an ad hoc basis. ECLAC has been strengthening coordination and collaboration between the divisions in recent years, but further improvements are needed. 114. For coordinated and collaborative programme work, it is recommended that:

Programmes be designed jointly by the divisions responsible for their implementation and this be reflected both in the development of a common logical framework for the programme and in the preparation of the outputs to be produced together;

Systems be set up so information can be shared between divisions, within the programme framework, for example, regular meetings between the project coordinators;

Formal spaces be created for exchange, such as regional meetings, as part of programme implementation, so that lessons learned can be identified and disseminated among ECLAC divisions, as well as outside the Commission; and

Communication be facilitated among researchers and consultants from the countries targeted by ECLAC, in order to increase the opportunities for synergy and coordination between countries and reduce duplication.

Recommendation 4: The Executive Secretary of ECLAC, via the Project Management

Unit, should ensure that all evaluations are accompanied by a response from management and an action plan

115. As previously mentioned, it was somewhat difficult to establish whether the recommendations from the previous evaluation had been implemented by ECLAC, and, if so, to what extent. It is thus recommended that the Executive Secretary of ECLAC:

Clarify the roles and responsibilities of the Project Management Unit and the Programme Planning and Evaluation Unit in terms of following up on the implementation of any recommendations arising from evaluations of completed programmes;

Ensure that all evaluations are followed by a response from management and an action plan. If any recommendations emanating from the evaluation of specific programmes are discussed within the broader context of ECLAC reform, the Project


50

Management Unit, in collaboration with the Programme Planning and Evaluation Unit, should ensure that this information is reflected in the plan to follow up the recommendations.

Recommendation 5: ECLAC should continue its efforts to incorporate a human rights-

based approach and gender considerations into its programmes and projects

116. A human rights-based approach and gender considerations were incorporated into the cooperation programme 2010-2011 to some extent. While some elements of these principles or perspectives (such as strengthening the capacities of duty bearers, access to information and transparency) were fully incorporated, other elements (such as strengthening the capacities of rights holders, participation and gender analysis) would have warranted greater attention from ECLAC. 117. More specifically, regarding the integration of the human rights-based approach, it is recommended that the Executive Secretary of ECLAC ensure that the divisions design, implement and monitor programmes and projects in accordance with the principles of capacity-building of duty bearers and rights holders, participation, access to information and transparency. Other United Nations agencies, for example, UNICEF and UN-Women, have developed guidelines for mainstreaming these principles, which the ECLAC divisions could use as a reference. 118. ECLAC is already taking action to mainstream the gender perspective. Therefore, it is recommended that the Executive Secretary, with the support of CELADE, continue these efforts, ensuring that, among other things, a gender analysis is conducted for all ECLAC programmes and projects, that their logical frameworks include gender-specific indicators and that data disaggregated by gender is gathered, used and presented in the Commission‘s publications.

2. Recommendation for SIDA and ECLAC

Recommendation 6: SIDA and ECLAC should capitalize on their long experience of

cooperation on social protection matters to disseminate the lessons learned in other Latin American and Caribbean countries, and in the case of SIDA, in other regions where it operates

119. Social protection is a multidimensional issue, encompassing several fundamental human rights in the economic and social sphere. Experience in international development cooperation has shown that the fulfilment of each right is linked to that of other human rights. Therefore, social protection lends itself to being used as a gateway to work on different aspects of social and economic development. 120. ECLAC and SIDA should capitalize on their long experience of cooperation on social protection to extract lessons learned and disseminate them, in the case of SIDA, to other actors in different regions and, in the case of ECLAC, to other actors in Latin America and the Caribbean where it works on the theme of social protection.


51

3. Recommendation for ECLAC and its donors

Recommendation 7: To ensure that the task is performed effectively, ECLAC donors

should play a greater role in monitoring the programmes they fund, supporting ECLAC in developing logical frameworks and providing feedback on the progress reports and final reports

121. Responsibility for monitoring a programme lies with both the donor and the executing organization. Therefore, donors should support ECLAC to ensure that the programmes they fund are monitored effectively. The evaluation showed that the design of logical frameworks and the reporting of results are still areas for improvement for ECLAC. Donors should provide more support to the Commission in order to develop logical frameworks that allow the correct identification of the different levels of outcomes and indicators for each level. They should also make it a requirement for reports to contain data on short- and possibly medium-term results. We also suggest that donors make sure that reports deal with more than the activities and outcomes of funded programmes. Moreover, there should allow for a delay between implementation of a programme and presentation of the report, so that short- and medium-term results can be identified.


52

H. LESSONS LEARNED

122. The evaluation was designed, among other things, to draw lessons from the cooperation programme for application in other contexts and programmes. The lessons set out below were drawn from the analysis of the data compiled or from our experience in preparing evaluations relating to social protection and public policy implications. 123. These lessons are intended for any organization whose work consists in conducting research with a view to contributing to public policy formulation, implementation, monitoring and evaluation. Lesson 1: In order to identify significant results, the period of time should be

sufficiently long to consider the cumulative effect of interventions designed to impact public policy

124. The evaluation of the 2010-2011 cooperation programme identified several short-term results in terms of capacity-building and to a lesser extent, medium-term results in terms of policy changes. The evaluation also considered the importance of taking into account factors related to time and the type of relationship (contributory rather than attributive) between activities concerning research and public policy change to determine the programme‘s effectiveness. As is widely recognized in the literature, policy change is the result of a long process and multiple actions by different actors, as well as being influenced by contextual factors. Therefore, the results of research activities and the relationship between these and changes in public policy could be identified and evaluated within a period of time longer than one biennium. Lesson 2: The effectiveness of research activities in promoting political debate is

greater if joint efforts are undertaken with other stakeholders 125. The evaluation found that programme activities have been most effective in terms of contributing to political debates when conducted as part of joint efforts with other actors. The opportunity to present studies and other products in forums, conferences and other events organized in conjunction with other actors also trying to influence public policy can not only reach a wider audience, but that same audience may also contain important actors and, above all, decision-makers. Lesson 3: The effectiveness and sustainability of research products are heightened

when the issues raised are part of the regular work of the organization 126. The evaluation demonstrated the importance of working on issues that are part of the regular duties of ECLAC. This does not mean that ECLAC should not work on new or innovative topics, but this must be done within a well-defined framework, with clear expectations and different strategies. Working on issues that are part of the main line of work helps ensure that products and reflections can be reused in other contexts and in other projects and can serve as a basis for other activities.


53

Lesson 4: The implementation of social protection initiatives should reflect the multidimensional nature of the issue

127. Social protection is a multidimensional issue that can be addressed from different angles, as in the case of the 2010-2011 cooperation programme, this, however, failed to have a multidimensional nature in its implementation. Although the multidimensional nature of social protection is an advantage because it can act on different human rights (health, education, gender equality, food and others) respect for which has not been guaranteed, it can become a disadvantage because it favours situations in which human rights are treated separately, which prevents a holistic view of the political action necessary to remedy the situation of infringement. Therefore, social protection initiatives require a holistic view of the problem. This has to be reflected also in the design and implementation of interrelated activities that are consistent with the various aspects of social protection. Lesson 5: Advocacy initiatives are more effective and sustainable when integrated

in the activities of rights-holders and duty-bearers alike. 128. The 2010-2011 cooperation programme focused on the influence of public policy, mainly through strengthening the capacity of duty bearers. As recognized by the same agencies of the United Nations, development cooperation interventions must build the capacities of duty-bearers and rights-holders so that the balance of power between these two groups can change in a sustainable way. Consequently, interventions related to advocacy work must recognize the respective roles of duty-bearers and rights-holders in the formulation, implementation, monitoring and evaluation of public policies and to evaluate the respective needs and develop strategies so that groups can participate, directly or indirectly, in the processes of change proposed by the interventions.


55

ANNEXES

ANNEX I Findings

ANNEX II Recommendations

ANNEX III Evaluation matrix

ANNEX IV List of persons interviewed

ANNEX V

List of recommendations arising from the evaluation of the cooperation programme between ECLAC and the Swedish International Development Cooperation Agency (SIDA) 2007-2009

ANNEX VI Profile of persons surveyed


56

Annex I Findings

FINDINGS Finding 1: The data collected suggest that the cooperation programme was relevant or very

relevant in the countries where it was implemented as it was consistent with national needs and priorities and the comparative advantage of ECLAC in the region.

Finding 2: The objectives of the cooperation programme 2010-2011 are relevant to the work of the ECLAC divisions.

Finding 3: Although the geographical priorities of Swedish cooperation have changed, social protection remains an important issue for the ASDI and also a means to work on other priorities. The cooperation programme 2010-2011 and, more generally, the experience that ECLAC has accumulated on social protection issues are therefore very relevant to the Swedish agency‘s cooperation priorities.

Finding 4: The data collected suggest that the programme has been effective in strengthening the capacities of the different actors in the region whose work is related to social protection, providing, among others, knowledge, data and tools for policy analysis.

Finding 5: The data suggest that the programme has helped to foster political debate at the regional level on the issues of ageing, women‘s unpaid work and comprehensive social protection. No evidence was found that the programme contributed to policy discussions on the funding of health systems and non-contributory pension schemes.

Finding 6: The promotion of best practices related to the development, implementation, follow-up and evaluation of social protection policies is one of the aspects of the programme that the respondents valued the most. During the data collection process, two examples were found of how such practices are being implemented in Costa Rica and Ecuador. And there is evidence of greater awareness at the individual level.

Finding 7: The level of integration of gender issues and the human rights-based approach in the programme has been modest. However, it should be noted that ECLAC cooperation with SIDA during 2010-2011 attached greater importance to gender issues linked to the issue of social protection during the programme development stage, when a component on women‘s unpaid work was being developed.

Finding 8: Assessing programme effectiveness in terms of short-, medium- and long- term results was hampered by several factors related to logical frameworks, the information reported to SIDA, the follow-up system and the programme‘s theory of change.

Finding 9: The cooperation programme 2010-2011 helped to better position ECLAC in the countries of Latin America and build staff capacities at the division level.

Finding 10: The data provide evidence of the continuity of the impacts of some of the outputs and services provided in a number of countries under the cooperation programme 2010-2011.

Finding 11: The data indicate a high level of sustainability of outputs and themes developed in the framework of the cooperation programme 2010-2011. However, it does not appear that the divisions have reflected on or decided how to use, moving forward,


57

the experience and the lessons learned from the technical assistance provided to the countries.

Finding 12: The data indicate that collaboration among ECLAC divisions during programme implementation has been limited and ad-hoc.

Finding 13: The data suggest that the ECLAC divisions ensured efficient programme implementation, drawing on the expertise of and existing relationships with the governments of the region and, in general, keeping on schedule.

Finding 14: Progress in implementing the recommendations set out in the previous evaluation appears to be limited, owing to, among other things, the lack of ECLAC management response to the evaluation and the lack of a plan for putting the recommendations into practice.


58

Annex II Recommendations

RECOMMENDATIONS

Recommendation 1: ECLAC should continue to improve its programme monitoring system Recommendation 2: For future programmes and projects, ECLAC should explicitly state the

theory of change underpinning them Recommendation 3: ECLAC should continue to strengthen coordination and collaboration between

divisions Recommendation 4: The Executive Secretary of ECLAC, via the Project Management Unit, should

ensure that all evaluations are accompanied by a response from management and an action plan

Recommendation 5: ECLAC should continue its efforts to incorporate a human rights-based

approach and gender considerations into its programmes and projects Recommendation 6: SIDA and ECLAC should capitalize on their long experience of cooperation

on social protection matters to disseminate the lessons learned in other Latin American and Caribbean countries, and in the case of SIDA, in other regions where it operates

Recommendation 7: To ensure that the task is performed effectively, ECLAC donors should play

a greater role in monitoring the programmes they fund, supporting ECLAC in developing logical frameworks and providing feedback on the progress reports and final reports


59

Annex III Evaluation matrix

EVALUATION MATRIX

Questions Indicators Data sources Data collection

methods

Relevance

What was the relevance of the ECLAC-SIDA cooperation programme for:

current and potential beneficiaries or users of the social protection systems in the selected countries?

the governments of the selected countries?

the counterparts of the programme in the selected countries?

the thematic divisions in charge of project implementation?

SIDA?

Level of consistency between the needs and interests of current and potential beneficiaries or users of the social protection systems in the selected countries (presented in studies and statistics at national, regional and international level) and programme objectives. Level of consistency between programme objectives and the priorities of the selected countries referred to in official documents. Perceptions of respondents (the counterparts from the selected countries, ECLAC and similar organizations) with respect to the relevance of the cooperation programme. Level of consistency between the programme objectives and the objectives of ECLAC or the thematic divisions established in their workplan. Examples of synergies between the programme and work carried out by the thematic divisions.

Studies and statistics at national, regional or international level and available online. Cooperation programme project proposal. Cooperation programme results framework. Official documents from the selected countries (for example, national development plans, sectoral plans relating to the area of social protection). Respondents (ECLAC, SIDA, counterparts from the selected countries, similar organizations).

Telephone interviews, communication via Skype and videoconference. Documents analysis. Internet search.

Effectiveness

To what extent have the expected outcomes of the ECLAC-SIDA cooperation programme 2010-2011 been achieved in terms of

References to ECLAC studies or publications in reports, minutes of meetings, official documents, policymakers‘

Respondents (ECLAC, counterparts in the countries). Reports to SIDA. Programme results

Interviews Documents analysis Survey


60


methods

:

fostering political debate?

strengthening capacities?

promoting the application of best practices?

speeches in the selected countries. Reports by counterparts on the application of ECLAC studies and publications in the formulation, implementation, monitoring and evaluation of social protection policies. Changes in social protection policies in the selected countries which counterparts perceive are associated with the programme. Changes reported by counterparts, for example, in:

- knowledge on specific matters relating to social protection systems (ageing, pensions, health and others)

- capacities to integrate gender issues into social protection policies

- capacities to identify the most vulnerable groups in terms of access to social protection systems

- capacities to ensure that resources are allocated for formulating social protection policies

- capacities to establish contact with stakeholders who work in the area of social protection

- capacities to formulate more inclusive social protection policies

Number of counterparts interviewed or surveyed who have used ECLAC studies which bring together best practices. Stakeholders‘ level of satisfaction (counterparts from selected countries and participants in activities)

framework. Reports, minutes of meetings, official documents, policymakers‘ speeches.


61


methods

with the different products and services received

How and to what extent were gender and human rights considerations incorporated into the ECLAC-SIDA cooperation programme 2010-2011?

Number of counterparts from the selected countries who have received support from ECLAC to integrate gender considerations into the formulation, implementation, monitoring and evaluation of policies. Number of counterparts from the selected countries who have received support from ECLAC to integrate human rights considerations into the formulation, implementation, monitoring and evaluation of policies. Number of counterparts surveyed who found that the technical assistance received has been useful for adapting or implementing policies or existing systems to the needs of the most vulnerable groups (such as women, children, older persons and youth at risk). Number of counterparts in the countries who have participated in conducting studies published by ECLAC.

Respondents (ECLAC, counterparts in the countries). Reports to SIDA. Workplans.

Interviews Documents analysis Survey

Efficiency

To what extent have the thematic divisions used complementarity and synergies, if they exist, to execute a programme?

Thematic division resources (for example, knowledge, experience, partnerships, procedures and instruments) shared with other divisions for programme execution. Opportunities during programme execution (for example, for seminars and workshops) which were taken up by different thematic divisions. Opportunities during programme execution (for

ECLAC (thematic divisions). Counterparts in the selected countries. Reports to SIDA.

Interviews Documents analysis


62


methods

example, for seminars and workshops) which could have been taken up, but were not, by different thematic divisions, as identified by respondents. Thematic division resources (for example, knowledge, experience, partnerships, procedures and instruments) which have not been shared with other divisions for programme execution, as identified by respondents. Duplication of activities or products by the different thematic divisions. Exchange of lessons learned, best or promising practices during programme execution.

To what extent has ECLAC ensured efficient programme execution?

Execution of activities as scheduled in the calendar or workplan. Reports from counterparts in the selected countries on ECLAC use of resources (knowledge, partnerships and others) which are available in the countries and the region for programme execution. Reports from thematic division teams on the use by ECLAC of resources (knowledge, partnerships and others) which are available in the countries and the region for programme execution. Comments by the counterparts in the selected countries with respect to the capacity within ECLAC to provide support requested on time. Number of recommendations fully or partially implemented.

ECLAC (thematic divisions) Counterparts in the selected countries. Reports to SIDA. Cooperation programme 2008-2009 evaluation report.



63


methods

Sustainability

To what extent are the results achieved in the selected countries sustainable after the cooperation programme?

Evidence (in documents or expressed by the respondents) that the counterparts used products (for example, publications, knowledge, recommendations, training manuals and partnerships) or services created by the programme. Level of counterpart participation (information, consultation, joint responsibility, management) in the different phases of the programme (design, implementation, monitoring and evaluation). Commitments (written or verbal) made by the counterparts in the selected countries to provide (human or financial) resources to monitor the results achieved. Commitments (written or verbal) made by ECLAC or other national, regional or international organizations to provide (human or financial) resources to monitor the results achieved in the selected countries. Types of mechanisms or strategies formulated to promote the sustainability of programme results (for example, partnerships or alliances between counterparts and other national, regional and international stakeholders). Factors, identified by respondents, which hinder the sustainability of results.

Counterparts in the selected countries. ECLAC (thematic divisions). Reports to SIDA. Documents in which the counterparts commit to investing (human and financial) resources. Counterpart organizations.


To what extent will the results obtained by the programme contribute to

Evidence (in documents) of the divisions‘ use of products (for example,

ECLAC (thematic divisions). Funding proposals.



64


methods

the work of ECLAC?

publications, knowledge, recommendations, training manuals and partnerships) or services provided by the programme. Number of proposals presented by the divisions which aim to gain resources to continue working on programme issues. Reports by staff in the divisions on the programme‘s influence on their work.

ECLAC documents which show the use of programme products.

Recommendations

What steps can be taken to improve the level of collaboration between the thematic divisions of ECLAC?

Previous analysis

Are there opportunities to continue collaboration with SIDA?

Previous analysis

Are there opportunities for ECLAC to continue working in the cooperation programme 2010-2011 areas?

Previous analysis


65

Annex IV List of persons interviewed

LIST OF PERSONS INTERVIEWED

Name Organization Category of respondent

Date of the interview

Costa Rica

Emiliana Rivera National Council for Older Persons (CONAPAM)

Government 17 October

Rose Mary Ruiz Bravo Member of the board of directors of the Joint Institute for Social Aid (IMAS), Ministry of Social Welfare


Mayra Díaz Méndez Director of IMAS, Ministry of Social Welfare

Government 5 November

Ivonne María Ruiz Barquero

Costa Rican Social Security Fund (CCSS), Financial Manager

Public institution 29 October

Jacqueline Castillo Co-author (case study of Costa Rica), Head of the Department of Statistics, CCSS

Public institution 26 October

Adolfo Rodríguez Herrera

Co-author (case study of Costa Rica) and Technical Secretary in the Office of the President

Academia/ government

19 October

Pablo Sauma Author (case study of Costa Rica) Academia 18 October

Juliana Martínez Social Research Institute, University of Costa Rica; researcher at the United Nations Research Institute for Social Development (UNRISD)

Academia 25 October

Carlos Sojo Research associate and former Director of the Latin American Faculty of Social Science (FLACSO)

Academia 2 November

Ecuador

Alba Pérez Author (case study of Ecuador) and member of the Transition Commission of the Women‘s Council and Gender Equality

Academia/ government

20 October

Nydia Pesantez United Nations Entity for Gender Equality and the Empowerment of Women (UN-Women)

Counterpart organization

24 October

Eduardo Cabezas Central Bank of Ecuador Public institution 5 November

Noralma Ordóñez National Institute of Statistics and Geography (INEGI)

Public institution Answers sent by e-mail

Pabel Muñoz Assistant Secretary-General for Democratization of the State, National Secretariat of Planning and Development (SENPLADES) (former Deputy Minister, Ministry of Economic and



66

Social Inclusion)

Mauricio Léon Deputy Minister, Ministry for Economic Policy Coordination (former Deputy Minister, Ministry for the Coordination of Social Development)


Paraguay

Jorge Quintás Former National Director for Older Persons, Institute for Social Welfare


Emigdio César Palacios Cáceres

Director-General for State Media of the Department for Information and Communication for the Development of Paraguay, former Director of Communication and Focal Point for the Inter-American Committee on Social Development (CIDES) in the Department of Social Action


Economic Commission for Latin America and the Caribbean (ECLAC)

Raúl García-Buchaca Programme Planning and Operations Division (PPOD)

24 October

Irene Barquero PPOD 25 October

Laetitia Montero PPOD 23 October

Simone Cecchini, Rodrigo Martínez

Social Development Division 18 October

Daniel Titelman, Pablo Carvallo, Olga Lucía Acosta

Financing for Development Division 22 October

Sonia Montaño Division for Gender Affairs 15 November

Dirk Jaspers and Sandra Huenchuan

Latin American and Caribbean Demographic Centre (CELADE) - Population Division of ECLAC

19 October

Rudolf Buitelaar Latin American and Caribbean Institute for Economic and Social Planning (ILPES)

6 November

Uruguay

Eduardo Méndez and Gabriel Lagomarsino

Banco de Previsión Social Public institution 6 November

General

Fabián Repetto Center for the Implementation of Public Policies Promoting Equity and Growth (CIPPEC)

Academia 15 October

Torsten Wetterblad Swedish International Development Cooperation Agency (SIDA)

Donor 19 October

Nathalie Brisson Lamaute

Former ECLAC staff member 22 October

Francisco Pilotti Inter-American Social Protection Network (IASPN) of the Organization of American States (OAS)

Counterpart organization

24 October

Ferdinando Regalia Inter-American Development Bank (IDB) Counterpart organization

23 October


67

Annex V

List of recommendations arising from the evaluation of the cooperation programme between ECLAC and the Swedish International Development Cooperation Agency (SIDA) 2007-2009

List of Recommendations Arising from the Evaluation of the Cooperation Programme between ECLAC and the

Swedish International Development Cooperation Agency (SIDA) 2007-2009

1. ECLAC should take measures to preserve the knowledge it has accumulated on the Nordic model in the course of implementing the ECLAC-SIDA technical cooperation programme and related earlier projects and should make such knowledge easily accessible. The agenda for ECLAC-SIDA collaboration should include the creation of a searchable website or database of Nordic best practices which are adaptable to the realities of both Latin America as a whole and its different countries, and this should be tackled without delay.

2. The objectives, vision and current overall logframe of the ECLAC-SIDA programme should be maintained as a long-term guide for mutual cooperation between SIDA and ECLAC. Shorter- and medium-term programmes and projects should adopt a more pragmatic and realistic logframe using the SMART indicators.

3. ECLAC should consider introducing a reasonable degree of methodological uniformity into the design of logframes for cooperation programmes with major donors and should consider creating a searchable database (in the ProTrack application or elsewhere) of such logframes to assist in their design.

4. Consideration should be given to concluding agreements for a three-year term of implementation when the complexity of the envisaged deliverables and the projected amount of the workload involved so justify.

5. On-the-spot surveys of participants at workshops and seminars should be enhanced by preparing more detailed questions aimed at obtaining more diverse and thorough evidence of the outcomes. Such evidence should be used for learning lessons in the course of implementation so as to improve the delivery of subsequent workshops and also enhance final reporting to the donor. These should be the main tools for management by project coordinators.

6. (i) Statistics on downloads of outputs should always be included in reporting on programme/project performance; (ii) all ECLAC outputs should be equipped with a pre-download questionnaire and an invitation to register to receive e-mail or Web-feed updates on news in the thematic subject area; (iii) downloads should be followed up after an appropriate time lag with a request to rate and comment on the downloaded publication.


68

7. In the future, and for selected, especially important, workshops, ECLAC should organize Internet-based discussions of studies to be presented at the workshop in order to foster workshop interactivity and productivity.

8. On-the-spot, post-workshop surveys should be conducted using e-mail, e-survey or other Internet-based tools available at ECLAC. Such exit surveys should be mandatory for the organizers of all ECLAC workshops and seminars. The qualitative components of beneficiaries‘ responses should be properly processed, analysed and summarized, and the findings should be used to further improve the modalities of delivering capacity-building outputs.

9. In the future, ECLAC should organize virtual communication facilities (forums, blogs, communities of practitioners, etc.) on the themes of the ECLAC-SIDA programme, using seminar and workshop participants as the nucleus for such virtual communities. ECLAC should use these communities to, among others, disseminate news on forthcoming events and new publications in relevant subject areas.

10. (i) To enhance the impact of its publications, ECLAC must be proactive; a passive stance produces no impact. ECLAC should promote publications through various physical and virtual channels and should work with academia in organizing discussions at which diverse views and positions are represented. ECLAC could be more proactive in disseminating its work, in particular through electronic alerts sent to stakeholders, thematic electronic mailing lists and the development of virtual forums and chats. The elements of a strategy aimed at greater visibility and exposure of outputs, including publications, should include wider broadcasting of programme outputs through thematically relevant websites of international organizations, academia and non-governmental organizations, and enhanced visibility of ECLAC outputs by posting them —or providing hyperlinks to them— on such websites.

(ii) The formulation and implementation of activities aimed at enhancing the visibility of technical cooperation programmes and activities should be more systematically explored and planned among the thematic divisions in charge of project implementation, the Programme Planning and Operations Division (PPOD) and the Information Services Unit of ECLAC. Relevant activities could include press communiqués, press conferences, new media initiatives and social networking, among others.

(iii) Securing more systematic involvement of the ECLAC Public Information and Web Services Section in assessing the visibility strategy of technical cooperation programmes early on in the process and defining the needs, priorities and ways forward might be a practical way of addressing the issue in earnest.

11. (i) ECLAC should establish a clear and unambiguous division of labour between the substantive personnel tasked with coordinating the delivery of technical cooperation projects and the administrative and programme management staff who provide the necessary support for such implementation. This division of labour should be meaningful and equitable.

(ii) The role of donor agency representatives present at ECLAC and the formal coordination mechanisms with them should also be clearly defined from the outset of the programme, in order to maximize the advantage of this presence in administrative, management and substantive terms.


69

(iii) The division of labour mentioned in (i) above should be reflected in the checklist, which has the status of a programme document and should be observed by all means. The checklist should reflect the requirement to use, to the extent possible, all the performance-enhancing and evidence-collection tools recommended earlier in this report: tools to gather information on users and electronic alerts for registered users about new thematic possibilities; pre-workshop electronic discussion facilities and post-workshop virtual networking spaces; e-surveys of workshop participants both on-the-spot and post-delivery, etc.

(iv) Training in the use of these tools should be provided to all concerned and brainstorming should be conducted whenever necessary on how to best organize team work in application of these tools. Coordination meetings should be held more regularly between DPPO and thematic divisions.

(v) Building on the progress achieved in the area of programme performance monitoring through the ProTrack and IMDIS systems, management should take steps to use both instruments to monitor how programmes/projects avail themselves of the above-mentioned electronic tools for collecting evidence from the outset. Particular emphasis should be placed on reporting on results and indicators, so that they can be used as practical tools to ensure that project management is kept on track. In the early stages of programme implementation, all stakeholders involved (PPOD, thematic divisions and donor representatives) should agree on a common understanding of coordination processes and of the detailed content of deliverables. This could be done when establishing the workplan of a programme and its different components.

(vi) The evaluation culture of ECLAC should be further enhanced, for instance through broad dissemination of evaluation findings and collaborative assessment of the implementation of recommendations at all levels.

12. (i) PPOD should ensure that the ProTrack platform captures, on a continuous basis, all the evidence collected from exit evaluation surveys and follow-up e-surveys completed by seminar and workshop participants, as well as from e-surveys of publication readerships. The results should be systematically processed and included in the progress and final reports presented to senior management and donors. The lessons learned from these surveys should be systematically fed back into future planning and into improving the modalities of programme delivery. Such evidence should be continuously accumulated over the course of the programme life cycle and the progress in accumulating such evidence should be continuously monitored.

(ii) PPOD should establish a meaningful template for final reports on programme performance, including clear and decisive criteria on the evidence the report should include with regard to results, outcomes and impact. The preparation of such a template could be carried out in coordination with donor representatives, or at least taking into account their requirements. The report should systematically include a summary description of the overall accomplishments of the programme as a whole, based on the indicators of achievement data collected during and near the end of project implementation, as well as an annex containing such evidence as number of publication downloads, press communiqués and other visibility practices, etc. PPOD senior staff should not sign off on any final report before they have properly certified that all requirements of the established template have been duly observed, so as to ensure the quality of the final report to donors.


70

13. (i) ECLAC should proactively explore the opportunity to provide policy advice in Colombia along the thematic areas of the ECLAC-SIDA programme, which is a crucial factor of the programme‘s sustainability; (ii) Serious consideration should be given to the idea of conducting workshops on topics at the departmental and municipal level in Colombia, in partnership with the United Nations Development Programme (UNDP), when relevant in the framework of the second phase of the ECLAC-SIDA programme to be implemented in 2010-2011; (iii) ECLAC should engage more systematically as a neutral broker in organizing small, focused technical policy discussion groups that gather key stakeholders from government institutions, academia and other relevant counterparts to discuss policy issues in an open manner and from various perspectives; (iv) Potential partnerships with other United Nations institutions (such as UNDP) should be explored more systematically at the national level to share knowledge and carry out technical cooperation activities in beneficiary countries; (v) The sustainability of the knowledge, achievements and results emerging from technical cooperation programmes should be systematically assessed and the opportunity to use them in future technical programmes should be systematically explored.

14. Suggestions made by beneficiaries regarding technical cooperation follow-up should be compiled, analysed and factored into the fund-raising strategy of ECLAC and into ongoing and future activities and workplans of the thematic divisions financed with regular budget and extra-budgetary resources.

15. To prevent the knowledge accumulated during the implementation of the ECLAC-SIDA programme from fading away and to ensure its sustainability, ECLAC should create knowledge management networks (in the broad sense of the term) covering the programme‘s thematic areas, to serve as a source for other cooperation programme initiatives.


71

Annex VI Profile of persons surveyed

PROFILE OF PERSONS SURVEYED

BREAKDOWN BY CATEGORY (Percentages)

Source: Prepared by author.