pacap impact and lessons 2005-2010 september 2010 · pacap impact and lessons 2005-2010 ......

PACAP

Impact and Lessons 2005-2010

September 2010

PACAP

PACAP – Activity Completion Report i

Document:

Philippines Australia Community Assistance Program

Impact and lessons 2005-2010

Version: FINAL

Program: PACAP

Client: AusAID

Contractor: GHD

Authors David Swete Kelly

Paul Crawford

Date: September 2010

This document is the property of the Philippines Australia Community Assistance Program.

It is permissible to copy and use any of the material in this report provided that the source is appropriately acknowledged. Further information is available from:

David Swete Kelly PACAP Program Director 23rd Floor, Prestige Tower, F. Ortigas Avenue, Ortigas Center, Pasig City, Metro Manila, Philippines www.pacap.org.ph Tel: +63 (2) 6317549 Fax: +63 (2) 6319654

© PACAP 2010

PACAP Basic Activity Data

PACAP – Activity Completion Report ii

BASIC ACTIVITY DATA

Country: The Republic of the Philippines

Modality: Program

Donor:

Delivery Organisation:

Implementing Contractor:

Key Dates:

Approval: December 2004

Mobilisation: 24 January 2005

Mid-Term Review: 30 September 2007

Extension: 24 January 2010 to 30 June 2011

Independent Completion Report: May 2010

Activity Completion Report: August 2010

Approved Budget:

AUD 29,832,898.65

Grant Funds AUD 21,510,128.14 (included in Approved Budget)

Target Provinces:

Responsive Assistance Scheme: National

Focal Community Assistance Scheme: Agusan del Sur

Bohol Misamis Occidental

Northern Samar Surigao del Norte

ACKNOWLEDGEMENTS

The PACAP Secretariat and GHD management would like to thank AusAID and NEDA for the creative partnership developed over the past five years. This has supported the team and inspired success. PACAP is also indebted to its various Governance Committee members at both the national and provincial level for their dedicated voluntary service and commitment. These groups have guided PACAP‘s engagement and significantly improved the quality of deliverables. PACAP, through its funding and its processes, has striven to assist local communities, but it is always the passion of the proponent organisations (our implementing partners) that make or break any intervention. Huge credit thus goes to the hundreds of partner organisations – NGOs, POs, LGUs, and NGAs – that have supported the delivery of community interventions, for it is your groups that represent the true sustainability of PACAP. Lastly to our communities – PACAP has striven to work with you to achieve your aspirations. This report reflects the significant successes and impacts achieved in improving the lives of poor Filipinos. We thank you for the privilege of working with you over the past five years.

PACAP Acknowledgements

PACAP – Activity Completion Report iii

Funding Window No. of Grant Agreements Grant Amount FOCAS Mindanao 122 248m

Agusan del Sur 37 81m Misamis Occidental 44 87m Surigao del Norte 41 80m

FOCAS Visayas 70 180m Bohol 35 95m Northern Samar 35 85m

Responsive Assistance Scheme (RAS) 296 363m Luzon 66 76m

Visayas 115 130m

Mindanao (non ARMM) 81 97m Mindanao (ARMM) 34 60m

Total 488 791m

Project Segregation by Island regions and ARMM.

Luzon- 66 projects - Php76 M Visayas- 185 projects - Php310 M Mindanao- 237 projects - Php405 M ARMM - 34 projects - Php60 M

PACAP Lessons Learnt

PACAP – Activity Completion Report iv

LESSONS LEARNT

1. For a program such as PACAP, ‗impact‘ is best understood in terms of the ‗magnitude‘ of the changes effected, the ‗timeframe‘ taken for the intended changes to be effected, and the ‗breadth‘ of focus within the target population with regard to beneficiaries. ............................................................................. 4 2. A program such as PACAP can foster a great variety of measurable impacts in the lives of beneficiaries within relatively short timeframes. ............ 5 3. PACAP impacts that resonated most with people related to their personal transformations rather than to generic community-level changes. ................... 8

4. A program such as PACAP benefits from an appropriate and meaningful way to guide the setting of its funding priorities. .............................................. 9 5. Beneficiary targeting, driven by PACAP‘s proponent organisations, produced significant improvements in the circumstances of poor households. Few significant benefits were observed among the ‗absolute poor‘. .............. 10 6. There would almost certainly have been value in PACAP having a mechanism to support the on-going development and funding of proven or ‗high potential‘ projects. ................................................................................. 11 7. A program such as PACAP can make significant contributions to a range of development cross-cutting themes; specifically gender, environment, disabled persons and peace building. ............................................................ 12

8. A program such as PACAP can foster significant and tangible changes in gender equity within diverse projects, whether they are specifically gender-focussed or otherwise. ................................................................................... 12

9. PACAP could have done more to address the spread of HIV/AIDS and other infectious diseases, as well as to strengthen communities in disaster preparedness. ................................................................................................ 15 10. RAS was able to identify and support a range of capable CSO partners to address local priorities in an efficient and low risk way. ............................. 15 11. RAS provided AusAID/NEDA with an efficient mechanism to engage with, and learn about, emerging development issues and priorities, without the design overheads associated with larger dedicated programs. ..................... 15 12. FOCAS is a less efficient modality for achieving beneficiary impact than RAS, but on the other hand has shown promise as a model for fostering sustainable changes in the way that local development priorities are conceived and implemented. ......................................................................... 18

13. FOCAS shows sufficient promise as a model for promoting local democratic governance in the Philippines, that it warrants further experimentation and study. ............................................................................ 20 14. FOCAS contributed to significant improvements in the organisational capacity of partner CSOs. An investment of 668 capacity building workshops facilitated for 46 partner CSOs within the context of a supportive and collaborative network were able to foster statistically significant changes in six dimensions of organisational capacity. .......................................................... 20

PACAP Table of Contents

PACAP – Activity Completion Report v

TABLE OF CONTENTS

Basic Activity Data ........................................................................................... ii Acknowledgements .......................................................................................... ii Lessons Learnt ............................................................................................... iv

Table of Contents ............................................................................................. v Table of Figures .............................................................................................. vi List of Acronyms ............................................................................................ vii 1. Introduction ............................................................................................... 1

1.1 Document Purpose ................................................................................. 1

1.2 Background .............................................................................................. 1 1.3 Overview ................................................................................................... 2

2. Performance ............................................................................................. 1 2.1 Overall Assessment ................................................................................ 1 2.2 Assessment of the Theory of Change ................................................. 1

2.3 Impact........................................................................................................ 4

2.4 Cross-cutting themes ............................................................................ 12 2.5 RAS Outcomes ...................................................................................... 15

2.6 FOCAS Outcomes ................................................................................ 18

Appendix A: Organisational Chart ..................................................................... I Appendix B: Revised Logframe ...................................................................... III Appendix C: Proponent Organisation Capacity ............................................. VII Appendix D: M&E Critique ............................................................................. XI

PACAP Table of Figures

PACAP – Activity Completion Report vi

TABLE OF FIGURES

Figure 1: PACAP projects map showing engagements between 2005 and 2010....................................................................................................................... 1

Figure 2: PACAP Theory of Change ........................................................................ 1

Figure 3: Key differences between RAS and FOCAS ........................................... 2

Figure 4: PACAP beneficiary types .......................................................................... 5 Figure 5: PACAP project types ................................................................................. 5

Figure 6: Changes in household impact indicators for PACAP in Bohol ............ 6

Figure 7: Relative household income of PACAP beneficiaries Vs non-beneficiaries for Bohol overall as well as three FOCAS engagements (100% means no difference) ............................................................................. 6

Figure 8: Percentage of PACAP Projects contributing to MDGs ......................... 6

Figure 9: Relationship between loan repayment rate and increasing loan size................................................................................................................................. 7

Figure 10: Development Risks ................................................................................ 11

Figure 11: Project alignment with Philippines Harmonised GAD Guidelines .. 13

Figure 12: Reported sources of risk for RAS projects ......................................... 16 Figure 13: Significant changes in proponent organisational capacity ............... 21

PACAP List of Acronyms

PACAP – Activity Completion Report vii

LIST OF ACRONYMS

ACR Activity Completion Report

AFA Area Focussed Approach

ARMM Autonomous Region of Muslim Mindanao

AusAID Australian Agency for International Development

CPS Country Program Strategy

CSO Civil Society Organisation

DBG Demand for Better Governance

EoI Expression of Interest

FMC FOCAS Management Committee

FOCAS Focal Community Assistance Scheme

GAD Gender and Development

GoA Government of Australia

HAI Hassall & Associates International

LGC Local Government Code

LGU Local Government Unit

M&E Monitoring and Evaluation

MDG Millennium Development Goal

MEIS Monitoring and Evaluation Information System

MSC Most Significant Change

MTPDP Medium Term Philippines Development Plan

MTR Midterm Review

NEDA National Economic Development Agency

NGA National Government Agency

NGO Non-Government Organisation

NSCB National Statistical Coordination Board

ODA Official Development Assistance

ODE Office of Development Effectiveness

PAC PACAP Advisory Committee

PACAP Philippines Australia Community Assistance Program

PCR Project Completion Report

PDC Provincial Development Council

PSC Provincial Stakeholders Committee

RAS Responsive Assistance Scheme

RFT Request for Tender

SGF Strategic Guidance Framework

SSM Soft Systems Methodology

PACAP Introduction

PACAP – Activity Completion Report 1

1. INTRODUCTION

1.1 Document Purpose

This document presents the broad achievements of the Philippines Australia Community Assistance Program (PACAP), an Australian bilateral aid program funded by the Australian Agency for International Development (AusAID) and implemented in partnership with the National Economic Development Authority (NEDA) of the Government of the Philippines. This report covers the period of program management contracted to GHD: 24 January 2005 to 30 September 2010

1.

1.2 Background

PACAP has been part of Australia‘s official bilateral development assistance (ODA) to the Philippines since 1986, during which time it has provided small grants assistance to poor communities and groups throughout the country in partnership with a wide range of development actors (Figure 1). Since 1986, PACAP has provided approximately 2,145 grants in partnership with an estimated 1,250 proponent organisations, benefiting approximately 1,460,000 people

2 in poor communities

throughout the Philippines.

Figure 1: PACAP projects map showing engagements between 2005 and 2010.

1 The initial contract (24 January 2005 to 23 January 2010) was extended up to 30 September 2010 to accommodate

a new design phase. 2 1,460,103 estimated.

PACAP Introduction


PACAP commenced at a time of major social and political upheaval in the Philippines, and from relatively small beginnings subsequently evolved in tandem with both the development of civil society within the country, and the evolution of ideas regarding the role that international donors can play in community development. It was the impact of the latter, however, that eventually led AusAID to engage in an extensive process of review and redesign (this taking place between March 2003 and August 2004). This culminated in a request for tender (RFT) being issued in August 2004. Hassall & Associates International (HAI)

3 won the tender, and the revised

program mobilised on 24 January 2005.

The new program design not only heeded the lessons of the previous years, and strengthened the program in a number of areas highlighted by the review team, but aligned the program more closely with AusAID‘s then-current Country Program Strategy (CPS), as well as the Philippines Government‘s Medium-term Philippines Development Plan (MTPDP). The new design also sought to strengthen cooperation between local government units (LGU) and civil society organisations (CSO) by focusing one of its two funding mechanisms on five target provinces in the southern Philippines.

The program was implemented by a Manila-based secretariat; along with five provincial offices in Bohol, Northern Samar, Surigao del Norte, Misamis Occidental and Agusan del Sur (see Appendix A for PACAP‘s organisational structure).

1.3 Overview

Following recommendations in the midterm review (MTR) of PACAP in 2007, a participatory workshop revised the program goal and objectives to reflect a stronger ‗results orientation‘ (see Appendix B for revised PACAP logframe).

The revised PACAP Goal was:

Poor communities in the Philippines are empowered to pursue economic growth and achieve better standards of living.

This goal has been pursued through two schemes:

Focused Community Assistance Scheme (FOCAS): involved the facilitation of partner networks in the five target provinces

4. In each

province, local stakeholders identified three to five strategic areas/themes. Each of these FOCAS areas/themes was managed by a local FOCAS Management Committee (FMC). The FMC developed a strategy and a portfolio of linked projects typically implemented by local Non-Government Organisations (NGOs) in partnership with local and National Government Agencies (NGAs). In addition, PACAP contracted established local NGOs (‗FOCAS Secretariats‘) to provide each FMC with administrative support. Such support included assistance with the preparation, appraisal, and documentation of projects, as well as project monitoring and review. All projects developed by the FMCs were required to be endorsed by a Provincial Stakeholder Committee (PSC)

5

prior to approval by the national PACAP Advisory Committee (PAC). FOCAS projects were generally implemented over a three-year period, and were funded up to Php1.5m per year. The FOCAS objective was: ―Devolved networks of civil society, government and private sector organisations are collaborating to identify and meet local priorities in selected provinces, and creating an enabling environment for further development‖.

3 Acquired by GHD in 2008.

4 These Provinces were jointly selected by the Governments of the Philippines (GoP) and Australia (GoA),) and

included Northern Samar and Bohol (in the Visayas); along with Agusan del Sur, Surigao del Norte and Misamis Occidental (in Mindanao). 5 The Provincial Stakeholders Committees were led by the provincial governments, but also included members of civil

society, academe and the private sector. In most cases they reflected the Provincial Development Council (PDC) mandated under the Philippine Local Government Code (LGC).

PACAP Introduction


Responsive Activity Scheme (RAS): enabled PACAP to engage widely and flexibly in response to emerging priorities, principally (but not solely) in the southern Philippines

6. Typically, RAS partners were established

NGOs with proven track records. These were engaged to implement discrete projects over 1 to 1.5 years. RAS projects were funded up to Php1.5m per year. The RAS objective was: ―Civil society organisations are innovating and responding to community needs throughout the Philippines and generating a foundation for new development approaches‖.

The ‗theory of change‘ that underpins the program is discussed in Section 2.2 and depicted in Figure 2.

All PACAP projects were appraised and approved by the PACAP Advisory Committee. This was comprised of representatives from AusAID, NEDA, the NGO sector, local government and academe. The PAC was also responsible for annual revisions of the Strategic Guidance Framework (SGF). The SGF aimed to set the program‘s priorities by guiding its engagement into areas of emerging concern to stakeholders.

In its recent phase (Jan 2005 to September 2010) PACAP has managed the delivery of 488 grant agreements worth in excess of $21.5m (see Figure 1). These projects facilitated poverty-focussed improvements in 60 of the Philippines‘ 81 provinces, benefiting about 332,000 beneficiaries

7, 61% of whom were women. Moreover, these

interventions involved 281 civil society partner organisations.

66

Twenty percent of RAS funds (10% of total PACAP funds) were allocated to projects in the north of the country. 7 Estimated 332,181.

PACAP Performance


2. PERFORMANCE

2.1 Overall Assessment

This report was prepared between April and June 2010 by the PACAP Secretariat, drawing principally on M&E data collected by the team, but also upon data provided by a series of independent studies, as well as a number of secondary sources. The aim was to provide a succinct, balanced and informative critique of the program‘s design, performance and context.

The overall finding was that the program was implemented in accordance with the design, with evidence that most of the work was of good quality. A key strength of the program was the establishment of constructive development partnerships that included capacity development opportunities for both NGOs and LGUs. There was ample evidence that these partnerships contributed to a range of positive local impacts. Nonetheless, weaknesses were found in some areas of the program. There were also a number of design shortcomings. It needs to be noted though that the two PACAP modalities (FOCAS and RAS) were in many ways experimental, and that reflection concerning these has provided both AusAID and NEDA with valuable strategic lessons. These findings are elaborated in the following sections.

2.2 Assessment of the Theory of Change

Figure 2: PACAP Theory of Change

The rationale for the current phase of PACAP‘s design was predicated upon those ideas of the liberal democratic tradition indicating that civil society can mediate

facilitates

community-

based

development

relationships

(Community

Engagement)

If the PACAP

secretariat...so that partners...

then ultimately

poor

communities in

the Philippines

will be...

appraises, funds

and monitors

partner projects

(Grant

Administration)

analyses and

learns about the

drivers of

change, and

causes of failure

(M&E)

throughout the

Philippines can

innovate and

respond to

community needs

(RAS)

in five target

provinces can

build

collaborative

networks

involving civil

society,

government and

the private sector

to meet local

priorities and

create an

enabling

environment for

further

development

(FOCAS)

empowered to

pursue

economic

growth and

achieve better

standards of

livingstrengthens the

capacity of

partners

(Capacity

Buillding)

PACAP Performance


between the state and its citizens to both generate good governance and to support local development

8. As depicted in Figure 2, PACAP‘s ‗theory of change‘ argued that

the provision of a range of functional support and funding to CSOs would enable them to facilitate the empowerment of poor communities throughout the Philippines.

This broad theory of change is consistent with current thinking, and - generally speaking - is accepted as a truism. Nevertheless, the details and modalities to effect this theory are singularly vague in the literature. For instance, there are innumerable ways that donors can support CSOs, but what are the best ways?

9 Likewise, any

number of factors can have a positive impact on a region‘s development, but which have the maximal sustainable impact? Furthermore, all development agencies need to face the question: is the strengthening of civil society a means to an end, or an end in itself?

PACAP progressed knowledge in relation to all of these, and to other questions as well.

The coexistence of RAS and FOCAS within PACAP enabled AusAID to test two different approaches / modalities for bilateral assistance to CSOs. Some differences included:

RAS… FOCAS…

..tended to work with proven/capable NGOs ..tended to work in less developed provinces with locally relevant CSOs, many of whom were less experienced and required capacity building

..worked through discrete partnership arrangements with individual NGOs

..supported the evolution of local collaborative networks

..pursued localised development impacts ..fostered broader impacts within geographic or thematic boundaries

..pursued short-term specified impacts ..impacts were longer-term and emergent

..impacts were firmly concerned with the ultimate beneficiaries of the projects (i.e. the CSO partnerships were a means to an end)

..was as much concerned with the strengthening of the CSO partners and networks as with beneficiary impacts (i.e. the CSO partnerships were to some extent an end in themselves)

..sought high-profile development impacts to affirm the bilateral relationship

..tended to abstract the role of the donor and to foster sustainable local capacity and empowerment

Figure 3: Key differences between RAS and FOCAS

The RAS engagement resulted from PACAP‘s long history as a small grants scheme, at a time when embassies and bilateral aid agencies more generally offered such schemes largely as an end in themselves. Small grants schemes were then a common means of affirming bilateral relationships, while engendering a positive public profile, and delivering meaningful local level impacts. However, with the emergence of the ‗development effectiveness‘ agenda more than a decade ago, several subsequent reviews of PACAP questioned the development purpose of such a small grants program, while also suggesting ways in which to maximise its impact. Many reviewers considered that the bilateral/public profile and local level impacts were insufficient justification in themselves for such a scheme‘s existence, and that the notion of ―effectiveness‖ specifies a more direct contribution to broader development outcomes. A direct consequence of this thinking then was that PACAP had, by 2003, integrated a new ‗Area Focussed Approach‘ (AFA) to foster more sustainable (albeit sub-national) impacts by strategically linking projects within geographic boundaries, and also by delegating portfolio management responsibilities to local CSOs. In the most recent phase of PACAP, the FOCAS modality has been, in point of fact, an extension of the thinking that underpinned the AFA. The difference

8 Mercer, C. (2002) ―NGOs, civil society and democratization: a critical review of the literature‖, Progress in

Development Studies 2, 1 (2002) pp 5-22. 9 AusAID‘s Office of Development Effectiveness (ODE) was tasked with carrying out research during 2009 to seek

answers to this question.

PACAP Performance


being that with AFA, PACAP created portfolios of discrete projects within a locality, while with FOCAS, PACAP has extended the process to the provincial level. By doing this it has engaged with the underlying dynamics, institutional factors, strategic agenda and local networks of a given Province, with the long-term view of laying the foundations for further development beyond PACAP assistance.

At first glance, the development paradigms underlying RAS and FOCAS are somewhat at variance, although looked at in another way they can be seen as complementary:

The design of RAS acknowledged the perspective that pursuing localised development impacts through initiatives implemented by passionate and capable people is a legitimate enterprise in itself. Put simply, doing ‗good things for little people‘ is its own end.

The design of FOCAS acknowledged the perspective that development initiatives should be catalytic, and contribute to systemic and lasting changes that extend beyond immediate project impacts. Put simply, development should be ‗more than the sum of its parts‘

10.

While both of these paradigms are defensible, they serve two different strategic purposes for AusAID/NEDA:

Beneficiary impact: RAS is a proven modality to efficiently impact on a maximum number of beneficiaries across a wide geographic area. It also minimises transaction costs and has relatively low risk.

Sustainable change: FOCAS, although a less tested modality, has shown promise as a way to sustainably strengthen local CSO and LGU collaboration and capacity, particularly in the formulation and implementation of local development priorities.

The broader perspective of Australia‘s Department for Foreign Affairs and Trade is likely to support the former paradigm. For many years PACAP‘s RAS has provided a robust vehicle for affirming the bilateral partnership and enhancing the Australian Government‘s profile throughout the Philippines. On the other hand, the current thinking manifested within most international development agendas leans towards the concepts of sustainability and systemic change implicit in the latter paradigm. The relative merits of these two paradigms are reflected in the analyses in the following sections of this report.

Yet the question for AusAID as it considers future engagement is whether the agency should:

accommodate the continuing co-existence of both paradigms;

consolidate investment in one or the other paradigm; or

push the envelope further by seeking national level impacts not currently delivered by either paradigm.

Much of the debate concerning the merit of PACAP can be framed in the light of these three options. Within AusAID, this debate was born out of a tension between the prevalent perception of the limited nature of sub-national impacts, and the ambition for national macro changes. PACAP‘s MTR posed the somewhat Socratic question: ―how much PACAP is too much PACAP?”, implying an alignment with the second of the above paradigms, and more than hinting that PACAP projects should ‗add up‘ to at least some value at the national level

11. By contrast, proponents of the

first paradigm have argued that there is inherent value in sub-national engagements

10

PACAP‘s design purposefully chose sub-national (provincial) engagement as the most appropriate entry point to craft partnerships of civil and government players that can sustainably deliver community development. 11

i.e. PACAP should end when either: a) it achieves the desired macro-level changes; or b) is deemed unlikely to achieve the desired macro-level changes.

PACAP Performance


that foster localised impacts; that these are of sufficient merit and need not achieve a broader ambition

12.

The question remains: should a program such as PACAP be required to achieve macro impacts, or is it pragmatic and defensible to effect local impacts only? The experience from this phase of the program has clearly shown that a modality such as RAS is an effective way to achieve the latter. A modality such as FOCAS, on the other hand, has demonstrated promising results as a mechanism to effect catalytic changes. Nevertheless, the impact of these changes has remained at the sub-national level. For better or worse then, it was never in PACAP‘s design to allow for scalable macro impacts, and although the experience of this current phase clearly demonstrates that the program can scale-up relationships and processes, its design is such that the scaling-up of the projects themselves can only, at best, be limited.

2.3 Impact

Lesson

1. For a program such as PACAP, ‘impact’ is best understood in terms of the ‘magnitude’ of the changes effected, the ‘timeframe’ taken for the intended changes to be effected, and the ‘breadth’ of focus within the target population with regard to beneficiaries.

The PACAP M&E Framework defined ‗impact‘ as ―significant and lasting changes in the circumstances of PACAP’s ultimate beneficiaries‖, the word ‗significant‘ referring to the extent of changes to which PACAP projects have plausibly contributed; the word ‗lasting‘ referring to the likelihood that benefits will endure, as well as their potential for replication and/or scaling-up.

There are at least three dimensions to the concept of ‗significant change‘:

Magnitude: the scale of the changes effected;

Timeframe: the amount of time for an intervention to effect the intended changes; and

Breadth: whether benefits are focussed on a narrow or a broad base of beneficiaries.

It was an intention of PACAP‘s design that different projects would generate different impacts with regard to these dimensions. For example, a community water supply project in Burgos (on Siargao Island in Surigao del Norte Province) reduced the incidence of water-borne disease from 21 cases per month to zero (‗magnitude’), by increasing access to potable water that was available immediately at project completion (‗timeframe’), for virtually all households in the community (‗breadth’)

13.

By contrast, the Loboc Youth Band Project14

in Bohol enabled 98 high school students (‗breadth’), to access scholarships that will generate educational and vocational impacts (‗magnitude’), to be realised following their future graduation/employment (‗timeframe’).

12

i.e. PACAP should continue indefinitely, or at least until there is an ‗acceptable‘ level of poverty within the Philippines. However it must be recognised that a program of limited resources and scope is unlikely to effect significant macro-level changes in a country such as the Philippines. Nevertheless, assisting CSOs to address localised priorities is a defensible rationale in its own right. 13

The project benefitted a total of 1,671 individuals. 14

For the first time in Loboc‘s music history, girls have become members of the Loboc Band. Of the total 495 students, there are 262 girls and 233 boys, (comprising 53% and 47% respectively of total students). Furthermore, the large number of girls in the band generated considerable awareness and interest in gender issues in the local population, especially amongst the women. To date, 98 students of the Loboc Music Project have received scholarships to attend Universities and Colleges as a result of the musical skill developed in the School Band. Holy Name University has accepted 30 students, University of Bohol 31 students, CVSCAFT 21 students, and BIT International College 16 students. Furthermore, these scholarship saved the students and their parents an average of P8,000 per semester or P16,000 annually. This amounts to about P1,568,000.00 annually for all parents.

PACAP Performance


Furthermore, PACAP has supported such a great diversity of both project and beneficiary types (see Figure 4 and Figure 5) that a single/universal definition of impact, meaningful for the whole program has been impossible. Rather, each project (or cluster of projects) targeted a specific beneficiary class through particular approaches/sectors, thereby generating project-specific impacts.

Figure 4: PACAP beneficiary types

Figure 5: PACAP project types

Lesson

2. A program such as PACAP can foster a great variety of measurable impacts in the lives of beneficiaries within relatively short timeframes.

The diversity of the PACAP portfolio may suggest that its resources were not only spread far and wide, but thinly. Yet the evidence of the majority of the individual projects is that the program has indeed contributed to the broad changes in the social and economic empowerment of the targeted poor communities anticipated by the revised goal (stated in Section 1.3). Yet, unfortunately, a more quantitative

58%

50%

26%

25%

22%

16%

10%

8%

6%

5%

5%

5%

2%

Smallscale Farmers

Rural Poor (in General)

Upland Communities

Artesinal Fisherfolk

Indigenous Groups

Lowland Rural Communities

OSY

Exclusively women

Children at risk

Disaster/IDPs

Differently Abled

Urban poor

Rehabilitation

Percentage of PACAP Engagements

Equipment 3%

Information 4%

Build networks 6%

Formal Training 7%

Infrastructure 8%

Informal training

10% Microfinance

13%

Other 21%

Social suport 28%

PACAP Performance


assessment of community level impact in Bohol through the use of a third-party household-level longitudinal survey turned out to be of little value (see Figure 6 and Figure 7). This shows that while general depravation levels (Figure 6) for all communities decreased over the five years there was no statistical significance discernable between PACAP beneficiaries and non-beneficiaries. Similarly for household income (Figure 7) the data was again too inconsistent to detect discernable changes. There was significant inconsistency in how the survey tool was applied across the years

15, as a result of which, any statistically relevant evidence

relating to PACAP‘s impact was masked within the large overall variability of the data. Nevertheless, other reported trends reveal that most communities that received PACAP support improved significantly over the five years.

Year Status Numbers surveyed

Deprivation (20

Indicators)

Child Malnutrition

Below Food

Threshold

Un- employment

2009 Beneficiary 42 19% 3% 23% 14%

Non- beneficiary

238 21% 5% 21% 13%

2004 Beneficiary 31 26% 12% 64% 15%

Non- beneficiary

220 27% 16% 65% 16%

Figure 6: Changes in household impact indicators for PACAP in Bohol

Figure 7: Relative household income of PACAP beneficiaries Vs non-beneficiaries for Bohol overall as well as three FOCAS engagements (100% means no difference)

Analysis of the Project Completion Reports (PCRs) required from all concluding PACAP projects, revealed that the majority (82%) of projects contributed to Millennium Development Goal (MDG) 1, i.e. they contributed to the eradication of poverty (see Figure 8).

Figure 8: Percentage of PACAP Projects contributing to MDGs

15

See Annex 3: PACAP planned to analyse a subset of Bohol provincial government poverty data to ascertain broad changes in poverty among PACAP beneficiary households (using both with/without and before/after datasets). Some useful insights were gained from this dataset, but in general the use of secondary data was fraught with difficulty, principally because PACAP was unable to control the integrity of the data captured, or the timeliness of its analysis. Pilot testing of all secondary data is needed to ensure that it is collected at a sufficient level of granularity and with enough consistency to ensure statistical analysis is possible.

0%

50%

100%

150%

Overall EnterpriseDevelopment

Eco-CulturalTourism

SustainableAgriculture

2004

2009

82%

70%

67%

18%

12%

8%

8%

4.9%

MDG 1 - Poverty

MDG 7 - Environment

MDG 3 - Gender

MDG 8 - Develop a Global Partnership

MDG 2 - Primary Education

MDG 4 - Child Mortality

MDG 5 - Maternal Health

MDG 6 - HIV/AIDS

PACAP Performance


Comprehensive insights into the economic, social and environmental impacts fostered by PACAP have been supplied by the PCR data. These impacts include the following:

o Economic impacts: Since 2005, PACAP has supported 306 projects enhancing social enterprises (including a large number of engagements that integrated agricultural production, social credit and enterprise development). For example: a sample of 99 of PACAP‘s 189 agricultural projects resulted in an increase in average per unit return of 252%. This translates into an average increase in monthly household income of PhP2,541 (from PhP3,296 before project commencement to PhP5,837 upon completion). A broadly similar trend was evident from a sample of enterprise development projects. Overall, these projects reported a median net increase in monthly income of 100%. This translates into an average increase in enterprise profit of 135%, out of which dividends were paid at an average of almost PhP12,000 per member per year

16. Of the 189 enterprise development projects, 88%

reported that they had achieved a balanced or positive cash flow by the final six months of the project.

Micro-credit provided by PACAP also proved to be an important contributor to economic growth in target areas. Over 14,000 people

17 benefitted from

PACAP‘s 134 microfinance projects. Loans were generally of PhP5,000, with a median repayment rate of 87%. These loans generated a reported average capital build-up of PhP16,000 per 100,000 PhP exposure. Interestingly, the data shows that those institutions that provided loans of between PhP5,000 and PhP7,500 (US$100 to US$150) per beneficiary, attained consistently higher repayment rates than those institutions that made either smaller or larger loans (see Figure 9). Also, on average, men tended to borrow just over twice as much as women (>PhP10,000 Vs PhP5,500 respectively), while also being much less likely to repay their loans within the loan period. (69% Vs >84% respectively).

Figure 9: Relationship between loan repayment rate and increasing loan size.

o Social impacts: PACAP projects benefitted a range of vulnerable groups. Examples include: indigenous groups (86 projects), disabled persons (19 projects), out-of-school youth (40 projects), and children at risk (23 projects). In addition, PACAP implemented a portfolio of 39 projects in conflict-affected areas.

16

Actual figure PhP11,883 17

Actual figure 14,221 people

0

50

100

150

200

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55

Repayment Rate % Loan Amount ('00 PhP)

PACAP Performance


Reportedly, 59% of all projects assisted people to determine and understand their rights, 77% strengthened the role of local civil society, and 71% improved people‘s access to information. Moreover, around 38% of all projects contributed to improved local service delivery, in particular improved water and sanitation (23%), health (13%) and education (13%). PACAP‘s water and sanitation projects increased access to potable water for over 14,000 households; this decreasing the average hours per day spent collecting water by 2.7 hours, or 73%. Almost 1,800 households were likewise provided with sanitary toilets, the reported subsequent decrease in the incidence of water borne diseases being 83%. Furthermore, PACAP projects provided training in health care for 9,258 people, and increased the number of community health workers by 1,140 people. As a consequence, approximately 34,000 men and 33,000 women can now better access primary health care, whilst to date, 5,027 men and 3,726 women have directly benefited from various forms of specialised health care. Finally, just over 14,500 children and adults (59% female) have benefitted from literacy and/or primary education. In this regard, PACAP worked particularly with remote and disadvantaged peoples (including people with handicaps) to build basic literacy and numeracy skills. Note that not only do these skills improve overall wellbeing, but they have allowed almost 8% of these people to find paid employment.

o Environmental impacts: 273 PACAP projects have contributed to MDG 7, the goal that pertains to environmental sustainability. 266 of these projects involved conservation-related activities, during the implementation of which 4.06 million trees were planted, and 138 local environmental ordinances were facilitated. These projects also resulted in either the formation or reestablishment of 122 community-based environmental protection groups. In terms of scale-up, 254 of these projects made a contribution to provincial or national initiatives.

Lesson

3. PACAP impacts that resonated most with people related to their personal transformations rather than to generic community-level changes.

Yet beyond the foregoing quantitative evidence of impact, PACAP‘s M&E arrangements also captured a range of qualitative impact information, including over 144 Most Significant Change (MSC)

18 stories. In all of the final three ‗winning‘ stories

(i.e. the three stories selected as being indicative of the ‗most significant changes‘ resulting from the activities of FOCAS in Visayas, FOCAS in Mindanao and RAS) profound personal changes were experienced by the individual/s concerned

19. By

way of example, the overall winning story is shown in the box on the following page.

This suggests that, despite the undoubted importance of the broad-based empirical impacts fostered by PACAP (as reported above), the changes that actually resonated most with people were those that involved their personal transformations.

18

The MSC Technique is a narrative method that involves the collection and selection of ‗winning‘ stories of significant change encountered by program beneficiaries. http://www.mande.co.uk/docs/MSCGuide.pdf 19

The three final winning stories all concern the social and economic empowerment of individual women: i) a physically disabled woman (FOCAS Mindanao); ii) a woman involved in home industry (FOCAS Visayas); iii) female participants in functional literacy classes (RAS).

http://www.mande.co.uk/docs/MSCGuide.pdf

PACAP Performance


Winning MSC Story: PACAP‘s overall ‗winning‘ MSC story was an example of a high magnitude, short timeframe, narrowly focussed impact.

Reported by: Roland Ordona Month: March 2009 Date submitted: April 2009 Told by: Teotina Pelenio When: March 11, 2009 Where: Brgy. Pisaan, San Francisco, Agusan del Sur Who involved: Members of SAFRAADAP

What happened: Teotima is a member of San Francisco Association of Differently-abled Persons Multi-purpose Cooperative. ―My one leg was paralysed after I had a vehicular accident. My being paralytic made me feel discriminated and I lost every opportunity for me to generate income and help my family needs. When I joined SAFRAADAP and become one of the beneficiaries of the PACAP funded project, I now earn 1,800.00- 2,000.00 a week. This helps me send my children to school and I feels more confident and motivated.‖

As shared by Roland Ordona (Manager of SAFRAADAP):,:, ―When SAFRAADAP did the full operation of their PACAP funded enterprise and generate high income (more than their milestone target income of 750,000.00 PhP), the community gave high respect and the PWD members no longer feel that they are discriminated. The PWDs changed their perspectives and feelings (as before they feel like beggars). The LGU do not consider them as clients but as a significant partner sector.‖

Lesson

4. A program such as PACAP benefits from an appropriate and meaningful way to guide the setting of its funding priorities.

At the heart of PACAP‘s pursuit of development impact was the program‘s Strategic Guidance Framework (SGF), the intention of which was to progressively refine the focus of the program, by ensuring that it remained responsive to emerging priorities. Changes to the SGF were reviewed annually by the PAC, and approved by AusAID/NEDA. However, in retrospect, it was not a particularly effective device for several reasons:

Timeframe: it gradually became apparent that an annual review of the SGF was too frequent in practice, due primarily to the time-lag associated with developing, implementing and reviewing new projects. The risk being that without the time needed for genuine action-learning cycles, the SGF could have contributed to a loss of focus within the program. Hence this was an issue that needed careful management.

Ownership: in charging the PAC with the responsibility for the annual crafting of the SGF, the agenda became subject to a broad range of interests. To ensure a tighter focus, it would be better for the agenda to be defined by the owners of the program (AusAID and NEDA) and then presented to the PAC for comment.

Relevance: all the priorities eventually incorporated into the SGF were inevitably broad, the reason being that they needed to accommodate virtually any reasonable development assistance request. Yet this raises the question of the relevance of the SGF in responsiveness to specific emerging priorities.

PACAP Performance


Hence, in retrospect, it may have been more effective for the design team and AusAID to select a specific development focus for the program from the outset, rather than delegating this to a rolling function. Arguably, the weaknesses of the SGF as a guiding mechanism exposed the program to criticisms regarding a lack of coherence in its undertakings, particularly from those stakeholders who were aligned with the ‗second paradigm‘ discussed in Section 2.2.

Yet despite the disappointments with the SGF, any future phase of PACAP or similar programs, must also tackle the practicalities of how best to guide engagements. This is necessary to ensure that PACAP always develops relevant and grounded interventions, while at the same time providing AusAID/NEDA with critical insights concerning emerging priorities.

Lesson

5. Beneficiary targeting, driven by PACAP’s proponent organisations, produced significant improvements in the circumstances of poor households. Few significant benefits were observed among the ‘absolute poor’.

Prioritisation was also an issue with regard to beneficiary targeting. Because PACAP was a ‗bottom-up‘ community assistance program, the submitted projects were largely prioritised by the proponent organisations

20. Hence, beyond the broad

parameters set by the SGF, PACAP‘s control over the selection of the ultimate beneficiaries of the program was very limited - their selection was already implicit in the projects by the time they were submitted by proponents for approval. From a much broader perspective though, the selection of beneficiaries was addressed at a macro-level during the program design stage through AusAID‘s ‗policy-level‘ selection of the country‘s poorest provinces and communities, and also by requiring that all projects approved by the PAC be poverty-focussed.

Nevertheless, the MTR advanced the opinion that PACAP tended to benefit the ‗mid-poor‘ rather than the ‗poorest-of-the-poor‘, this topic having emerged relatively recently as the subject of vigorous academic and policy debate. Certainly in its design, PACAP was positioned to work with the poor and only the poor, but not, on the other hand, with the ‗absolutely destitute‘. The ‗absolutely destitute‘ are important, but the pervasive feeling is that these people are best helped through social protection schemes managed through national governments supported where necessary by development agencies.

That said, the evidence does show that PACAP targeted the poor and has significantly contributed to the alleviation of household poverty. The household data collected in 2004 (see Figure 7) prior to the beginning of PACAP‘s most recent phase, clearly reveals that those beneficiary households subsequently selected by PACAP had incomes at the time that were equal to the community average. Furthermore, this average was particularly low, as the communities selected were all 4

th to 6

th class, and hence noted for a high incidence of poverty. It is clear then that

(at least for Bohol where this data was collected) PACAP was firmly targeted at the mainstream poor. Moreover, more recent data shows that by 2009 two thirds (66%) of PACAP beneficiary households had incomes that were above the community average. This data then confirmed that PACAP had been effectively targeting both poor communities, and the poor households within these poor communities; and that it had noticeably contributed to a reduction in their poverty.

20

N.B This approach can be contrasted with true ‗community-driven‘ approaches which emerge through protracted processes of local engagement directly with partner communities.

PACAP Performance


Lesson

6. There would almost certainly have been value in PACAP having a mechanism to support the on-going development and funding of proven or ‘high potential’ projects.

The issues of prioritisation related to the SGF and beneficiary targeting, were not the only shortcomings to become apparent as PACAP evolved. It also became evident that the Secretariat was too limited in its capacity to nurture and scale-up the impact of successful projects. The program had no institutional arrangements to select and extend or expand projects that were found to be particularly successful, or that were likely to have a more significant beneficiary impact if scaled-up (particularly in the case of RAS). While PACAP did maintain and enhance support to effective partners this was only partially effective in scaling up and sharing innovations and improved development approaches.

The PACAP M&E arrangements tracked factors likely to erode beneficiary impact (‗Development Risks‘) using a modification of the MSC technique. A total of ninety-nine stories of significant risk were collected and categorised against the ‗STEEP framework‘ in order to identify trends. From these it was found that the major source of risk reducing the impact of PACAP projects was related to the technical issues surrounding project implementation (37.6%), followed closely by economic constraints (37.1%). Yet considering the difficulty of the socio-environmental milieu in the Philippines, it is interesting that the smallest percentage risk proved to be environmental/ecological (26.8%). Undoubtedly though, this reflects the predominantly social development nature of the PACAP portfolio.

Figure 10: Development Risks

When the overall PACAP development risks were disaggregated by sub-program it became apparent that the proponents implementing FOCAS projects considered them almost twice as risky as did the proponents implementing RAS projects (47% FOCAS compared to 27% RAS). Yet this finding clearly reflects the fact that FOCAS proponent partners tended to be less experienced and technically capable than RAS partners.

13.7% 18.0% 11.2% 12.2% 13.7%

19.0% 19.5%

25.9% 14.6%

17.1%

67.3% 62.4% 62.9% 73.2% 69.3%

Social cultural Technical Economic(Resource

Constraints)

Environmental Relational/Political

Yes More or Less No

PACAP Performance


Yet despite the challenges PACAP encountered due to the aforementioned weaknesses in its design, and the risks associated with its implementation, there has been ample evidence that the program has made a significant contribution to the lives of individuals, of households, and of communities as a whole. The overarching relevance of a program such as PACAP was borne out in the findings of the National Statistical Coordination Board (NSCB) released in March 2008. This confirmed that the incidence of poverty in the Philippines had actually increased from 24.4% in 2003 to 26.9% in 2006 - a national increase of 654,610 families in poverty. These ‗macro‘ statistics provide a contextual backdrop that justifies the continuance of community development as a relevant element of the Philippines-Australia partnership.

2.4 Cross-cutting themes

Lesson

7. A program such as PACAP can make significant contributions to a range of development cross-cutting themes; specifically gender, environment, disabled persons and peace building.

The diversity of the PACAP portfolio has meant that it was in a position to contribute significantly to a range of cross-cutting themes such as gender equality, the environment, disabled persons, promoting peace, and others.

Gender

Lesson

8. A program such as PACAP can foster significant and tangible changes in gender equity within diverse projects, whether they are specifically gender-focussed or otherwise.

PACAP projects overwhelmingly benefited women relative to men. A planned total of 302,500 beneficiaries proved in practice to be 332,000 beneficiaries, of which 201,000 (60.6%) were women. Proponent organisations reported that 8% of projects exclusively benefited women, and that 67% contributed to MDG 3 (the goal pertaining to gender equality and the empowerment of women). Moreover, PACAP was aligned with the Philippines Harmonised Gender and Development (GAD) Guidelines. When it was assessed against the five criteria

21 of these guidelines, the program was rated

as ―gender responsive‖ overall. Of PACAP‘s projects (see Figure 11), 70% ―improved women’s basic welfare relative to men‖. Similarly, more than 77% of projects improved ―women’s access to factors of production (land, labour, capital, and services)‖, while 69.5% created ―new awareness of culturally determined gender roles‖. In 90% of projects, women ―participated equally with men in the planning and implementation‖, while in 81% they had ―equal control over the distribution of project benefits‖.

There was also ample evidence of a direct flow of project benefits to women: 67% of loan recipients in micro-finance projects were women; 48% of water users‘ association members were women; 59% of basic literacy beneficiaries were women; 49% of primary healthcare recipients were women, and; 53% of health training recipients (4,880) were women. Female membership of the agricultural organisations involved in the implementation of PACAP projects also increased by 234% (almost four times the relative increase of men at 59%). The program also provided specialist

21

(welfare, access, ―conscientization‖, participation and control).

PACAP Performance


health care for 3,726 women, and supported the posting of 979 female health workers within target communities.

Figure 11: Project alignment with Philippines Harmonised GAD Guidelines

The strong emphasis on gender equity by PACAP was broadly appreciated by both proponent organisations and beneficiaries, with 88% of the projects ‗exclusively benefiting women‘ being considered a success (c.f. 83% for all projects). The types of projects most likely to ‗improve women‘s basic welfare‘ were projects that predominantly involved social credit and/or agricultural enterprise. By contrast, social services projects were less effective in improving women‘s basic welfare.

Yet beyond the empirical evidence of PACAP‘s contribution to gender equity, the PACAP Coordinator reported qualitative evidence of improved gender equity in a thesis written as part of a Master of Arts in Humanities, majoring in Women‘s Studies. This research involved an in-depth case study

[1] of the ways in which a PACAP-

supported women-centred microfinance project had impacted on the women‘s families, communities, economy and culture. The project focussed on traditional handloom weaving in the indigenous Islamic villages of Amito and Dayawan, in Marawi City, in Mindanao. The research found that the project had assisted in the revival of a centuries‘ old traditional industry, essentially an art-form, with consequent improvements in the economic welfare of not just the women, but of their families as a whole. This economic improvement was mainly brought about through the provision of improved access to factors of production such as credit, training, marketing and group organising. In some cases, the women were, as a result, able in turn to finance their husbands‘ livelihoods (e.g. hollow-block production and fishing). The handloom weaving project also fostered a range of intangible benefits that may be broadly labelled ‗empowerment‘, including: women travelling alone, women leading their own businesses, women involved in their own procurement, and women controlling their own finances. Women‘s empowerment was also furthered when the Dayawan Women‘s Organization was able to successfully lobby for specific ordinances within Marawi City. The changes wrought broadly improved the social status of women, within families, within communities - and more profoundly still - within themselves.

[1]

Padilla, L. M. P. (2009) The Handloom Weaving Enhancement Project of the Women in Barangays Amito and Dayawan, Marawi City: An analytical case study of a women-centred economic project, Master of Arts Thesis, St. Scholastsica‘s College, Manila.

70.0% 77.7%

69.5%

89.7% 81.3%

25.6% 18.3%

25.1%

8.9% 16.7%

4.4% 4.0% 5.4% 1.5% 2.0%

Welfare Access Awareness Participation Control

Yes More or Less No

PACAP Performance


Conflict and Fragility: PACAP‘s work in ARMM communities has led to:

1. Direct development support to eleven peace and development communities and programs for internally displaced people in Sulu, Maguindanao, Cotabato City and Basilan;

2. Partnerships with ex-combatants who cooperated with the government through the ―arms to farms‖ initiatives in Lanao Sur, Lanao Norte and Maguindanao;

3. Establishment of 15 literacy programs that incorporate culture of peace in the curriculum;

4. Provision of water systems, health services, sanitary facilities, nutrition enhancing activities, facilitated by 17 projects in Jolo, Maguindanao, Lanao Sur and Basilan;

5. Provision of opportunities in education for children, and in education and employment for out-of-school youth, and internally displaced persons;

6. Through Peace and Development Advocates in seven Peace and Development Councils, communities have drafted local peace agreements with the relevant LGU‘s units, the military, and rebel groups regarding about:

respecting the right for peaceful coexistence;

the right to basic services; and

upholding the rights of children at all times;

7. One barangay went even further with a provision about policies against carrying firearms in the community, thereby establishing itself as a sanctuary for internally displaced persons;

8. Finally, PACAP funded projects have encouraged people involved in the armed struggle to renounce conflict, and instead work towards economic development with civil and public organizations.

Finally, a Gender and Development study commissioned by PACAP in Bohol22

found that in PACAP projects, women ―hold key positions and can influence the direction and implementation of projects” (p 6).

Environment

Proponent organisations reported that 68% of PACAP projects included conservation-related activities, and that 70% contributed to MDG 7 (the goal pertaining to environmental sustainability). 99% of these projects were considered to have been successful in helping to preserve the environment. Moreover – as previously reported - with PACAP‘s support, 4.06 million trees were planted, 122 community-based environmental protection groups were established, and 138 new local environmental ordinances were enacted. 254 of the projects also contributed to

national or provincial environmental initiatives.

On the downside, proponent organisations also reported that the performance of 12% of all PACAP projects was adversely affected by factors in the physical environment such as. flooding, rain, drought, disease and difficulty in access.

Disabled, vulnerable and marginalised persons

Approximately 5% of PACAP projects specifically focussed on disabled persons, these projects benefiting almost 6,000 individuals. The project types were either social enterprise (50%), or social services (50%). Of projects that supported disabled persons, 40% involved health-related activities, and 50% involved education-related activities.

Another 2% of PACAP projects focussed on persons under rehabilitation, 22% on indigenous groups, 10% on out-of-school youth, 6% on children at risk, and 5% on victims of disaster. In toto, 147 PACAP projects supported social services to marginalised groups of one kind or another.

Peace

Proponent organisations reported that 192 PACAP projects (49%) supported efforts to reduce conflict and insecurity, benefiting almost 160,000 individuals. Under the Demand for Better Governance (DBG) initiative, PACAP invested in around 39 projects that specifically targeted conflict-affected areas (see box). Stakeholders involved with peace-building projects within the

22

Conducted by the Sidlak Gender Resource Center.

PACAP Performance


Autonomous Region of Muslim Mindanao (ARMM) interviewed during the 2008 PACAP Quality Audit, reported that not only had many effective projects been funded by PACAP, but that these were unlikely to have attracted the support of other donors.

Unsurprisingly, 30% of projects that worked to reduce conflict and insecurity were adversely affected by detrimental political and stakeholder relationship factors.

Other cross-cutting themes

Lesson

9. PACAP could have done more to address the spread of HIV/AIDS and other infectious diseases, as well as to strengthen communities in disaster preparedness.

Other cross-cutting themes that were not engaged in to the extent of those reported above, included HIV/AIDS and disaster mitigation/preparedness. Only 5% of PACAP projects worked to combat the spread of HIV/AIDS and thereby contribute to MDG 6 (the goal concerned with combating HIV/AIDS, malaria and other infectious diseases). This is primarily because HIV and AIDS are considered a less significant development problem in the Philippines than they are in most neighbouring countries. Nonetheless it can be easily argued that, if the situation is not to worsen, then a more direct focus on this issue would be beneficial in the future.

Disaster mitigation and preparedness is undoubtedly an important issue in the Philippines, given the frequency of natural disasters affecting the country

23.

Nevertheless, only 5% of PACAP projects directly supported disaster victims (there are usually many more immediate funding mechanisms available to disaster victims), although 25.5% of projects did help, in one way or another, to prepare people for disasters. Yet given the emergence of climate change as a development challenge both on a global scale and within the Philippines itself, there is considerable justification for any future phase of PACAP to strengthen its support in this domain.

2.5 RAS Outcomes

Lesson

10. RAS was able to identify and support a range of capable CSO partners to address local priorities in an efficient and low risk way.

11. RAS provided AusAID/NEDA with an efficient mechanism to engage with, and learn about, emerging development issues and priorities, without the design overheads associated with larger dedicated programs.

PACAP‘s RAS component supported 296 projects in 60 of the 89 provinces in the Philippines, in partnership with 253 CSOs. These projects benefited an estimated 281,000 people (38% men, 62% women). Project completion reporting indicated that the intended beneficiaries considered 99% of RAS projects to be successful i.e. they largely achieved what was originally planned. Moreover, 59% of projects were completed on time, a further 25% requiring only a short extension. 98% were also completed within the agreed budget (grant and counterpart). Interestingly, social

23

Natural disasters are recurring hazards across the Philippines. The nation of many islands is located within the Pacific Ring of Fire, in consequence having many active volcanoes and seismic fault lines. It is also located in the typhoon belt of the western north Pacific basin - over 65 per cent of tropical cyclones enter or originate within the region. Based on the records of the Philippine Atmospheric, Geophysical and Astronomical Services Administration (PAGASA), the country faces an average of 20 tropical cyclones a year, of which 5 to 7 are particularly destructive. The country is ranked highest in the world in terms of vulnerability to tropical cyclone occurrences, and third in terms of people exposed to such events (UNDP (2004) Global Report on Disasters)..

PACAP Performance


credit programs performed the worst of all RAS engagements, analysis of project completion data revealing that the larger the loan, the less successful and more risky it was likely to be

24.

Risk analysis suggested that RAS proponent organisations considered the main factors likely to erode RAS performance were relational and political concerns (see Figure 12)

Deeper content analysis of reported implementation risk events revealed that factors affecting RAS implementation were more likely to be ambiguous/force majeure risks or resource constraints than were those associated with FOCAS implementation. Of relatively less significance were risks associated with partnering, staff capacity and strategy.

Figure 12: Reported sources of risk for RAS projects

As noted in Section 1.3, the objective of RAS was:

Civil society organisations are innovating and responding to community needs throughout the Philippines and generating a foundation for new development approaches.

Implicit in this objective are at least three themes. RAS projects were:

Responsive: the solutions to identified community needs were relevant and timely;

Innovative: they possessed both merit and novelty; and

Foundational: they also, generally, had a wider applicability than the beneficiaries specifically targeted.

Each of these themes is now discussed in turn.

Firstly, the concept of responsiveness has always been central to the PACAP program throughout its long history. The responsiveness of the RAS modality in particular has allowed the Australian Government to provide timely support for identified needs throughout the Philippines. Time and again it has proved itself to be both a grounded development mechanism for addressing emerging needs, as well as a way to affirm the bilateral partnership—winning ‗hearts and minds‘. Moreover, the idea of ‗responsiveness‘ had at least two dimensions within the context of RAS: timeliness and relevance. Timeliness meant that the program could efficiently reorient as required, in order to address emerging needs identified by either AusAID/NEDA or its CSO partners. Furthermore, the operational efficiency of the

24

Over 12,000 people benefited from credit with a typical loan value of PhP5,000. Median repayment rates were 87%,% with repayment and project completion rates decreasing as the size of the larger loans Increased.

11.4% 9.8% 10.6% 7.3% 8.9%

11.4% 13.8% 13.8% 12.2% 18.7%

77.2% 76.4% 75.6% 80.5% 72.4%

Social cultural Technical Economic(Resource

Constraints)

Environmental Relational/Political

Yes More or Less No

PACAP Performance


PACAP Secretariat (see Section Error! Reference source not found.) was such that the RAS program could quickly resource projects to respond to identified needs

25. Relevance, on the other hand, was a function of the ‗bottom-up‘ nature of

the project development process. The fact that RAS could fund projects within a broad strategic framework anywhere throughout the country meant that CSOs were able to submit concepts firmly grounded in local realities. Furthermore, the mechanisms employed by PACAP to appraise and verify project concepts, ensured that the objectives and approaches outlined in the submitted proposals were both appropriate and relevant to local needs. A good example concerns the community response to the 2006 oil spill disaster in Guimaras. During project implementation, fisherfolk were not only trained in farming but organised into a cooperative to run a communal farm to provide them with both food and income. This then not only sustained them in the wake of the immediate disaster, but has enabled them to become more resilient in the years since, through income diversification.

Secondly, within the context of RAS, the concept of innovation had at least four dimensions:

i) who was involved (i.e. the nature of the change agents / beneficiaries); ii) where the projects were located (i.e. the operating context or environment); iii) what was done (i.e. the technical aspects of the work); and iv) how the project was delivered (i.e. the approach or method of

implementation).

RAS projects were considered innovative against one or more of these dimensions. For example, the Upper La Paz Community-based Agro-Forestry and Upland Development Project in Zamboanga City, implemented by KRDFI

26, aimed to address

squatter farm encroachment on the Experimental Forest Reserve of the Western Mindanao State University (WMSU). This innovative project facilitated modified tenurial instruments for ninety squatters. Outcomes included: squatter-farmers co-opted as co-conservators of the area; reduced conflict with the WMSU forest guards; reduced forest encroachment; the establishment of 150 hectares of integrated agro-forests; and contributions to WMSU‘s agro-forestry course content.

Thirdly, the concept of foundation was a reflection of the aspiration that RAS would be an ‗incubator‘ for innovative development approaches that could be scaled-up or replicated. For example, the Atikha program for the children of Overseas Filipino Workers (OFWs) initially developed a curriculum to help mitigate the adverse effects on children of having parent/s who worked overseas. Yet the success of this initial work has led to agreements with DepED National to mainstream migration issues in fifteen primary & secondary schools in the three Provinces of Batangas-Laguna-Cavite, with future plans to extend this to twenty Provinces.

Yet despite these three themes (responsive, innovative, foundational) being so central to the concept of RAS, there were few examples of success defined specifically in these terms. On reflection, this was probably a function of several factors, including:

Complex process: development is a complex social process within which ‗innovations‘ as such are not easily identifiable. It is unsurprising then that genuine innovations (of the sort one cannot mistake), are, by definition, relatively rare, even in the case of RAS.

Risk averse system: by design, RAS partnered with CSOs who were experienced players with a track record of performance and credibility within their sectoral or thematic foci. Hence, one possibility is that this approach may have stymied innovation, as the CSOs were effectively engaged to do ‗more of the same‘. Moreover, there may have been subtle disincentives for the PACAP Secretariat (and indeed the PAC) to

25

The average time between the first submission of Expressions of Interest (EoI) and the signing of RAS grant agreements was 157 days. 26

Kasanyangan Rural Development Foundation, Inc.

PACAP Performance


support high-risk (aka ‗innovative‘) projects. The alternative view (the reality according to many stakeholders) being that the use of credible partners was, in fact, a source of innovation – under RAS they upgraded previous approaches to a new level, in particular by expanding their approaches to common problems by the introduction of new services.

Evolutionary process: considerable time and a high volume of interventions are necessary for the genuine innovations to emerge and be recognised. In practice this involves the evolutionary algorithm: diversity selection retention.

Irrespective though of the effectiveness or otherwise of RAS‘s ‗innovation aspiration‘, it is defensible for any bilateral donor involved in community assistance to have a living, breathing mechanism that enables efficient engagement with proven partners who can identify a legitimate need, intervene to effect desired changes, and then withdraw. As discussed in Section 2.2, this is a fundamentally different paradigm to other more protracted modalities of donor engagement, such as FOCAS, but it is probably also an irreplaceable engagement given particular circumstances, such as the need to engage with, and learn about unfamiliar and changing situations, or the need to promote a particular vision or priority of AusAID/NEDA.

2.6 FOCAS Outcomes

Lesson

12. FOCAS is a less efficient modality for achieving beneficiary impact than RAS, but on the other hand has shown promise as a model for fostering sustainable changes in the way that local development priorities are conceived and implemented.

The FOCAS engagement was involved in a total of five target provinces in Mindanao and the Visayas. These FOCAS engagements involved 192 grant agreements that broadly covered nineteen substantive thematic interventions. These interventions, benefiting around 97,000 people, were predominantly concerned with water and sanitation, enterprise development, and environmental conservation

27. Project

completion reports indicated that all FOCAS projects largely achieved what they set out to achieve – all were considered a success by their intended beneficiaries. The majority (80%) were also completed within the agreed timeframe, and 93% were completed within the agreed budget (grant + counterpart).

Yet when compared with RAS, FOCAS cost more money, took longer to establish, benefited less people, involved a narrower spectrum of sectors, encountered more risks, yet all to achieve a similar beneficiary impact. Hence, on the face of it, this suggests that RAS was a more effective modality than FOCAS. However, as highlighted in Section 2.2, FOCAS was concerned with considerably more than the implementation of a coherent portfolio of local development projects. It was equally concerned with the establishment and support of local collaborative networks, in order to generate sustainable changes in the way development priorities were both defined and dealt with. In other words, FOCAS worked at a more fundamental level than RAS, making the delivery of beneficiary impacts a secondary feature for FOCAS. The FOCAS objective was:

Devolved networks of civil society, government and private sector organisations are collaborating to identify and meet local priorities in selected provinces, and creating an enabling environment for further development.

Hence a substantial proportion of the considerable effort and resources that PACAP invested in FOCAS was in strengthening these ‗devolved networks‘ and partnerships. Around 193 FMC and 28 PSC training events alone were convened to strengthen

27

Compared with RAS that benefited 234,000 people and also addressed education, health and social concerns).

PACAP Performance


local institutional capacity. Moreover, this investment was borne out in both quantitative and qualitative evidence of stakeholder engagement in local development agendas. Indicative of the levels of commitment shown were the number and regularity of FOCAS-related meetings convened: a total of 724 FMC meetings and 174 PSC meetings over the life of the program. A FOCAS Participatory Governance Review commissioned in September 2009

28 confirmed the contribution that PACAP

had made to building functional collaborative development networks. It noted:

“FOCAS is remarkably successful at developing partnerships between government, civil society organisations (CSOs), and private sector organisations” (p 16).

“The collaborative relationships that are brokered through PACAP’S FOCAS are broadening the scope and increasing the intensity of CSO and private sector participation in governance. In making this judgement we base our assessment on the perspectives of the participants in the study who favourably contrasted their FOCAS experience with other venues for governance, particularly the system of local development councils at the provincial, municipal, and barangay levels” (p iii).

“PACAP is building on a favourable platform created by changes in the political and public sector context. The key partnership bodies formed to govern FOCAS – the Provincial Stakeholders Committee (PSC) and the FOCAS Management Committee (FMC) – are the key sites where this collaboration is developed.” (p iii)

Moreover, the findings of the Participatory Governance Review were in agreement with the findings of the Annual Quality Audits, which consistently found positive stakeholder engagement with FOCAS processes. For instance, stakeholders positively compared the capabilities of FOCAS with the incapacities of the Local Development Councils (LDCs), indicating, as one example, that FOCAS had ‗put flesh‘ on the intentions behind the Local Government Code (LGC) in a way that the LDCs themselves had been unable to do. Through FOCAS then, the ‗supply‘ and ‗demand‘ sides of governance were able to come together in a new way, in many cases changing the ‗commodities of power‘

29, primarily by shifting the concentration

of power to a broader base. The tangible benefits of FOCAS most frequently communicated by local stakeholders included:

Bridging capital30

: FOCAS brought together different classes of stakeholder, fostering their voluntary and constructive collaboration.

Grounded: FOCAS forced local stakeholders to make plans that were achievable within the available resources and timeframes.

Area-based: FOCAS encouraged integrated planning and implementation that was relevant to local needs, and that maximised local potential.

Leverage: FOCAS partners were sufficiently encouraged by the program‘s success and local relevance to contribute an average of 41% toward the costs of the interventions – well in excess of the 30% required by PACAP protocols.

28

Norrup, S. (2009) Participatory Governance in the Focused Community Assistance Scheme, PACAP 29

Reynolds M. & Holwell S. (2010) Systems Approaches to Managing Change: A Practical Guide, Springer, London 30

Social capital may be understood as comprising two dimensions: ‗bonding‘ and ‗bridging‘. The former is inward-looking and reinforces exclusive identities and homogenous groups. The latter is outward-looking and encompasses people across social divides. In the words of Putnam ―Bonding social capital constitutes a kind of sociological superglue, whereas bridging social capital provides a sociological WD-40‖ (Putnam, R. D. (2000) Bowling Alone. The collapse and revival of American community, New York: Simon and Schuster, p 22 – 23).

PACAP Performance


Lesson

13. FOCAS shows sufficient promise as a model for promoting local democratic governance in the Philippines, that it warrants further experimentation and study.

FOCAS was a new model that drew on the Philippines broad experience with Participatory Methodologies, along with some pertinent elements of Community Driven Development and the older Integrated Area Development Approaches. However FOCAS, like all evolving models needed fine-tuning. Hence its significant evolution during the five and a half years of PACAP‘s recent phase generated many useful lessons. Nevertheless, FOCAS is not yet a concrete model, and more needs to be learned, particularly regarding how to institutionalise and expand its impact. The view of the PACAP implementing team is that the model shows sufficient promise to warrant further experimentation and study. Stimulus questions include: Could FOCAS be set up in such a way that it performs a stronger governance reform function? In particular: how could FOCAS influence the LGC as well as the way that PDCs and MDCs operate? Also: how could some of the tantalizingly preliminary impact on National Government Agencies (NGAs) be scaled up?

Lesson

14. FOCAS contributed to significant improvements in the organisational capacity of partner CSOs. An investment of 668 capacity building workshops facilitated for 46 partner CSOs within the context of a supportive and collaborative network were able to foster statistically significant changes in six dimensions of organisational capacity.

In addition to the support provided to the collaborative networks and institutional fabric discussed above, FOCAS made significant investments in individual proponent organisations. A total of 668 capacity building workshops were facilitated for 46 proponent organisations with the aim of strengthening governance, management, finance and leadership capacities. Significant investments were also made in the technical support of proponent organisations. In this regard, key informant interviews with CSO representatives during PACAP‘s annual quality audits consistently revealed that implementing partners were indeed able to actualize improvements in their organisational capacity as a result of FOCAS processes. The 2008 Quality Audit (p14) found that, in general, proponents identified six broad areas of improved capacity:

Improved proposal preparation skills;

Improved networking skills;

Improved internal systems and processes;

Stronger relationships with LGUs;

Reinvigorated NGO sector within FOCAS provinces; and

Enhanced credibility with other donors.

One Quality Audit interviewee stated that ―The whole engagement process with PACAP was itself a learning exercise‖.

PACAP‘s M&E arrangements included a process to track changes in the capacity of proponent organisations using twenty indicators aligned with six dimensions of CSO capacity: strategy, leadership, resources, partnering, and management systems. A deeper analysis of the findings is provided in Appendix D. For present purposes however, the outstanding fact is that proponent organisational capacity improved

PACAP Performance


across all six dimensions[1]

of capacity. This is revealed in Figure 10 by the skew from left to right between baseline (blue) and endline (green).

These findings provide compelling evidence that capacity development delivered through a comprehensive partnership model can indeed strengthen the internal functioning of CSOs. The findings were also in keeping with the areas emphasised by PACAP‘s capacity training program, the strongest gains being made in the priority areas of training - ‗partnering‘ and ‗management systems‘; the weakest changes

[2]

being detected in the areas of ‗leadership‘ and ‗resources‘, where less emphasis was placed.

Figure 13: Significant changes in proponent organisational capacity

[1]

There were statistically significant changes measured against fourteen of the twenty indicators. [2]

i.e. no significance at the 5% level for all indicators within the dimension.

PACAP Appendix A: Organisational Chart

PACAP – Activity Completion Report I

APPENDIX A: ORGANISATIONAL CHART

PACAP Appendix A: Organisational Chart

PACAP – Activity Completion Report II

Lourdes Padilla PACAP Coordinator

Lilian Santos Finance Officer

Adrian Ocampo

IT Officer

David Swete Kelly Program Director

May Fe de Vera Project Officer

Leon Felipe Fajardo

Project Officer

Pedro Carlos Baclagon Area Manager

Melissa Morales Area Manager

Maria Paz Limpo Area Manager

Maria Consuelo Gomez

Development Communication Specialist

Monina Hernandez

Provincial Project Officer (Surigao del Norte)

Riene Tagupa

Provincial Project Officer (Misamiz Occidental)

James Arceu Honculada

Provincial Project Officer (Agusan del Sur)

Melissa Morales

Provincial Project Officer (Northern Samar)

May Blanco

Provincial Project Officer (Bohol)

Corporate Support

Program Operations

Carol Sabella Administrative Officer

William Trumata

Transport Officer

Cristina Tangco

Executive Assistant

Robert Sabella

Office Maintenance

Ricardo DG Soto Jr. Project Officer

FOCAS Mindanao FOCAS Visayas RAS

Dr Paul Crawford M&E Adviser (Part Time)

PACAP Appendix B: Revised Logframe

PACAP – Activity Completion Report III

APPENDIX B: REVISED LOGFRAME


PACAP – Activity Completion Report IV

Goal OVI MoV Risks

Poor communities in the Philippines are empowered to pursue economic growth and achieve better standards of living.

Av. change in beneficiary household income

% PACAP projects reporting defined constructive changes for women

Av. change in profitability of PACAP-supported small enterprises

Access to basic services (water & sanitation, education, health) by beneficiary households

No. of new and revised local environmental ordinances implemented

Proponent Project Completion Report Pervasive culture of disempowerment in communities erodes momentum for change

Lack of trust and transparency, or corruption, erodes project success

Communities lack the resources and capacity to realise sustainable benefits

Communities lack financial resources for counterpart

Ecological or climatic factors negatively impact on project benefits

Changes in local leadership affects ongoing support and direction

Peace and order deteriorate and disrupt community initiatives

% households experiencing food shortages in past 6 months

% of households constructed from 'makeshift' materials

Bohol household poverty data (PACAP FOCAS subset)

Qualitative changes of significance to beneficiaries

Winning MSC stories

Objectives OVI MoV Risks

Civil society organisations are innovating and responding to community needs throughout the Philippines, and generating a foundation for new development approaches.

Stakeholder perceptions of merit of RAS in allowing innovation and flexibility

Annual Internal Quality Audit (key informant interviews)

Proponent organisations do not align with PACAP values and practices

Proponent organisations do not adequately engender support and participation of beneficiary communities

Proponent organisations lack capacity to identify and implement innovative projects

Insufficient funds allocated to RAS to achieve significant results

AusAID-NEDA support for RAS reduces

Peace and order deteriorates and disrupts RAS proponent organisation operations

No. RAS projects implemented (by project type, beneficiary class and MDG)

% RAS projects completed on time, within budget & achieving planned results

Proponent Project Completion Report

Drivers of success, causes of failure and dimensions of innovation

Independent study/review (case studies)

Devolved networks of civil society, government and private sector organisations are collaborating to identify and meet local priorities in selected provinces, and creating

Stakeholder perspectives on FOCAS participatory governance outcomes

Independent study/review (case studies) Civil society network members do not align with PACAP values and practices

Unresolved conflicting priorities erode success of local development networks

Changes in implementing partner capacity

Proponent Score Card


PACAP – Activity Completion Report V

an enabling environment for further development.

# PSC and FMC meetings, trainings and related activities

% PDCs & MDCs with CSO representation

PACAP records Capacity limits of local partners negatively impact on planned changes

Local networks unable to sustain financial costs beyond PACAP engagement

Changes in leadership (LGU and CSO) affect viability of local networks and projects

AusAID-NEDA support for FOCAS reduces

Peace and order deteriorates and disrupts FOCAS proponent organisation operations

% FOCAS projects completed on time, within budget & achieving planned results

Proponent Project Completion Report

Stakeholder perceptions of merit of FOCAS in laying foundation for future development

Annual Internal Quality Audit (key informant interviews)

The following points elaborate the rationale for the Goal:

Consistent with the original PACAP design, the subject of the goal (i.e. the human actors) is explicitly ―poor communities in the Philippines‖ (i.e. the ‗ultimate beneficiaries‘ of PACAP projects).

―Empowerment‖ reflects the fact that many PACAP-supported projects build the capacity of beneficiary communities to meet their own needs and advocate for their rights.

The emphasis on ―economic growth‖ reflects the strong livelihood-enhancing element of PACAP projects and also AusAID and NEDA‘s strategic focus for the program.

―Better living standards‖ reflects the extent of PACAP-supported work in social development including peace-building and improving access to basic services.

The following points elaborate the rationale for the RAS Objective:

―Civil society” reflects the fact that civil society organisations (NGOs, POs etc.) are the key mechanism or vehicle through which PACAP can realise benefits for poor communities (as stated in the Goal).

“Innovating and responding” reflects the ideals of RAS as a flexible and responsive funding scheme within parameters set by the Strategic Guidance Framework (SGF). It also acknowledges the complex challenges that confront grant aid for community development.

―Community needs‖ explicitly links the Objective to the Goal, and embodies the ideal of PACAP as a community-driven initiative.

“Throughout the Philippines” reflects the wide-reaching potential of RAS to respond to emerging needs and opportunities throughout the country.

―Foundation for new development approaches” emphasises the value of RAS as an ‗incubator‘ for innovative community development approaches (several of which have influenced FOCAS initiatives).

The following points elaborate the rationale for the FOCAS Objective:

―Devolved networks‖ reflects the successful FOCAS approach of fostering strong linkages and ownership of development initiatives at the local level.


PACAP – Activity Completion Report VI

―Civil society, government and private sector organisations” articulates the range of actors working collaboratively—in many cases for the first time.

―Collaborating to meet local priorities” reflects the community-led approach to needs prioritisation, and the strong ownership fostered in the development agenda.

―Selected provinces” expresses the narrow geographic emphasis in the five target provinces selected by AusAID and NEDA.

“Creating an enabling environment for further development” emphasises the value of FOCAS in fostering values, practices and capacities among local development partners that are likely to enhance the success of future development interventions.

PACAP Appendix C: Proponent Organisation Capacity

PACAP – Activity Completion Report VII

APPENDIX C: PROPONENT ORGANISATION CAPACITY


PACAP – Activity Completion Report VIII

During this phase of PACAP twenty indicators of organisational capacity for FOCAS partners were assessed on an annual basis. These indicators related to six dimensions of organisational capacity as follows.

Analysis of changes between baseline (2005) and endline (2009/10) data revealed that, overall, proponent organisation capacity improved across all twenty indicators during the time of partnership with PACAP. Improvements in fourteen of these twenty indicators were statistically significant (at the 5% level)

31. The most significant changes (p < 0.5%) were

measured against six indicators depicted below, represented in the histograms by a skew from left to right for baseline (blue) and endline (green).

31

Improvements against six of the twenty indicators (2, 5, 6, 7, 9, 10) were not statistically significant (at the 5% level)

In the past year, the proponent organisation has…

Strategy

1. implemented programs and projects guided by a clearly articulated strategic direction 2. a majority of members with unified direction/shared aims 3. implemented programs/projects that demonstrate commitment to poverty reduction 4. implemented strategies/approaches that are developmental

Leadership

5. leaders (Exec. Director & supervisors) with demonstrated capacity to perform their roles 6. leaders (Exec. Director & supervisors) who demonstrated integrity in relation to staff & partners

Staff

7. maintained sufficient number of staff to implement and support programs and services 8. staff with technical competence to perform their expected tasks

Resources

9. owned or accessed sufficient assets 10. sufficient cash to cover day-to-day operations

Partnering

11. actively worked with other organizations involved in the same services and/or sector 12. been willing to work with government and private sector to promote interests of marginalized 13. treated beneficiaries as partners in project implementation 14. promoted community ownership of initiatives 15. met documentation/reporting obligations to partners in a timely manner

Management Systems

16. enforced rigorous financial controls to ensure proper use of funds 17. adequate administrative procedures to support delivery of services/programs 18. implemented governance and decision-making procedures to guide the actions of its members 19. used clear structures and communication channels to promote internal information flow 20. generated reliable and relevant information to guide decision-making


PACAP – Activity Completion Report IX

The data for the twenty individual indicators was grouped under the six dimensions of organisational capacity to produce six indices of proponent organisation capacity. These index values were compared at baseline and endline to reveal positive changes across all six dimensions of capacity. The most significant change was in relation to organisational resource management (p=0.006). The less significant change related to partnering (p=0.468).


PACAP – Activity Completion Report X

PACAP Appendix D: M&E Critique

PACAP – Activity Completion Report XI

APPENDIX D: M&E CRITIQUE


PACAP – Activity Completion Report XII

Critique of PACAP M&E Arrangements

The monitoring and evaluation (M&E) arrangements for PACAP were outlined in a M&E Plan submitted as an early milestone of the program in 2005, and then revised periodically throughout the life of the program. M&E was considered an important part of the program design, and accordingly was resourced with a part-time (276 days) international M&E Specialist, a fulltime local Database Officer, a fulltime Data Assistant, and a part-time international M&E Information System (MEIS) Specialist

32.

This paper reports the salient lessons learned in terms of the relative strengths and weaknesses of the M&E arrangements.

Challenges

From a professional standpoint, the M&E arrangements for PACAP posed a range of general and specific challenges.

General challenges encountered by M&E professionals:

Concepts of validity: The field of M&E is affected by methodological debates in the broader field of social research. Some stakeholders value quantitative/objective measures of performance, these being representative. Such stakeholders frequently hold a corresponding suspicion of qualitative/subjective methods, as these are indicative, such methods relying on the professional judgement of the individuals involved. Yet other stakeholders regard quantitative measures as problematic, particularly in cases where the social context precludes a meaningful experimental design. Hence such stakeholders tend to value insightful/context-specific information that can be generated only by using qualitative methods.

Measurement: most human changes are amorphous and hence are difficult to measure in an absolute sense. There is no consensus then on the units of measurement of phenomena such as ‗empowerment‘, ‗capacity‘ or ‗wellbeing‘.

Attribution: programs such as PACAP are implemented within ‗open systems‘, meaning that multiple factors in social/political/economic/environmental milieu contribute to and/or detract from the anticipated changes. This renders the definitive attribution of changes to particular interventions challenging, even at the very best of times.

Diverse and changing information needs: different stakeholders at different times require different information to meet their needs. Frequently these expectations are only articulated in retrospect, making M&E forward planning difficult. Often there is also impatience with the time it takes to detect changes, especially those relating to impact and outcomes.

Organisational behaviour: ensuring that staff systematically capture prescribed data, and that they otherwise comply with agreed reporting protocols, involves a complex set of variables that includes (but is not limited to): motivation, incentives, knowledge/capacity and resources.

Time and resources: rigour and comprehensiveness is constrained by the time and resources dedicated to M&E. Budget and time constraints frequently mean that only rapid or minimalist methods are available to gather information.

Specific M&E challenges facing PACAP:

32

The MEIS Specialist was paid under a license agreement for ‗eM&E‘—the commercial MEIS used by the program to capture, analyse and report performance and risk information.


PACAP – Activity Completion Report XIII

Geographic distribution: performance data for PACAP was routinely required from projects located throughout the Philippines—especially in the south—these being places that were typically weak in terms of information communication technology (ICT) infrastructure.

Demographic distribution: PACAP projects targeted a wide array of beneficiary types, ranging from disabled children to agricultural cooperatives. Hence there was no universal definition of ‗impact‘, and no common/standard definition of ‗ultimate beneficiaries‘.

Sectoral/thematic diversity: PACAP engaged in a range of technical sectors with the result that standardised indicators lacked meaning. Consequently, each project developed its own basis for judging success. Yet while this approach was meaningful at the ‗project level‘, aggregation to the ‗whole-of-program‘ level proved problematic.

Ambiguity of purpose: at different times during the life of the program AusAID‘s focus and expectations of PACAP changed, with consequent changes in the nature of the performance information that was expected from it. Also, PACAP was periodically asked to deliver information to support emerging knowledge gaps within AusAID (e.g. information regarding support for disabled persons), such ad hoc information requests being difficult to anticipate, and hence challenging to support.

Relative Strengths

The salient strengths of the M&E work for PACAP included:

Leadership: the prominence given to M&E within PACAP was a direct function of the strong emphasis placed on evidence-based learning and accountability by the Program Director. It is widely recognised that successful M&E is contingent on team members‘ compliance with agreed protocols; and this in turn is contingent on senior management attributing significant importance to the assigned tasks. In the case of PACAP, the Program Director instituted a range of measures to give prominence to M&E processes—such as including M&E compliance as part of some annual staff performance appraisals, and periodically auditing data integrity.

Dedicated database officer: successful M&E analysis is contingent on the integrity of mundane data management processes. The importance of having a capable and committed database officer to manage these tasks was highlighted in PACAP with the dramatic improvement in data quality that accompanied the replacement of PACAP‘s first database officer for performance reasons.

Breadth of data: a strength of the PACAP M&E arrangements was the breadth of data that was systematically captured during the life of the program. This data not only covered all levels of the design logic

33, but

included a balance of quantitative and qualitative information. It also integrated performance data and risk data to facilitate a ‗systems perspective‘ on program performance. Data was also captured on a range of cross-cutting themes, including gender and contribution to Millennium Development Goals (MDGs)

34.

M&E tools: some of the M&E tools developed for PACAP, while not perfect, proved to be valuable instruments. Of particular note was a ‗Proponent Scorecard‘ which was a brief survey (20 questions) implemented annually to provide a snapshot of developing partner capacity across a range of institutional dimensions. Also, a ‗Project Completion Report‘ proved to be a valuable way of garnering a ‗census‘

33

i.e. implementation quality and performance; partner capacity and outcomes; ultimate beneficiary impacts. 34

N.B. PACAP‘s capture of MDG contributions preceded AusAID‘s official alignment with the MDGs at the corporate level.


PACAP – Activity Completion Report XIV

of project information across the diverse PACAP portfolio. This instrument collected a range of generic/standard information (e.g. gender, MDGs, project management etc.) as well as a range of sector-specific technical information.

Annual Quality Audits: the use of a consistent method to conduct annual quality audits produced valuable trend information during the life of the program. Furthermore, the use of a blend of both qualitative and quantitative data provided compelling insights into implementation performance. This in turn triggered valuable internal reflection and continuous improvements.

Asynchronous database: PACAP subcontracted the services and software of Aid-IT Solutions to support its M&E data management. Yet although the MEIS was not without its challenges, it nonetheless provided a technically innovative way of capturing, updating and sharing data from geographically distributed staff, located in ICT-challenged operating contexts

35. The considerable success of the PACAP MEIS is

noteworthy, especially given the common failings and inadequacies of ‗bespoke‘ information system in many other AusAID initiatives.

Relative Weaknesses

Key weaknesses identified concerning the PACAP M&E arrangements included:

Performance culture: ensuring staff compliance with M&E protocols is a universal challenge. The successful inculcation of a ‗performance culture‘ within program staff requires deliberate change management strategies. In retrospect, such change management strategies within PACAP could have been better planned for and implemented. Accordingly, weak M&E compliance remained a persistent challenge throughout the life of PACAP. This in turn had a negative effect on data quality, reporting reliability and staff morale.

Mainstreamed M&E: both the M&E Plan and MEIS were underpinned by the fundamental concept of ‗mainstreamed M&E‘. This concept - current at the time PACAP was mobilised - aimed to make M&E a part of regular management and operational responsibilities. Yet PACAP‘s thorough testing of this approach concluded that mainstreaming M&E is problematic because: i) M&E tasks are technical in nature, and hence are frequently beyond the capacity and skill of program staff to adequately implement; ii) program staff are already frequently overburdened with the technical/operational aspects of their roles—M&E is seen as yet one more ‗add-on‘; iii) mainstreaming M&E tasks, by definition, increases the number of actors involved with data management, with a consequent negative impact on both data quality and ownership of the data sets.

MEIS limitations: the MEIS used by PACAP was a commercially available customisable system. This meant that it was ‗off the shelf‘, yet with generic capabilities that could largely be customised to meet the unique requirements of particular programs. Nevertheless, there were inevitable limits to the extent to which such customisations proved satisfactory

36. Hence some debate ensued regarding whether or not the

program should have invested in a ‗bespoke system‘ at all. Nevertheless the system had its advantages. It was basically free of many of the technical/fundamental ‗bugs‘ that have typically plagued rapidly

35

N.B. the technical challenges faced by ACCESS Phase II (PACAP‘s sister program in Indonesia) in trying to extend Ersula (the ACCESS information system) to enable distributed data access, highlights the relevance of the technical innovations within the PACAP MEIS. 36

For example, a particular frustration arose from standard data security measures that restricted the team‘s ability to edit/modify data in the system after it was entered. While the rationale for this restriction remains clear, it nevertheless inhibited attempts to ‗scrub‘ errors in the datasets in order to improve data quality.


PACAP – Activity Completion Report XV

developed systems for other programs37

. It was considerably cheaper than the cost of developing a bespoke system with similar technical capability. It was also rapidly deployed following mobilisation, without the long lag-time typically associated with bespoke system development.

Use of partner government systems: PACAP‘s M&E arrangements can be criticised through the lens of the Paris Declaration and Accra Agenda for Action for insufficiently working through partner government systems. However, the M&E Plan was discussed with NEDA at various stages during its development, and there was a general sense that the level of information reported to NEDA was adequate, and that no further integration of systems was required. The level of ‗comfort‘ within NEDA was further reinforced by the active involvement of NEDA representatives on the PAC, and, of course, through the PCC. Hence, though more could have been done to integrate PACAP M&E processes within LGU systems, it remains unclear what such integration would have achieved, especially given that it was never requested by any government partner.

M&E capacity building: PACAP provided a range of capacity building opportunities for implementing partners, including M&E training. This was commensurate with AusAID‘s broader investment in the program‘s M&E. Nevertheless, in retrospect, more could have been done to strengthen the capacity of proponent organisations in M&E. For instance, the program‘s systematic M&E requirements could have been extended down to partner level, thereby providing ‗learning-by-doing‘ opportunities. Yet this would have required significantly more resources than was allocated for M&E.

Capacity of LGUs: the original PACAP design focused almost exclusively on the role of civil society organisations (CSOs) in facilitating local development outcomes. However, an important evolution of PACAP (especially within FOCAS) was the increasing importance of LGU-CSO collaboration. Yet no commensurate M&E methods were developed to assess changes in LGU capacity. In retrospect then, this was a shortcoming of the M&E arrangements.

Secondary Impact data: a range of methods were employed to assess the outcomes and impact of PACAP. One mechanism employed for FOCAS was the analysis of a subset of Bohol provincial government poverty data in order to ascertain broad changes in poverty among PACAP beneficiary households (using both with/without and before/after datasets). Some useful insights were gained from this dataset, but, in general, the use of secondary data proved fraught - PACAP was unable to control the integrity of the data captured, or the timeliness of data analysis.

Impact data for RAS: As noted above, a secondary dataset was used to assess the household impact of PACAP projects among FOCAS beneficiaries. Yet no equivalent mechanism was put in place for RAS, and so knowledge of the general impact of RAS projects at the household level is even less well developed than for FOCAS.

General Lessons Learned

Rigorous M&E planning is valuable: there is great value in a rigorously defined M&E Plan that has been prepared during inception to provide an ‗information architecture‘ for the life of the program. The real value of the PACAP M&E arrangements was only borne out during the preparation of the this report, at which point all the ‗pieces of the puzzle‘ were drawn together to create a comprehensive ‗performance picture‘ for the program.

37

Evidence suggests that less than a third of bespoke systems developed for international development programs are deemed to have been successful.


PACAP – Activity Completion Report XVI

Staff M&E compliance should be proactively addressed: staff M&E responsibilities should be given appropriate prominence by program leadership, and included in staff performance appraisals. Hence from the very beginning, consideration should be given to the provision of inherent incentives/ disincentives for staff compliance with M&E protocols. Weak compliance erodes the fundamental integrity of M&E data, which in turn squanders the resources invested in M&E work, and renders performance analysis a questionable undertaking at best. Yet addressing this issue is likely to require the development of a ‗performance culture‘ within the team, a process which should draw on established organisational change management methods.

‘Mainstreaming’ M&E doesn’t work: core M&E responsibilities are best implemented by a dedicated team of M&E professionals, rather than being ‗mainstreamed‘ within broader technical, managerial and operational roles. A specialised M&E Unit is more likely to produce high quality data that is accurate, timely, relevant and accountable. The apparent additional expense of engaging a specialised M&E Unit would be amply justified by the higher quality of performance information and learning.

There is value in having a specialist MEIS and associated technical support: ICT proved a valuable support to PACAP‘s M&E arrangements as it enabled the capture, storage, manipulation and reporting of a more comprehensive range and volume of data than would have been possible using paper-based or less sophisticated technology-based systems. However, the architecture and functionality of M&E systems is a complex area that benefits from precise end-user specification. Hence in those instances where time, resources, skill and experience are available, programs would certainly benefit from developing bespoke MEIS to meet their precise information needs—on the other hand, this presupposes that program staff can articulate these needs from the outset.

pacap impact and lessons 2005-2010 september 2010 · pacap impact and lessons 2005-2010 ......

Documents