Monitoring and measuring change in market systems-rethinking the current paradigm-

(Discussion paper1, rev. 22 - 9 Oct 2012)

1. Introduction

As is widely experienced, markets are not static, but in constant flux. The changes are not predictable and often surprising, even after thorough market analyses and strategic planning. Our interventions turn out results that were either unforeseen or unanticipated. This type of behavior creates specific challenges for monitoring and evaluation of development interventions.

Changes in markets must be monitored and measured using approaches that embrace their dynamic and unpredictable nature. Advances in complexity sciences can provide useful principles and concepts to build the foundations of more effective and relevant market and financial system evaluations.

This paper is intended to promote online and in-person discussions that will bring together practitioners and experts to explore two broad issues:

the obstacles and challenges that field practitioners face when trying to monitor and measure changes in market systems using the current linear, top-down, rigid, mechanistic paradigm

the principles and guidelines that can bring donors and practitioners together to build systemic M&E frameworks and tools that produce not only relevant evidence about their impacts on markets systems, but also appropriate and timely information for field practitioners to help them navigate the fast changing and unpredictable landscape of market systems.

2. Main obstacles and challenges with the current M&E paradigm

Based on various online and face-to-face discussions with market development practitioners, three broad issues that encapsulate their major concerns about current M&E practices have been identified. While these issues, as described below, paint a rather dire picture of the current M&E paradigm, practitioners do acknowledge that there have been positive and important debates and initiatives that are creating fertile spaces for further innovations and improvements. However, some of such debates and initiatives are still based on a paradigm

1 This paper is one of the main inputs to the systemic M&E initiative promoted by SEEP’s MaFI (The Market Facilitation Initiative). Systemic M&E is one of the solutions proposed in the MaFI-festo, which is a document that attempts to synthesize the most pressing issues that MaFI members consider critical and urgent to make the international cooperation development system more “facilitation- and complexity-friendly” and, ultimately, more cost-effective. Note: there are still aspects of the MaFI-festo where not all MaFI members agree.This paper was written by Marcus Jenal and Lucho Osorio-Cortes using inputs from MaFI members. It is intended to promote discussion, reflection and debate between development practitioners, donors, policy-makers and researchers. The paper will evolve with the on-line and in-person discussions that will take place in MaFI, USAID’s Microlinks and the SEEP Annual Conference (see the plenary here). The ideas in this paper do not necessarily reflect the opinions of The SEEP Network or fhi360.2 Changed top-down accountability for upward accountability and minor typos corrected.

1

that relies on simplistic and static worldviews rather than embracing complexity to then build better models to make sense, transform and measure reality.

2.1. Issue One: Excessive focus on our direct effects on the poor

Donors, parliaments and eventually taxpayers ask development projects to prove their direct impacts on the poor: “how many schools or hospitals did WE build; how many cows or bags of improved seeds did WE delivered; how many training workshops did WE host and how many people attended; etc”. The list of the inputs and services that “we” –the development agents, can deliver is long.

Measuring direct impacts on the “poor” is possible when using a direct delivery approach because the relationship between the provider and the receiver is straightforward and easily verifiable; consequently, it is possible to attribute changes at the level of the “poor” to a project intervention. Accountability is, thus, translated into an act of keeping track of the resources delivered into the system and demonstrating their positive and direct effect on poverty indicators such as income and employment levels.

Detailed planning and standardized result measurement frameworks are believed to help development projects to plan their impact and measure it afterwards. M&E approaches, procedures, and tools focus on collecting information on numbers of beneficiaries and changes at beneficiary level. Attribution of the changes to project interventions is done using extrapolations based on alternative scenarios without the project.

2.2. Issue Two: Excessive focus on extraction of information and accountability to the donors

The belief in upward accountability has survived because it gives NGOs, donors, parliaments and taxpayers a comforting sense of control over the process and certainty about the outcomes of their development initiatives. This is needed by the whole development “industry” to survive in a demanding political economy exacerbated by global recession.

In the reality of the field, practitioners are dealing with highly complex and dynamic economic, social and cultural systems and trying to navigate fast-changing and in many cases unpredictable landscapes of opportunities and risks.

Instead of supporting project implementation, M&E procedures catering for upward accountability become an additional burden of data collection –most of which is meaningless, irrelevant or outdated, tied to predefined and rigid work plans and log-frames, and decoupled from realities and needs on the ground.

The focus on changes at beneficiary level has incentivized project managers to directly intervene for the poor, mostly ignoring the need for changes in the wider system to make the intervention sustainable and scalable. Deliverables and impact targets are defined during the planning phase incorporated into planning tools and contractually fixed by the donors and the implementers. There are strong incentives for the implementers to try to control the evolution of the system to make it fit within the agreed plans, instead of promoting adaptive management that “flows” with and leverages the energies of the system.

2

2.3. Issue Three: Sustainability understood as longevity of our legacy

The understanding of sustainability as the permanence of what development agents do or provide to the poor drives the efforts of the current M&E system. The goal is to prove that positive changes in poverty indicators will persist beyond the project and by virtue of the project’s deliverables. The dynamic and complex nature of market systems is, however, incompatible with this approach: prices change, industries strive or collapse, droughts devastate crops, new technologies or business models destroy jobs, etc. Too much focus on the permanence or longevity of what development projects provide is distracting donors and implementers alike from building and measuring the ability of the market system actors to create their own solutions to withstand current and future shocks, and adapt to changes that are very difficult to foresee.

Rather than small adaptations to current M&E practices, these issues ask for a radical rethink the current M&E paradigm, in particular in the field of inclusive market development where complexity plays a crucial role. Indeed, the thinking that constitutes not only the foundations of M&E but also of project planning and design has to shift from a mechanistic view on the economy and society to an understanding that socio-economic systems are complex and dynamic networks of many different, interconnected and interdependent actors.

3. Proposed principles for a systemic M&E framework

The following are five principles that are based on complexity theory and on the premise that systemic M&E approaches, frameworks, tools and incentives have to be designed and used to measure "systemic change".

Systemic change is thereby defined as transformations in the structure or dynamics of a system that leads to impacts on large numbers of people, either in their material conditions or in their behavior.

These principles are intended to shed some light on the question of "how would a systemic M&E framework look in practice" and to promote debate and convergence between donors, researchers and practitioners in the field of inclusive market development about how to build fully-operational and user-friendly systemic M&E frameworks.

3.1. In systemic interventions, all beneficiaries are indirect beneficiaries

In systemic interventions all beneficiaries are indirect from the point of view of the project because their context, relationships and possibilities to access tangible and intangible assets are affected by changes driven by the market system itself, not by the project.

The relatively few market actors with whom a project engages directly are in fact collaborators. A project enables collaborators to test new ideas, demonstrate benefits, mobilize large numbers of other market actors, etc. Collaborators actually work with the facilitators to transform the structures and dynamics of their market system.

3

If the project wants to achieve impact at scale, it is not feasible for the facilitators to work directly with the vast numbers of marginalized actors that constitute the project target populations. Therefore, projects have to aspire to systemic change, reaching target populations indirectly.

As a consequence, the focus of accountability has to shift from counting direct beneficiaries and assessing direct impacts at beneficiary level to a broader view of changes in the structures and dynamics of the market system with indirect effects on the target populations.

Recommendation: shift the accountability focus from counting direct beneficiaries to proving contribution to systemic change and estimating systemic (indirect) beneficiaries.

3.2. The deeper the systemic change, the larger and longer lasting the effects

Current M&E practices prescribe the use of indicators such as beneficiaries' incomes or number of new jobs created. These types of indicators detect changes in the stocks and flows of the system (e.g. how much money farmers have in their pocket, how many new jobs were created last month or how much milk a group of farmers sells in one day). These indicators can be useful to detect changes in the system but these are changes that take place at the most superficial levels of the system and can therefore be produced with relative ease by the presence and investments of development agents.

Superficial changes in the system can also expose market actors to a high risk of relapsing to their earlier (pre-project) states or even worse states such as dependency on donor funding or conflicts fueled by project subsidies. Focusing the attention on these superficial changes creates a false illusion of success that can easily mislead donors when deciding where, how much and when to invest, and practitioners when deciding how and when to act, what and how much to subsidize, and when to exit.

It is very important to pay more attention to deeper and more structural changes in the market system, such as the creation of new networks, associations, or business models; increased access to information; shifts in power dynamics; collaboration around jointly agreed objectives, etc. Information about what goes on at deeper levels of the market system can be used to determine whether the system is changing its trajectory towards a horizon of more inclusion, productivity and efficiency, and whether it will stick to its new course beyond the life of the project.

Recommendation: put more emphasis on changes at deeper levels of the system

3.3. The facilitator does not change the market system; the actors with whom the facilitator collaborates do Systemic change affects large numbers of people. The extent of the change and its impacts are shaped by the structure of the system (e.g. legislation, infrastructure or networks) and the capacity of its actors to use the system’s resources. Deep changes in the system are caused not by the project, but by decisions taken by a large number of actors who are beyond the reach and control of the facilitator. As a consequence, the further away an observed impact is from the facilitator’s interventions (i.e. the more links in the chain of events that led to changes in the situation or behavior of the actors), the more difficult it becomes to attribute the impact to the interventions. Systemic M&E frameworks should surrender any attempts to

4

attribute impact at the end of the chain to a specific intervention and focus on the effect of the facilitator on the groups of immediate collaborators that actually drives the transformations of their own market system. These groups of collaborators can be called precursor networks. Precursor networks are any type of informal or formal network or group of actors that creates spaces and routines for market actors to learn from one another; align objectives, strategies and activities; pool and leverage resources; experiment with new ideas and disseminate the successful ones throughout the system, etc. Precursor networks can be informal farmer groups, cooperatives, chambers of commerce, lead firms, multi-stakeholder platforms, grassroots associations, etc.

Three basic types of precursor networks can be proposed: networks that empower market actors to engage with others, networks that bring different market actors together to transform the system, and networks that disseminate knowledge throughout the system.

A better understanding of how these networks can be created, sustained or improved and how they drive change throughout the market system will allow development agents to improve their interventions and to establish more robust causal links between the actions of facilitators and the behavior of these networks (which in turn would allow to infer the facilitators’ contributions to improvements in the wider system).

Recommendation: assess the effects of the facilitators on the collaborators and infer systemic impact from the performance of the precursor networks.

3.4. Complex systems behave in extreme and unpredictable ways and are sensitive to the presence of the project

The highly dynamic and unpredictable nature of complex systems undermines the effectiveness of tools and approaches that are currently used in market development, such as long-term strategic analysis and planning, log-frames, or fixed indicators and goals. These tools and approaches are based on a world of relative stability and predictability, and repeatability of causes and effects.

In order to deal with the dynamism and unpredictability of market systems that we find in reality, new approaches need to be built on the principles of variation, experimentation and adaptation. To implement these approaches, successful projects also require skillful facilitators who can sense and adapt to conditions on the ground and respond appropriately and swiftly to the actions, intentions and fears of market actors –some of which are generated by the presence of the project itself.

The facilitators’ responsiveness is not only determined by their skills and knowledge but also by the organizational environment they belong and are accountable to: their project team, their organization, their professional associations and networks, the donors, and, of course, the M&E frameworks, tools and procedures that the facilitators are required to use.

The systemic M&E frameworks required by the field of inclusive market development should not only provide evidence of impact to the key development agents, but also fast, reliable and relevant information to the facilitators to help them navigate the “systemic jungle” effectively, for example: taking advantage of unforeseen opportunities, minimizing conflicts, leveraging the resources of public and private market actors, minimizing operational costs,

5

contributing to deeper impacts, etc. Systemic M&E is needed not just for “proving” but also for “improving”.

Recommendation: invest in M&E frameworks, procedures and incentives that, besides proving the effectiveness of the project, enable facilitators on the ground to adapt, learn and collaborate effectively.

3.5. Adaptability is the ultimate manifestation of systemic sustainability:

As mentioned above, one key aspect of systemic M&E frameworks is the focus on how the facilitators are influencing the performance of precursor networks and how these networks are moving the system towards a more desirable horizon. Additionally, it is important to assess if the system is building the necessary conditions to avoid or minimize future shocks and benefit from new trends whilst staying inclusive, productive and efficient.

Predicting how a market system will react to a shock is practically impossible. It is, however, possible to assess if a system is becoming more or less adaptable – and therefore more sustainable. A better understanding of how a market system builds and maintain resilience and adaptability would provide important information to build systemic M&E frameworks that pay attention to the structures, dynamics and parameters that would allow development agents to assess trends towards more or less sustainability.

Recommendation: do more research about the components and drivers of resilience and adaptability of different types of market systems (sectors and subsectors) in different contexts and design M&E frameworks to detect these changes.

6