performance measurement in smallholder supply chains: a

Performance Measurement in Smallholder Supply Chains:

A practitioners guide to developing a performance measurement approach

2

Small-‐scale farmers—whose output supports a population of roughly 2.2 billion people worldwide—manage roughly 85% of the world’s farms. And every day, companies trade with these farmers in a wide variety of products. As companies seeks more transparency through complex supply chains and invest together with development and financial organizations, there is more and more interest in tools and approaches to gain insight into the sustainability and livelihoods of farmers. Over the past years, with the support of the Ford Foundation, the Sustainable Food Lab has convened a learning community focused on sharing the learning and challenges associated with performance measurement in small-‐scale producer systems. This community consists of a number of M&E practitioners from companies, and NGOs working with smallholder farmers in developing countries. Members of the community include individuals from the Committee on Sustainability Assessment (COSA), the International Social and Environmental Accreditation and Labeling Alliance (ISEAL), the Center for Development Innovation at Wageningen (CDI), Rainforest Alliance (RA), and many others. This group has come together around performance measurement in order to accelerate learning about engaging smallholder supply chains. We have reached agreement on the purpose and scope of smallholder performance measurement, and we’ve worked to increase consensus and consistency on a subset of indicators that are used consistently across performance measurement projects as well as on common approaches to measuring these widely accepted indicators. This work has yielded some valuable contributions to the field of performance measurement that we aim to share here in the form of a methodology guide—a guide to help those working with smallholder farmers do so effectively by allowing learning from evidence. This guide summarizes years of work to build consensus on why we undertake performance measurement, and how we do so effectively and affordably in a credible way. This guide is intended to illustrate that there is an easy and credible path for performance measurement in smallholder agricultural supply chains, and to lie out that path so that practitioners may design their own performance measurement approach. We have used the example of the Sustainable Food Lab’s performance measurement work in smallholder sugarcane in Paraguay as an example to help bring this guide to life. This work is described in summary on page 6, and referred to throughout the paper. Paraguay project examples are denoted by green text.

3

TABLE OF CONTENTS: Introduction……………………………………………………………………………………………………………………………………3 Part 1. Designing Your Performance Measurement Approach………………………………………………………..6 Part 2. Administering Your Survey: Methodology Considerations………………………………………………….11 Part 3. Closing the Learning Loop………………………………………………………………………………………………….18 Conclusions…………………………………………………………………………………………………………………………………..19

INTRODUCTION: LEARNING FROM EVIDENCE A growing number of companies are expanding their smallholder sourcing programs—including the use of 3rd party certification. Many are interested in the possibility of cost-‐effective approaches to better understanding the farm-‐level sustainability of their smallholder supply chains. Similarly, development organizations are looking for practical ways to build livelihood monitoring into their agricultural enterprise work in order to complement their impact studies, improve learning and effectiveness, and to strengthen the links through the supply chain. An affordable way to measure progress is critical for increasing transparency about the conditions and needs of producers, and to building effective information feedback loops in order to learn from evidence about what works when engaging smallholder supply chains. Performance measurement is a Monitoring and Evaluation (M&E) approach intended to measure status (current stage of conditions) and track change over time. The goal of performance measurement is to provide modest (in scale, scope, and cost) approaches to measuring conditions and change that complement other more sophisticated impact measurement techniques. As pointed out in the COSA Global Report, “Impacts can take many years to evolve and manifest…in the meantime, investments continue and require ongoing direction and decision-‐making”1. This is where performance measurement can play a very useful role.

Performance measurement can be useful for a single study to measure current conditions of producers within a supply chain (such as average farm productivity at the farm level or average household revenue), and for repeated measurements of whether activities are being accomplished as expected, and whether the main outcomes are moving in the right direction. This approach can allow for some general analysis of correlation between the adoption of better management practices and specific outcomes (e.g., crop yields), but is not necessarily rigorous enough to demonstrate attribution of outcomes to 1 Understanding Sustainability: First global report on COSA findings in agriculture. COSA, 2013.

Figure 1. Impact studies compared to performance measurement

4

specific activities. Attribution questions—how much change can be attributed to a specific intervention—require more rigorous methods, including counterfactual groups for comparison. These approaches can be complementary, as illustrated in Figure 2. Typical Characteristics of Performance Measurement:

• Data collection over a range of indicators • Drawing on data from existing sources in the system (such as certification audits or supplier

data) or through fast and affordable survey techniques • Changes are measured by tracking change over time, not from control group comparisons • Household surveys are often designed to be straightforward, relatively quick, and easily

administered by non-‐professional enumerators • The focus is often on understanding the supply chain and the producers in it, not on evaluating

the impacts of very specific interventions

Figure 2. Example of performance monitoring used between baseline and impact assessment

This practitioner’s guide outlines the process of developing a performance measurement approach from the bottom up, and then advises the reader on the most critical considerations in terms of how to collect the data. We begin with a review of the steps necessary to use your company’s purpose for undertaking performance measurement to define your Learning Questions, and subsequently, the indicators and metrics you will use to answer those questions. Part 2 focuses on the details of administering the survey you devise in Part 1. We will cover methodology considerations like; who interviews the farmers, what time of year you interview, data collection tools, and more. Part 3 explains the importance of completing the learning cycle by using the data collected with this performance measurement approach in order to learn, strategize, and adapt for more sustainable supply chains.

5 Every sound performance measurement initiative begins with an exploration of purpose: why PART 1.

Company Case: Sustainable Food Lab’s Values Based Sourcing Paraguay Sugar Project PURPOSE: Many of the members of the Food Lab’s Values Based Sourcing Group use sugar as a key ingredient, and many of the companies involved currently source sugar from Paraguay, which has a fast-‐growing supply of Fairtrade, organic sugar. Typically, there has been very little visibility into sugar supply chains. The group was interested in learning more about the sustainability of this sugar supply chain at the household level. Data was collected over a period of two years in order to test and compare data collection methodologies. LEARNING QUESTIONS Are the basic needs of farmers and their families met?

• Social indicators: Number and gender of farmers, improvements in income, assets, food security Are the farmers earning enough to keep on farming this crop?

• Economic Indicators: Crop income, crop profit, hectares in production Are farmers realizing their farm potential?

• Commercial Indicators: Productivity, adoption of best practices Can the land sustain continued cultivation?

• Ecological Indicators: Soil, water, climate/energy Are farmers experiencing good supply chain relationships?

• Trading Relationship Indicators: Access to services, transparency, capacity of producer organizations DESIGING DATA COLLECTION METHODOLOGY: In the first year of the project, data was collected at the household level with a 20-‐minute survey. Focus groups were also conducted to gather qualitative data. The second round of surveys was administered at the household level as well using the same indicators as year one, but with refined metrics. A Progress Out of Poverty Index was included in year two, and the survey was refined. Questions that did not meet the standards of being both useful and efficient were removed. Year two did not include focus groups. Sampling: In year one, we visited farm households based on location. In year two, we used systematic random sampling to identify the survey subjects. We established a sample size of approximately 300 based on the population size and a desire for a 95% confidence level with a 6% margin of error. Who collects the data: We used SFL staff and paid enumerators to conduct the first round of surveys. Producer organization agronomists trained by SFL staff conducted the second round of surveys. ICT Tools: We used iForm Builder to create the survey for a mobile data collection with a tablet. iForm builder allows for simple survey design, as well as collection and aggregation of data. Data was analyzed in Excel. Data was uploaded directly to the iForm builder database and analyzed by SFL. Plans to embed this approach in the supply chain: This approach was designed and revised with the idea that it will be embedded into the supply chain. We tested embedding this ongoing monitoring process by training producer organization agronomists to administer surveys. KEY LEARNINGS: Below is a summary of top-‐level data for the second year of surveying. In order to learn from this data and use it to inform the way they work, the Values Based Sourcing Group used this data and other more nuanced correlations to begin to shape a story about these smallholders.

LEARNING QUESTIONS TOP LEVEL DATA 2013 Are the producers able to meet their basic needs?

-‐Food insecurity 29%, cause for concern -‐ 47% likely to fall below national poverty line of $1048. -‐ Access to Electricity and water, communication device and land ownership at 99%

Are farmers earning enough to continue growing this crop?

-‐Average planted area is 12.5ha -‐Prices (premium included) exceeded known production costs -‐High dependence on premium (25% of net cane income) -‐100% of farmers plan to plant same volume or more next year.

Are farmers realizing their farm potential?

-‐Average yield is 53% of potential yield (38 tonnes/ha vs. 71) -‐Farmers using only 50% of the fertilizer they’d like to use -‐53% best practice adoption

Can the land continue sustained cultivation?

-‐71% adoption of soil conservation practices -‐100% use of riparian buffer -‐Average of 12% of property set-‐aside for forest -‐87% note changes in weather patterns -‐ 99% do not burn cane residues

Are famers experiencing healthy trading relationships?

-‐Good access to services from PO: 71% have tractor service, 68% offer technical assistance, 54% offer cane transport, 50% offer inputs -‐High concern about fairness of price and transparency of exporter

6

PART 1. DESIGNING YOUR PERFORMANCE MEASUREMENT APPROACH Why is your organization interested to learn more about the conditions of the smallholders in your supply chains? Being clear on your purpose is essential to making good choices about what information to collect and how to collect it. Start by clarifying your purpose – why you are interested in farm level data and what detail you need, and your theory of change (how your organization’s actions may impact farm and household). This will help clarify your Learning Questions and Indicators—what you most want to learn, and finally your methodology and specific metrics. See Figure 3 for an illustration of this process.

Figure 3. Designing Performance Measurement Approach Before getting into the details of designing your approach, however, it is useful to understand a few key terms that are frequently used in talking about performance metrics.

Common Terminology

Impact Area: The broad category of social, environmental, or economic change aspect to be tracked. Indicator: Qualitative or quantitative descriptors of a measureable state or condition. These can be

designed to reflect current status, change, or comparison.2 Learning Question: The questions that you would like to be able to answer from the measurement

data, such as “Are the basic needs of the farmers being met?” Metric: The means of measure; the specific quantification of an indicator. Performance Measurement: A monitoring and evaluation approach to understand sustainability

status and track change over time. Performance measurement does not measure causation or attribution of impacts to specific interventions.

Survey Question: The specific question that is asked to the interviewee to collect data on the metric,

which will inform the indicator. Survey questions can also contain important guidance on who should ask the question and why it is being asked.

Theory of Change: A causal flow that illustrates how a proposed set of interventions and inputs

will result in specific outputs, contributing to different outcomes leading to certain impacts. The assumptions underlying this causal flow are important to monitor.

2 ISEAL M&E Guidance Note: Getting started on your Monitoring and Evaluation System. ISEAL, 2013.

1. PURPOSE

2. THEORY OF CHANGE

3. LEARNING QUESTIONS & INDICATORS

4. DATA COLLECTION

METHODOLOGY

5. METRICS & SURVEY

90

3

BUDGET

7

1. Define your purpose. Defining your purpose for performance measurement requires asking why you are interested in assessing the sustainability of these smallholder farmers. What will you do with the data that results? Is the data intended to inform your smallholder engagement strategy? Is it for sustainability reporting? Measurement can serve a number of goals, including:

• An initial investigation to understand the primary social, economic, and environmental issues within a crop and location;

• To evaluate whether a supply chain is meeting goals around social and environmental performance; and,

• To see if programs designed to achieve specific sustainability goals are progressing in the intended direction.

A critical component of clarifying purpose is being clear about the degree of attribution needed from the measurement. Do you need to quantify the contribution of your investments and interventions to specific outcomes? For example, if you need to know exactly how your farmer field school program improved productivity and crop income amidst everything else going on in the supply chain, then you want conduct a full impact study. If you want to track how activities are progressing —the number of people attending trainings and if things are heading in the right direction (productivity and crop income improving) then performance measurement can work well for you. For example, when a small group of Sustainable Food Lab member companies decided to join together to undertake performance measurement to learn more about their smallholder suppliers, it was because they were hoping to learn whether Fairtrade certified smallholder sugar producers in Paraguay were facing any major sustainability risks and whether the farmers were benefiting from trade within this supply chain. This purpose, together with the group’s theory of change, helped them define their performance measurement approach. This group was not looking for specific attribution of impact to their investment in Fairtrade. 2. Clarify your Theory of Change. Theory of Change may seem like complicated jargon, but it is really a simple concept—making clear and explicit the chain of causal logic between your activities (such as trade or a specific investment program) and the outcomes you hope to achieve, highlighting the key assumptions that link them together. How will the activities lead to the desired change in the outcome measures? What needs to happen along the way? What else needs to be true for this to happen? This is important because it can help identify and focus important questions and indicators along the way between where your organization might contribute to change and the longer-‐term outcomes at the farm and household levels. In the example of a fair trade situation, illustrated in Figure 4, the broad causal assumptions are that investment in Fairtrade certification will ensure farmers are well organized, that it will facilitate access to trainings on best management practices, and that they will receive payment of a Fairtrade premium. It is assumed that these three inputs will result in greater farmer empowerment, higher adoption of best management practices, and increased income for farmers. These improvements should help farmers to capture more value for their product, increase farm productivity and invest premiums in farms and community needs. Some M&E language may help you in thinking through your Theory of Change:

• Inputs are the investments you make (capital, time, expertise, and other resources).

8

• Activities are the things you do with the inputs, that is, how you leverage them towards your desired outcomes.

• Outputs are the direct results of those activities and are usually reported quantitatively, e.g. number of farmers trained, or number of trainings held. Outcomes differ from outputs in that outputs are what you do, while outcomes are the difference made by the outputs.

• Short-‐term outcomes are the changes that occur more immediately. The long-‐term outcomes are changes that occur over the long term.

ISEAL has worked with voluntary standards systems on building theories of change, and has articulated the process well in their ISEAL M&E Guidance Note: Getting started on your Monitoring and Evaluation System.

Figure 4. Example Theory of Change for a Company Sourcing Smallholder Fairtrade Sugar 3. Identify your Learning Questions and Indicators. Before diving into specific indicators, it is helpful to develop your Learning Questions. As illustrated in Figure 5, Learning Questions grow out of your theory of change. They are the questions that you are trying to answer through the collection of data. It is important to distinguish between useful questions and interesting questions. Useful questions are ones that will yield actionable data, whose answers will help supply chain actors make informed decisions. Interesting questions are ones that we would love to know the answers to, but have little bearing on what we might do differently. The Paraguay sugar project was an initial investigation into a specific crop and origin. Therefore the Learning Questions focused on developing an understanding of the status of the farms and households, grounded in a theory of what factors influence smallholder success in a fair trade sugar supply chain. Questions included: “Are the basic needs of the farmers being met?” and “Do farmers have satisfactory access to services like training and inputs?” Indicators were then chosen to inform the Learning Questions.

! ! !!!!!!!! !!!!!!!!

…farmers adopt better practices…

…farmers experience increased productivity

and quality

…and train farmers in better farm practices… ...improving the

quality of farmer and community

livelihoods.

Theory of Change

Activities OutputsShort-term Outcomes

Long-term Outcomes

…farmers form producer groups…

…farmers more able to access market…

…and offer farmer group capacity

building…

We will invest in smallholder farmers…

Inputs

9

Figure 5. Learning Questions for Paraguay Sugar Theory of Change

From the Learning Questions, you can look to indicators – the specific factors that you want to measure and potentially track over time. Often, it will take multiple indicators to inform a Learning Question. When you choose indicators, it is useful to strike a balance between focus (the minimum number of indicators needed to answer your questions) and common approaches (the indicators and metrics that are commonly used to gain insight into smallholder agriculture). Using common indicators where possible increases the consistency of analysis and therefore potential learning between studies. It also reduces the burden on suppliers and farmers if other organizations ask for similar indicators.

Figure 6. Getting from Impact Area to Survey Question

There is a substantial body of research on appropriate indicators and metrics for measuring sustainability, but no simple answers. To help provide some clarity and peer-‐review on what indicators are most important to consider under different circumstances, the Sustainable Food Lab worked with a number of organizations to identify common indicators for smallholder sustainability measurement, and common approaches to measuring these indicators. More detailed discussion of specific performance measurement indicators and metrics can be found in, Towards a Shared Approach for Smallholder Performance Measurement: Common indicators and metrics.

! ! !!!!!!!! !!!!!!!!

…farmers adopt better practices…

…farmers experience increased productivity

and quality

…and train farmers in better farm practices… ...improving the

quality of farmer and community

livelihoods.

Theory of Change

Activities OutputsShort-term Outcomes

Long-term Outcomes

…farmers form producer groups…

…farmers more able to access market…

…and offer farmer group capacity

building…

Learning Questions Are farmers realizing the potential of their farms by accessing services and adopting better farm practices? Are they

experiencing increased farm productivity?

Do farmers have strong trading

relationships and access to services?

Livelihoods: Are the basic needs of the farmers being

met?

Gender: Are women participating in the crop and

accessing the benefits?

Environmental Performance: Are farmers stewarding the

land?

We will invest in smallholder farmers…

Inputs

10

Through this review, we found that the most common and important impact areas to monitor in agricultural value chains include Livelihood, Gender, Environmental Performance, Farm Productivity, Access to Services and Trading Relationships.

Table 1. Recommended Performance Measurement Indicators Impact Areas Learning Question Indicator

Livelihood Are the farmers meeting their

basic needs and seeing improvement?

Food Security: Access to sufficient food

Income

Assets

Poverty Likelihood

Perceived Well-‐Being

Gender Are women participating in the

crop and accessing to the benefits?

Women’s participation

Equitable Access to Training Participation in Decision-‐Making

Environmental Performance

Can the land sustain continued cultivation? Adoption of conservation practices

Farm Productivity

Are farmers realizing the potential of their farm?

Trainings Received

Adoption of best practices

Estimated Productivity

Net Crop Income

Access to Services

Do farmers have access to services? Access to credit, training and inputs

Trading Relationships

Are farmers experiencing good trading relationships?

TBD

4. Determine your data collection methodology. Now that you know what you want to learn, and the indicators you must measure against, it is time to think about the logistics of how you are going to gather the data. Will you hire consultants to design and administer your survey? Will you embed the data collection within the supply chain by training staff of a supply chain partner to administer your survey? You will find the specifics around the most important considerations for designing your data collection methodology in Part 2 of this document below. 5. Identifying specific metrics and survey questions. After you have decided how you will collect your data, you can determine the right metrics for your effort, and design a survey from those metrics. For example, if you have determined that you will collect data by sending enumerators to the farmer’s home to interview him or her, your metrics can include a visual assessment of certain farm

11

characteristics. Conversely, if you plan to administer a survey at a farmer gathering place such as a coop or field school, you will choose metrics that are more appropriate for farmer self-‐reporting. The experience of COSA indicates that “survey questions that are designed to be used for multiyear comparisons or cross-‐country comparisons are best when they are specific and not subject to a wide range of interpretation.”3 Keep this in mind as you translate your metrics to survey questions. Who you will interview will also determine how you word your questions. For example, if the farmers in your supply chain are literate and have had some basic schooling, you may be able to ask them questions involving percentages, such as “What percentage of your children between the ages of 6-‐12 attend school more than 80% of the time?” In many cultures and with many farmers, however, this question would be much too awkward. In that context, you’ll want to ask a number of separate questions to get all of the pieces of information you are looking for, and do the math to get your percentage in the data analysis stage. The Paraguay sugar pilot was designed as a household survey and for this reason, we were able to include a different set of metrics than we may have otherwise. For example, interviewing the farmer in the home meant that we could ask him or her questions that required a bit more privacy than one might find at a group gathering place. Being in the home also meant that we could include metrics like the Progress Out of Poverty Index (PPI) that encourages visual confirmation of responses to questions like, “What is the primary roofing material of your house?” After developing your survey, schedule a validation workshop to test the cultural appropriateness of your survey with a group of local people who have firsthand experience in this crop. From this workshop you can gather feedback to adapt the survey to the local context. This will ensure better quality data collection and more accurate results. If you have specific concerns with the survey or approach, test these in the local community before administering the survey. You can do this with a pilot study, or a validation survey. The process of adopting specific survey questions will be based on your data collection methodology and will be iterative. It should change as you learn what produces actionable data and what doesn’t.

PART 2. DESIGNING YOUR DATA COLLECTION METHODOLOGY: IMPORTANT CONSIDERATIONS There are many things to consider when designing your data collection methodology, such as:

1. Who are you collecting the data from? 2. How do you design your sampling methodology so that it is credible and yet still affordable? 3. Where and how will you collect data? For example, will you survey the farmers at their

farms or at a local gathering place, will you use smartphones to collect the data, or will you use paper surveys?

4. What data needs to be qualitative and what data should be quantitative? 5. Who collects that data?

3 3 Understanding Sustainability: First global report on COSA findings in agriculture. COSA, 2013.

12

Beyond these main considerations, it may be useful to think about how you might embed performance measurement within ongoing operations if the aspiration is to collect data on an ongoing basis. You’ll also want to think about how different methodology considerations impact the cost of performance measurement, and how you will analyze your data to maximize learning. To the degree that you can make data collection useful for many actors within the supply chain, especially the producers and producer organizations that the data refers to, you will receive a higher return on your investment in performance measurement. When designing your performance measurement approach, keep in mind the data needs of others in the supply chain and make an effort to include their metrics in your survey. 1. Who to collect the data from: Who you will collect performance measurement data from will depend on the approach you have chosen. For example, if your primary focus is on gathering data around yield and productivity, you may be able to interview the staff of the farmer group and look at data they have collected, rather than talking with individual farmers. If you are looking for information about farmers’ livelihoods, you will need to hear from them directly. Typically, interviews are of the head of the household, because this person tends to know the most about the farm production. In some cases however, when looking for information around gender, and information about the food that the family consumes it is important to interview the female head of the household as well. The most important thing to do when deciding your unit of analysis—who you will interview—is to think about the type of data that you want to gather, and then consider who will most easily, and reliably have this information. Interview that person. Consider any biases that people may have and consider crosschecking the data with another source to look for possible inconsistencies. When designing the Paraguay sugar pilot, we decided to focus on learning the farmers’ perceptions of the quality of their trading relationships, their outlook on the future of growing sugarcane, their food security status, and more. This kind of data requires talking with farmers directly. 2. How to develop a methodology for a sample that is credible and affordable: A sample is the subset of the larger population who will be surveyed. Determining this subset is called sampling. Sampling simply refers to the method of choosing who will be interviewed. M&E relies on statistical methods of sampling, often random. Developing a sampling strategy requires using statistics, but it does not need to be complicated. Many sample size calculators exist to help you determine how many households must be interviewed in order to have statistical power, and therefore confidence in making inferences about the population. The population is the larger group from which your sample is chosen. This is the group about which you will be attempting to make inferences. Again, you can keep things simple by using one of a number of good online calculators for determining a statistically significant sample. There are a few options for how to use these available calculators. One option is to enter the number of interviews that you can afford (we discuss budgeting in the next section) and the total population size and the calculator compute what kind of confidence level that produces, as well as a margin of error (also called confidence interval). A confidence level is usually expressed as a percentage and gives an estimated percent likelihood that data gathered is accurate for the population sampled. The margin of error is the statistical way to express the random occurrence of error in a survey result. To obtain a smaller margin of error, you must have a larger ratio of sample size to population size.

13

Green Mountain Coffee Roasters states that for M&E of their smallholder supply chain investments, “A larger sample size is almost always preferable, but sample size must be balanced against the resources available.”4 In Paraguay we determined that we needed a 95% confidence level with a 6% margin of error. The Fairtrade sugarcane supply base population was made up of 3000 farmers. We used a sample size calculator to determine that we would need to conduct roughly 300 interviews—this was our sample size. Alternatively, we could have determined how many interviews our budget would allow, and entered that in to the calculator along with the population size, to come up with the confidence level and margin of error. To some degree, how you plan to collect data will determine the sample size as well. If you decide to gather data directly from the farmer in his or her home, it might be very difficult to gather hundreds of interviews. Alternatively, if you push a very short set of questions to the farmers via text message, it will be possible to do many more interviews. This is a choice of depth versus breadth. Consider using a simple method of randomization to avoid selection bias in the survey results. “Random selection means that every participant has an equal chance of being selected for inclusion in the sample.”5 If your sample is not randomized, it is possible to end up with data on farmers who are merely easy to reach instead of having a sample that is representative of the population. Often, when working with organized farmers, your target population is organized into subgroups, making stratified randomization an easy choice. This division into subgroups before randomization is called stratification. The randomization then takes place within each subgroup or strata.6 3. Where and how to do the surveying: When you know what kind of data you are looking for, who you will be collecting data from, and how many interviews you need to do, the question of how to collect the data becomes an easy one to answer. Essentially, the more detailed the data, the more important it becomes for enumerators to speak with the farmers directly. For example, if collecting data on environmental metrics such as the number of native trees on the farmer’s property, it will be important to travel to the farmer’s land to conduct the interview. If the survey only asks the farmer five questions, and they all have simple yes or no answers, you might consider using an SMS or voice survey via cell phone. Figure 7 below shows a continuum from lightweight to deeper dive methods for surveying producers.

4, 5 Olson, M.B., Georgeoglou, U., Mendez, E. V., Pino, M. GMCR Monitoring and Evaluation Guide for Supply Chain Outreach Funded Projects. 2012. 6 How to Design an Evaluation. Abdul Latif Jameel Poverty Action Lab. 2012. http:/www.povertyactionlab.org/methodology/how/how-‐design-‐evaluation

14

Figure 7. Continuum of costs and quality in data collection Consider the frequency of collect data collection as part of the determination of where and how to survey. More frequent data collection will be more costly and labor intensive. It may be possible to make up some of these costs—by interviewing farmers at a group gathering point, or using SMS surveys, for example—without too much risk of jeopardizing the credibility of the data if you survey often. There are a few options for how to collect the chosen data, and within those options there are several ways to engage with the respondent:

• Interview farmer groups o Conduct focus groups

with a number of farmers o Interview farmer group

leaders

• Interview farmers directly o Interview farmers at a

group gathering point such as a collection center

o Interview farmers in focus groups

o Interview farmers in their homes

In the last decade or so, a wave of interesting data collection technology has been developed. This technology has opened the door to many new opportunities for reaching farmers at a scale that would have previously been very expensive and time consuming. With increased production of low cost cellular phones, and their rise in popularity, we are now able to reach small farmers via text message (SMS) or voice message quite easily. Mobile devices such as tablets and smart phones also allow the use of apps that have been designed to allow the user to administer surveys in the field far from a power source or an Internet connection. The potential that this technology holds is very exciting, but we want to take caution that we don’t put the proverbial cart before the horse. It is important to develop a sound measurement process that serves your organization’s needs and provides useful data before getting attached to a particular data collection technology. In other words, your Learning Questions, purpose, indicators, and metrics should determine your data collection methodology and subsequently the collection technology. Details on specific data collection technologies and how they might fit your project’s needs may be found in a useful guide titled, Mobile-‐Based Technology for Monitoring & Evaluation linked to in Resources below.

15

• Survey farmers through voice message or text message7 to their phones o Ask farmers to call a number to enroll in the survey program, then call them with a

recording to ask them to respond to simple questions by using the number keys on the phone

o Ask farmers to text a number to enroll in the survey program, then text them a number of simple question that they can respond to with a text.

In years one and two of the Paraguay project, we used mobile tablets to collect data from farmers in their homes. We spoke with the farmers directly in both cases. In year three, we will be administering a shorter survey as a more frequent “check-‐in”. This survey will be administered by producer organization agronomists at a farmer gathering point, such as the cooperative office on the payday for the Fairtrade premium. The plan is that data will be collected this way in year four as well. In year five, we will administer the longer survey again. These regular in-‐home studies, complemented by more modest annual monitoring will serve as a strategy for tracking change over time, and staying ahead of any changes. It is important to keep in mind the effect that interview location can have on the quality of data. It may be the case that being in the presence of other farmers influences the farmer to withhold some kinds of information. Similarly, it can be beneficial to interview farmers in their homes so the interviewer may visually check the surroundings to verify the farmers’ responses. 4. Who collects the data? Equally important as where data is collected, is who does the data collection. Deep dive impact studies use highly trained M&E professionals to collect data from producers. Since performance measurement is intended to be a less resource -‐intensive complement to impact studies, it makes sense to find alternatives to highly trained professionals. Many practitioners of performance measurement are using lightly trained staff of producer organizations, students, local youth and local NGO staff to serve as enumerators. There is still much to learn about the costs and benefits of each. Agronomists and producer organization staff tend to know a lot about the production and sale of the focus crop and in this way they are a good option for enumerators. On the other hand, depending upon the culture, producers may not want to divulge survey information to people they know well. Presumably, producers know the staff of their producer organization fairly well. It is very important to manage for enumerator bias when choosing who will administer the survey. Proper enumerator selection and training, as well as survey controls that promptly identify data entry errors can help you manage bias.8

7 According to the experience of the Grameen Foundation, before planning SMS surveys, it is important to “double-‐check your understanding of literacy rates among your respondents. It is not uncommon for people to indicate that they know how to read and write and for organizations to understand this to mean literate.” Out of Home Data Collection and the PPI. Grameen Foundation, 2013. 8 Understanding Sustainability: First global report on COSA findings in agriculture. COSA, 2013.

16

In Paraguay, we tested two types of enumerators. In the first year of data collection, we used trained research professionals to collect the data. In year two, we trained the agronomists employed by the producer organizations to do the data collection. We found that we saved a lot of time and money by enlisting the help of the producer organizations for data collection. While we were, in a sense, shifting some of the data collection burden on to the producer organizations, many of the organizations found the process useful and planned to use the data and the learning from the collection process (including new ICT tools) to improve the way they work with their farmer members. In some cultures, local youth may serve as good enumerators. Producers may feel comfortable talking with them, they can be eager to work, and likely cost very little. For this scenario it is important to ensure at least a high school equivalent level of education. By using local people involved in the supply chain, the survey method can increase the capacity of these people to continue to collect data in the future. In this way, data collection becomes embedded in the supply chain for the benefit of all. 5. Embedding performance measurement in the supply chain The opportunity to embed a performance measurement program in the operations of a supply chain is attractive to the extent that parties within the chain can benefit from the regular reporting of producer level data. A lot can be learned about the sustainability of the producers in the chain by looking at trends and changes over time. Embedding the data collection in the chain may lead to opportunities for more frequent data collection than occasional collection efforts from outside the system. In supply chains where there is little visibility, it can be hard for performance measurement practitioners to reach their producers. There may not be any traceability in the chain to determine which producers work with specifically within the identified supply chain. In these instances, embedding data collection within the supply chain is unlikely to be possible. Where there is visibility into the chain, and performance measurement practitioners know their producers, coordinated supply chain engagement is necessary in order to successfully embed the data collection in the business systems. Practitioners must work with those often in contact with the producers to understand their systems and the ways they work with smallholders in order to design an approach that does not place too much burden on one party. Some opportunities for embedding data collection include:

• Coordinating data collection with 3rd party standards audits in the case that the producers hold some certification;

• Coordinating data collection with farmer cooperative audits in the case that the farmers are organized in groups that have hired technical auditing staff;

• And, coordinating data collection with ongoing data collection taking place as part of harvest deliveries of producer paydays.

The Paraguay study was designed with the idea that it would be embedded into the supply chain. We tested embedding this ongoing monitoring process by training producer organization agronomists to administer surveys. They were given mobile tablets to use to collect the data electronically and will be surveying farmers in the coming year using these devices and a shorter, more streamlined survey that

17

requires less of their time and resources to administer. Date will be uploaded in to a database, analyzed by SFL, and shared with all relevant supply chain partners. Frequently, when farmers are organized, producer organizations are asked to collect data and report that back up the chain. This is a sensible way of collecting and reporting producer data as the producer organizations are interacting with the farmers on a day-‐to-‐day basis. But the costs associated with collecting farm level data can inappropriate to ask of a producer organization, especially when the organization is already asked to collect data by a number of different groups. It is important to recognize the resources required to collect farm-‐level data, and compensate or incentivize those who collect it. As mentioned above, designing a performance measurement approach to address the needs of producer organizations as well as your own is important. To some degree, the data can serve as an incentive. Costs of Performance Measurement Performance measurement can be a lower cost alternative to deep dive impact studies, but it is not always inexpensive. Initial costs of developing an approach, identifying indicators and metrics and designing a survey can be a substantial up-‐front investment. If the performance measurement program is ongoing however, these costs can be considered an investment over the number of years of the program. Ongoing monitoring does not need to be expensive especially if a performance measurement program has been embedded in the supply chain at some level. The initial costs of a performance measurement approach are typically five-‐fold. Development of approach and survey-‐ This can often be the largest cost of the project. Some

organizations hire consultants to help them develop a performance measurement approach. Supply chain engagement can be an integral part of designing a performance measurement approach.

Data collection software -‐ Data collection software ranges from open-‐source software like Magpi, to

more costly, customized options. The software chosen will be determined largely by the data collection plan and number of interviews planned. For more guidance on choosing data collection software, read Mobile Based Technology for Monitoring & Evaluation listed in resources section below.

Training enumerators-‐ If enumerators will be trained to collect data, there must be a budget for travel

and time for trainers, as well as facilities for training. Data collection-‐ The cost of interviewing will vary widely depending on who does the interviewing and

how they plan to reach farmers. If the survey uses the staff of producer organizations to do the data collection, they need to be paid as they are being paid by the organization. In this case, it would be wise to budget for them to travel to the farmers’ homes or farmer gathering points depending on the collection methodology. Figure 7 above plots the relative costs of various performance measurement methodology considerations.

Data analysis-‐ Data analysis is time consuming and is often better outsourced. Local university students

or interns can provide data analysis as a lower cost alternative to a private consultant.

18

Data reporting-‐ It is not a good idea to skimp on reporting. A good report has some costs, but without it, your data may be ignored. Budget ample time and money for reporting the performance measurement data. Plan to spend some money preparing and presenting data back to the producers who supplied the raw data as well as to other stakeholders.

PART 3. COMPLETING THE LEARNING CYCLE In order to gain the highest return on investment in performance measurement, it is necessary to think of the data as a tool for informed decision-‐making. As COSA explains, “the objective is to utilize reliable information for the purpose of improving policy or business decisions in order to accelerate sustainable outcomes.”9 Transforming the data from a simple spreadsheet into that tool for decision-‐making can be difficult, but to skip this step would mean throwing away the investment. Data becomes a tool through effective analysis and reporting. Analysis Data analysis may be best done by someone with experience with statistics. Many newcomers to data analysis report only on summary data—column averages and sums— rather than data that tells a story. Data is meaningless unless put in context. This can be done by comparing it to benchmarks or other variables or looking for correlation between the data points collected and other variables. Tracking change in a specific variable over time is also effective. When characterizing farmers within a specific supply chain, it is important to look at distributions rather than just averages. Even after removing outliers, averages can be deceptive. It is much more useful to understand the number of farmers that fall within a certain range and how that compares with other farmers. Figure 8 below shows the same data as an average and as a distribution.

9 Understanding Sustainability: First global report on COSA findings in agriculture. COSA, 2013.

“Ultimately the M&E system is about learning and improvement—the systems needs to feed the organization and its stakeholders with the information they need to make decisions about how to improve their work. Internal and external stakeholders will support M&E if they see that it delivers valuable information.” ISEAL M&E Guidance Note, 2013

19

Figure 8. Distributions versus averages

As can be seen from the distribution on the left, the story of land ownership, yield, poverty likelihood and gross income is much more interesting looking at the averages of these figures (on the right). In fact, trends in the data become clear by lining up these distributions, which reveal possible correlations, for example, between land size and yield. Reporting How data is presented to stakeholders will determine the extent to which it is understood and acted upon. A few key reporting tips to remember are:

• Understand the audience. This will determine how to present the data. Poll the intended audience on preferred formatting for reviewing this data is a good idea. Ask what questions they are interested in before analyzing the data to avoid unnecessary analysis.

• Always provide the audience with sufficient contextual information for meaningful interpretation of the data. For example, knowing a farmer’s average daily income is not meaningful without information on local costs of living or national poverty lines.

• Use images to present the data whenever possible. • Tell a story with the data; help the audience to reach conclusions as to what the data means and what the overarching take-‐aways are.

Following the guidance above will enable the data to be turned into a useful tool to make business decisions. Additionally, it is very important to schedule a data review process with the farmers or farm organizations that provided the initial data. After the data has been analyzed, work with the producers to identify the reporting format that will work best for them, and plan an in-‐person presentation of the results that encourages feedback and questions. This is a crucial piece of completing the learning cycle for the organization doing the data collection, and a basic courtesy for those who provided the raw data initially. If there is a chance these producers will be engaged in future data collection as process is embedded within the supply chain, this will be even more important. It is a good idea to incorporate producer concerns and feedback into future iterations of the survey.

20

As with data collection tools, many recent technological developments in the field of data visualization hold exciting potential for reporting and subsequently learning from the data gathered. For example, Geographic Information Systems (GIS) mapping allows the overlay of data over maps, and enable the user to look at correlations between where things are and other important variables such as farmer productivity or the incidence of food insecurity. Tools such as Sourcemap and Geotracability allow for the visualization of supply chain data all the way to the producer.

CONCLUSIONS While this guide goes into some depth explaining how to develop a performance measurement initiative, the most important piece of advice is to get started. Developing a tailored approach, conducting surveys and analyzing results can be a long process. The sooner the approach is determined, the sooner learning from the data and subsequent strategy adaption can begin. Clarity in purpose and the desired change outlined by a theory of change will go a long way in determining the success of the initiative. This careful thought process lays the groundwork needed to ensure that collecting data that will be useful to your organization. With a strong theory of change and a clear purpose, Learning Questions will become apparent. The Learning Questions will then help determine the appropriate indicators, and the indicators will lead to the metrics. More detail on choosing the appropriate indicators and metrics for a tailored performance measurement approach can be found in Taking a Shared Approach to Performance Measurement: Common indicators and Metrics. With metrics in hand, developing a survey that aligns with the budget, timeline and target group appropriate for your organization becomes easy. The key considerations to keep in mind when developing a survey and data collection methodology, are balancing credibility with affordability—what is the most scientifically sound study that can be done within the budget constraints, and developing a process and product that is culturally relevant. Lastly, try to create a feedback loop for learning and improving throughout the process of designing, administering the approach, and analyzing the data. Keep in mind that as much can be learned from the process of data collection as the data itself. And with the data in hand, take special care to report it to stakeholders in a way that is relevant to them and suits their learning styles. After all, learning is really about the most important outcome of this investment. OTHER RESOURCES (NOT CITED): § Taking a Shared Approach to Performance Measurement: Common indicators and Metrics.

Sustainable Food Lab, December 2013. § Assessing the Sustainability of Smallholder Sugar in Paraguay § Mobile Based Technology for Monitoring & Evaluation

performance measurement in smallholder supply chains: a

Documents