VALUE CHAIN MONITORING AND EVALUATION GUIDE

MODULE 8

Integrating Other Design Features into the M&E

System

April 11, 2023

2

GETTING STARTED

OBJECTIVE OF MODULE 8This module focuses on developing select support functions for the system including putting in place data management plans and checks to ensure data quality.

April 11, 2023

3

GETTING STARTED

A good M&E system includes a number of design features:

•Develop data collection instruments.

•Pilot test data collection instruments.

•Arrange data collection logistics.

•Establish data quality oversight procedures.

•Develop electronic database.

•Establish data management procedures.

•Establish data analysis procedures.

•Establish reporting procedures.

•Assign clearly defined roles to everyone involved.

April 11, 2023

4

All M&E system design

pieces developed to this

point.

Senior M&E staff and/or

country program M&E staff

working on related

initiatives

MATERIALS / INPUTS RECOMMENDED

April 11, 2023

5

STEP-BY-STEP GUIDE

1

•Develop data collection instruments

2

•Pilot test data collection instruments

3

•Arrange data collection logistics

4

•Establish data quality procedures

5

•Develop Project Database and Data Entry Procedures

6

•Establish data management procedures

7

•Establish data analysis procedures

8

•Establish information dissemination procedures

9

•Assign clearly defined roles

April 11, 2023

6

STEP-BY-STEP GUIDE: STEP 1Data Collection Instruments:

•Must be developed for every level - observation & measurement

•Must be developed in a participatory way, in collaboration with future users:

• Front-line staff members using them routinely• Project evaluators designing baseline, midterm and endline

•Important that these tools are developed in local languages and tested for accuracy

•Must be accessible to project participants, staff and partners – whoever will be using them

April 11, 2023

7

STEP-BY-STEP GUIDE

1

•Develop data collection instruments

2

•Pilot test data collection instruments

3

•Arrange data collection logistics

4

•Establish data quality procedures

5

•Develop Project Database and Data Entry Procedures

6

•Establish data management procedures

7

•Establish data analysis procedures

8

•Establish information dissemination procedures

9

•Assign clearly defined roles

April 11, 2023

8

STEP-BY-STEP GUIDE: STEP 2

Pilot Test•Should be done prior to full rollout•Identifies inappropriate, poorly worded, or mistranslated questions•Validates the existing coding scheme•Identifies omitted questions•Trains field researchers in tool implementation• Varies in importance

April 11, 2023

9

STEP-BY-STEP GUIDE

1

•Develop data collection instruments

2

•Pilot test data collection instruments

3

•Arrange data collection logistics

4

•Establish data quality procedures

5

•Develop Project Database and Data Entry Procedures

6

•Establish data management procedures

7

•Establish data analysis procedures

8

•Establish information dissemination procedures

9

•Assign clearly defined roles

April 11, 2023

10

Ensure all necessary logistical arrangements, to include:

•Selecting research team members•Transportation to and from the research site•Meals and lodging•Security arrangements•Communication with respondents or potential respondents•Means of communication between research teams and with the head office•Protocols for addressing problems or questions as they arise•Safeguarding of blank and completed information gathering instruments•Storage and transfer of research findings

STEP-BY-STEP GUIDE: STEP 3

April 11, 2023

11

STEP-BY-STEP GUIDE

1 •Develop data collection instruments

2 •Pilot test data collection instruments

3 •Arrange data collection logistics

4 •Establish data quality procedures

5 •Develop Project Database and Data Entry Procedures

6 •Establish data management procedures

7 •Establish data analysis procedures

8 •Establish information dissemination procedures

9 •Assign clearly defined roles

April 11, 2023

12

Establish data quality procedures to prevent:

STEP-BY-STEP GUIDE: STEP 4

Data collection errors

Data tampering

Security lapses

April 11, 2023

13

Quality control procedures that mitigate threats:

STEP-BY-STEP GUIDE: STEP 4

Quality M&E System Training

Random Quality Checks

Routine Refreshers

Collaboration with data collectors

In-field Supervision

Reviews of Data Collection Forms

April 11, 2023

14

STEP-BY-STEP GUIDE

1 •Develop data collection instruments

2 •Pilot test data collection instruments

3 •Arrange data collection logistics

4 •Establish data quality procedures

5 •Develop Project Database and Data Entry Procedures

6 •Establish data management procedures

7 •Establish data analysis procedures

8 •Establish information dissemination procedures

9 •Assign clearly defined roles

April 11, 2023

15

Electronic databases for survey/questionnaire results:

•Common format is to list questions across

columns, and respondents down rows

•Data management programs most often used:

-Microsoft Excel -SPSS

-Microsoft Access

•Should capture info in the order it was asked

•Should include built in data controls

•Should include a ‘codebook’

STEP-BY-STEP GUIDE: STEP 5

April 11, 2023

16

Consolidation of Qualitative Data:•Should be compiled and managed in a central space•Carefully consider data collection tools

STEP-BY-STEP GUIDE: STEP 5

Challenges Lessons learned

Anecdotes Success stories

Easy Access for

the Reader

April 11, 2023

17

STEP-BY-STEP GUIDE

1 •Develop data collection instruments

2 •Pilot test data collection instruments

3 •Arrange data collection logistics

4 •Establish data quality procedures

5 •Develop Project Database and Data Entry Procedures

6 •Establish data management procedures

7 •Establish data analysis procedures

8 •Establish information dissemination procedures

9 •Assign clearly defined roles

April 11, 2023

18

Critical components of data management:

STEP-BY-STEP GUIDE: STEP 6

Data Entry

Data Access

Data Backup

Data Storage

Data Revision

April 11, 2023

19

STEP-BY-STEP GUIDE

1 •Develop data collection instruments

2 •Pilot test data collection instruments

3 •Arrange data collection logistics

4 •Establish data quality procedures

5 •Develop Project Database and Data Entry Procedures

6 •Establish data management procedures

7 •Establish data analysis procedures

8 •Establish information dissemination procedures

9 •Assign clearly defined roles

April 11, 2023

20

Raw Data

STEP-BY-STEP GUIDE: STEP 7

Useful Informati

on

FINAL M&E EVALUATION

AWESOME PROJECT

KENYA

April 11, 2023

21

STEP-BY-STEP GUIDE

1 •Develop data collection instruments

2 •Pilot test data collection instruments

3 •Arrange data collection logistics

4 •Establish data quality procedures

5 •Develop Project Database and Data Entry Procedures

6 •Establish data management procedures

7 •Establish data analysis procedures

8 •Establish information dissemination procedures

9 •Assign clearly defined roles

April 11, 2023

22

Disseminating Data to Internal/External Users:

•Procedures include timing, content and format of information

•Externally, determined by donor requirements

•Internally, co-developed with staff who will use theinformation

STEP-BY-STEP GUIDE: STEP 8

April 11, 2023

23

STEP-BY-STEP GUIDE

1

•Develop data collection instruments

2

•Pilot test data collection instruments

3

•Arrange data collection logistics

4

•Establish data quality procedures

5

•Develop Project Database and Data Entry Procedures

6

•Establish data management procedures

7

•Establish data analysis procedures

8

•Establish information dissemination procedures

9

•Assign clearly defined roles

April 11, 2023

24

Effective operation requires that all involved understand roles and responsibilities clearly, which should be included in:

•Staff job descriptions

•MOUs with implementing partners

•Contracts with external researcher

STEP-BY-STEP GUIDE: STEP 9

April 11, 2023

25

COMMON PITFALLS•Failure to consider all the features necessary for

a functioning and effective M&E system.•Projects do not pilot test data collection tools and instruments.•Projects do not establish data quality procedures or conduct periodic assessments of data gathering activities and data quality.•Projects do not establish data management procedures or plan for staff turnover•Projects do not plan to and/or invest in conducting sufficient staff training

• Including follow up mentoring•Too much focus on donor needs, not enough on other M&E system clients.

April 11, 2023

26

QUESTIONS?

COMMENTS?

April 11, 2023

27

Multiple ways to continue the discussion and continue learning:• Initiate a monthly session on the M&E guide

and case studies from across CARE. Contact cpennotti@care.org

• Join the Market Engagement Community of Practice on LinkedIn.

• Join a task force to review and refine the universal indicators. Contact nardi@careinternational.org

Want to Learn More?