activity monitoring, evaluation, and learning plan

Version 2 submitted March 04, 2021

ACTIVITY MONITORING,

EVALUATION, AND LEARNING PLAN

ROUTINE IMMUNIZATION TRANSFORMATION AND EQUITY

M-RITE Activity Monitoring, Evaluation, and Learning Plan 2

TABLE OF CONTENTS Abbreviations 3

Introduction 4

Alignment with MOMENTUM Knowledge Accelerator 5

Objectives 5

Results Framework 6

Performance Monitoring 8

Indicators 8

Data Management and Reporting 16

Data Management and Quality Assurance 16

Reporting 16

Data Analysis and Use 17

Collaborating, Learning, and Adapting 18

Learning Agenda and Evaluation 19

MEL Roles and Expectations 20

Appendix A: Contextual Indicators 21

Appendix B: Prioritized Performance Indicator Reporting Sheets 26

Appendix C: Learning Agenda, Themes, and Questions 40

Appendix D: Supplementary Indicators 44

References 59


ABBREVIATIONS

AMELP Activity Monitoring, Evaluation, and Learning Plan

CLA collaborating, learning, and adapting

CRaD Country Readiness and Delivery

DHIS-2 District Health Information Software-2

EIR electronic immunization record

EPI expanded program on immunization

FP family planning

GAVI Gavi, the Vaccine Alliance

HCD human-centered design

HMIS health management information system

IA2030 Immunization Agenda 2030

IR intermediate result

MOMENTUM Moving Integrated, Quality Maternal, Newborn, and Child Health and Family Planning and

Reproductive Services to Scale

M-RITE MOMENTUM Routine Immunization Transformation and Equity

M&E monitoring and evaluation

MCSP Maternal and Child Survival Program

MEL monitoring, evaluation, and learning

MKA MOMENTUM Knowledge Accelerator

MNCH maternal, newborn, and child health

PHC primary health care

RH reproductive health

RI routine immunization

TOC theory of change

UNICEF United Nations Children’s Fund

USAID United States Agency for International Development

WHO World Health Organization

XM cross-MOMENTUM


INTRODUCTION

MOMENTUM Routine Immunization Transformation and Equity (M-RITE) is one of a suite of the United States

Agency for International Development (USAID) projects designed to increase the capacity of host-country

institutions and local organizations to introduce, deliver, scale up, and sustain the use of evidence-based,

high-quality maternal, newborn, and child health (MNCH) services; voluntary family planning (FP); and

reproductive health (RH) care. M-RITE will apply best practices and explore innovations to increase equitable

immunization coverage in USAID-supported countries around the globe, with a particular focus on reducing

the number of zero-dose children. M-RITE will also contribute to mitigating the effects of COVID-19 on

immunization services and support the eventual introduction of a COVID-19 vaccine. Working at multiple

levels and in alignment with global strategies such as the Immunization Agenda 2030 (IA2030) and Gavi 5.0,

M-RITE will focus on sustainably strengthening routine immunization (RI) programs by identifying and

addressing persistent bottlenecks and barriers in the planning, delivery, demand, and uptake of

immunization services. The theory of change (TOC) in Figure 1 describes M-RITE’s desired outcomes and their

relationships to the drivers of equitable immunization coverage. The TOC identifies the strategies that M-RITE

will deploy and the contextual challenges (including COVID-19) that influence health outcomes.

FIGURE 1: M-RITE THEORY OF CHANGE

This Activity Monitoring, Evaluation, and Learning Plan (AMELP) describes the tools that M-RITE will use to

support its approaches to monitoring, evaluation, and learning (MEL), and our overall methodology for

generating evidence, supporting adaptive management and learning, and achieving results. M-RITE’s AMELP

reflects the guiding principles of measurement for performance monitoring and accountability, measurement

for learning and program improvement, and measurement for advocacy.


The M-RITE AMELP aligns with USAID’s Collaborating, Learning, and Adapting (CLA) practices to inform

project strategy and implementation to accelerate sustainable progress toward the intended result of

improved immunization coverage and equity. The AMELP aligns with USAID’S ADS 201, the Evaluation Policy,

and the CLA Toolkit.1,2,3 This AMELP was also developed in consultation with MEL counterparts from IA2030

and Gavi, and aligns where possible to those key immunization monitoring and evaluation (M&E) frameworks

to support principles of donor harmonization, alignment with local priorities and processes, and local

ownership in support of the Journey to Self-Reliance, as well as process indicators tested during the Maternal

and Child Survival Program (MCSP). We draw indicators and data management tools from country health

management information systems (HMIS) and other established reporting and data management

mechanisms to support country ownership and use of project-supported data.

ALIGNMENT WITH MOMENTUM KNOWLEDGE ACCELERATOR

In accordance with the letter of commitment between M-RITE and MOMENTUM Knowledge Accelerator

(MKA), the two MOMENTUM projects will work closely together. For example, the AMELP aligns with MKA’s

existing approaches, tools, and indicators, including the cross-MOMENTUM (XM) M&E framework and

learning questions, and guidance on data sharing and data systems. M-RITE’s AMELP benefitted from

available MKA tools including its technical brief on complexity-aware monitoring and adaptive learning toolkit

draft. M-RITE will work with MKA on analysis and dissemination, and on data collection as needed. These

linkages are noted throughout this report.

OBJECTIVES

The main objectives of the M-RITE AMELP are to:

● Support the M-RITE team to monitor and evaluate the project’s implementation and progress

toward its intended and intermediate results (IRs).

● Enable ongoing adaptive learning during activity design and implementation—including the

development and implementation of learning questions—to identify what is and is not working,

make mid-course corrections, and improve the overall impact of M-RITE.

M-RITE’s MEL approach aims to achieve maximum learning that can be applied immediately to the project

goal of improving immunization equity and coverage. Recent stagnation in immunization coverage and

persistent under-immunization among certain populations requires a MEL approach that focuses on the root

causes of under-immunization, for example through new indicators supplemented by qualitative information;

the use of equity-driven analyses to stimulate action; and learning questions that target the most entrenched

obstacles. M-RITE will contribute to improving immunization equity and coverage in USAID-priority countries

and any others that buy into the project. We will collaborate with other stakeholders to strengthen health

system capacity and performance, tracking activities and outputs and measuring changes in outcomes, where

feasible. In the case of national-level outcome and impact-level results, however, it will not be possible to

attribute specific intervention coverage increases and mortality reductions to the project.

The AMELP includes M-RITE’s results framework, performance monitoring indicators, learning agenda, and

complementary M&E strategies. The AMELP also includes the project’s data quality assurance, data

management and reporting systems, and M-RITE’s approach to CLA practices.


M-RITE will update indicators, tools, and our underlying TOC as we learn. The AMELP is a living document and

will be updated yearly as needed. Throughout the project, the senior management team will review

indicators with USAID to ensure they conform to current USAID guidelines, reflect changing context, and

remain relevant for adaptive management.

RESULTS FRAMEWORK All MOMENTUM awards have four identical results and IRs:

● Result 1: Access to and use of evidence-based, quality MNCH/FP/RH information, services, and

interventions scaled-up and sustained.

o IR 1.1: Improved service readiness to provide quality MNCH/FP/RH interventions in public

and private sectors, including emergency care.

o IR 1.2: Improved MNCH/FP/RH practices at the individual, family, and community level.

o IR 1.3: Increased demand for and utilization of quality MNCH/FP/RH interventions and care

by individuals, families, and communities.

● Result 2: Capacity of host-country institutions, local organizations, and providers to deliver evidence-

based, quality MNCH/FP/RH services improved.

o IR 2.1: Increased effectiveness of country institutions and systems to sustainably plan and

manage quality MNCH/FP/RH services and care, including anticipating and responding to

crises, as well as long-term demographic and epidemiologic transitions.

o IR 2.2: Enhanced capacity of country governing bodies to align MNCH/FP/RH policies and

guidelines with international standards and evidence-based, innovative, and promising

practices.

o IR 2.3: Strengthened engagement with civil society, local institutions, and community-based

and faith-based organizations to narrow MNCH/FP/RH equity gaps and quality of care and

hold health system accountable.

● Result 3: Adaptive learning and use of evidence in MNCH/FP/RH programming through sustained

host-country technical leadership increased.

o IR 3.1: Increased appropriate and timely availability and use of data for decision-making in

MNCH/FP/RH policy and programs at the global, regional, and sub-national country levels.

o IR3.2 Increased knowledge generation, translation, and management strategies adopted to

support best practices in MNCH/FP/RH policies and programs expanded at the global,

regional, national, and sub-national country levels.

o IR 3.3: Testing and adoption of innovative practices, including the use of digital health

technologies, to improve MNCH/FP/RH outcomes increased.

o IR3.4 Contributions to USAID’s global technical leadership in MNCH/FP/RH by USAID-funded

partners and USAID-supported countries increased.

● Result 4: Cross-sectoral collaboration and innovative partnerships between MNCH/FP/RH and non-

MNCH/FP/RH local organizations increased.

o IR4.1: International and national public-private partnerships increased.

o IR 4.2: Health partnerships with educational institutions expanded.

o IR 4.3: Health partnerships with corporate and philanthropic organizations increased.

o IR 4.4: Health and non-health organization partnerships expanded.


M-RITE will contribute to the ultimate health impact goal through reduced rates of zero-dose children,

reduced rates of under-vaccination, and strengthened country capacity to mitigate entrenched obstacles and

advance the Journey to Self-Reliance.

The Results Framework in Figure 2 illustrates how implementation of M-RITE’s planned activities are

expected to contribute to the achievement of MOMENTUM IRs and results. In line with the XM MEL

framework, Results 2–4 contribute to achieving Result 1. Activities are designed to contribute to multiple IRs,

recognizing the interdependence within and across activities, IRs, and results. Figure 2 uses multiple arrows

between boxes to demonstrate that multiple activities or IRs are often necessary to achieve a given result.

Feedback loops, particularly from Result 3, demonstrate the need for adaptive learning to optimize

outcomes.

Figure 2 also reflects M-RITE’s holistic and cross-cutting use of project strategies. Guided by systems thinking,

immunization expertise, and a human-centered design (HCD) approach to identify and understand the

perspectives of families, communities, and health workers and the drivers of their immunization-related

behavior, M-RITE uses the following strategies to design and deliver project activities:

● Catalyze country leadership and commitment to equity.

● Foster resilient systems and communities.

● Partner for impact.

● Use data better.

● Maintain, adapt, reinstate, and improve RI services.

FIGURE 2: RESULTS FRAMEWORK


PERFORMANCE MONITORING

M-RITE’s performance monitoring approach identifies indicators and associated data sources along the

results framework to monitor multiple steps of a given activity’s anticipated causal chain, identify potential

gaps and drivers of success, and inform implementation adaptations as early as possible.4 M-RITE’s goal is for

performance monitoring to result in adaptive learning. Occasionally, M-RITE’s approach to performance

monitoring and choice of indicators will serve an advocacy purpose, such as through the inclusion of birth

dose vaccination indicators that M-RITE may not have full responsibility to affect. M-RITE’s learning-oriented

performance monitoring approach will:

● Draw on national- and district-level HMIS/District Health Information Software 2 (DHIS-2) data to

measure its own performance, strengthen existing systems, and avoid duplication.

● Surface relevant, timely, and useful information that can directly influence learning and project

improvement.

● Enable ownership and engagement of project team members and other stakeholders who are

responsible for, can learn from, and can take action to improve activity design, implementation, and

outcomes.

● Ensure team understanding of process and context so that why something works, or does not, is as

apparent as whether it works.

M-RITE’s performance monitoring approach is informed by best practices from causal link monitoring to

maximize adaptive learning and management.4 We will use principles and tools from causal link monitoring

to identify inputs, processes, and assumptions necessary to achieve desired outputs and outcomes; improve

activity design based on this reflection; and monitor and adapt as real-world chains unfold. We will focus

causal link monitoring efforts on activities in which the causal chains have greater uncertainty (e.g., because

the intervention or system is complex), and will build detailed logic models with staff and stakeholders to

identify assumptions necessary for the achievement of the desired outcomes. This will help identify process

and contextual indicators that are most critical to monitor during activity implementation, which in turn will

enable faster adaptive management of changes and unintended consequences. Where program design

allows, M-RITE will use quasi-experimental and qualitative methods such as interrupted time-series,

difference-in-difference, and contribution analysis to estimate M-RITE’s contribution to observed changes in

outcomes. Rich qualitative data and learning-focused case studies will help to explain observed indicator

trends or estimates of M-RITE’s contributions.

In the dynamic context of COVID-19, the use of complementary and complexity-aware monitoring will be

critical to identify emerging causal pathways and change mechanisms within this project and immunization

programs (e.g., to monitor and understand the unintended consequences of revised service delivery

guidelines or new immunization service delivery models). Appendix A includes indicators and data sources to

monitor context including social, economic, and political drivers, and indicators to monitor COVID-19’s

consequences on demand for and care-seeking of immunization services, vaccine and health worker

availability, patient access to clinics, and service quality.

INDICATORS

The AMELP includes performance monitoring indicators that will be used to measure outputs, initial outcomes, and outcomes that contribute to or reflect IRs and results at baseline, and then to monitor their change in response to M-RITE activities. In addition to these traditional performance indicators, we also include indicators that are included for advocacy purposes (marked with an “A” in the table) and indicators


that will be used primarily for learning purposes (marked with an “L” in the table). Indicators that require tailoring to country needs and context to be most useful are marked with a “T”. The indicators are organized according to results and reflect anticipated life-of-project activities and outcomes. The indicator table below contains information on indicator type, the IR it measures, data source, and USAID reporting frequency. The Performance Indicator Reference Sheets (PIRS) in Appendix C provide full definitions, rationale for indicators, data sources, and disaggregation (addressing the project’s core principles). Many indicators were drawn from partners such as Gavi, IA2030, MCSP, the UNICEF Equity Reference Group, as well as relevant indicators from the XM MEL framework. M-RITE will also monitor supplemental performance indicators (Appendix D) and activity-specific monitoring indicators. These indicators, while not reported to USAID, will be used to monitor the project and help to tell the M-RITE story. The AMELP recognizes indicators may be modified according to countries’ indicator definitions and data

availability. M-RITE will use routine data (e.g., HMIS, DHIS-2, electronic immunization records [EIRs]) where

possible to support near real-time AMEL, but will supplement it with estimates and survey data, particularly

in countries in which data quality is a significant concern.

TABLE 1: M-RITE PERFORMANCE MONITORING INDICATORS

Code Indicator name (* indicates dependent on country scope)

Type Definition notes (see PIRS for full definitions)

Data source Indicator source

Disaggregation Reporting frequency

Result 1: Access to and use of evidence-based, high-quality MNCH/FP/RH information, services, and interventions scaled-up and sustained

1.0.1 Dropout between first dose (DTP1) and third dose (DTP3) of DTP-containing vaccines in M-RITE- supported areas.±

Outcome The difference between the number or percent of children <12 months who received DTP1, but not DTP3 vaccine.

HMIS XM By SDP, country, urban/rural, district

Annual

1.0.2 Number or percent of children <12 months who received DPT3/Penta 3 vaccine in M-RITE -supported areas.

Outcome HMIS XM By SDP, country, urban/rural, district

Annual

1.0.3 Number or percent of surviving infants who received 1st dose of measles-containing vaccine (MCV) in M-RITE -supported areas.

Outcome

HMIS XM By SDP, country, urban/rural, district

Annual

1.0.4

Number or percent of surviving infants who received 2nd dose of measles-containing vaccine (MCV2) in M-RITE -supported areas.

Outcome HMIS XM By SDP, country, urban/rural, district

Annual

1.0.5 Number or percent of surviving infants who did not receive 1st dose of DPT/Penta vaccine in M-RITE-supported areas.

Outcome Zero-dose indicator

HMIS IA2030 By SDP, country, urban/rural, district

Annual

1.0.7 Number or percent of live births who received a birth

dose of BCG (BCG0) in M-RITE-supported areas. (A)

Outcome HMIS IA2030 By SDP, country, urban/rural, district

Annual

1.0.8 Equity: DTP3 coverage among historically unreached

communities or group. (T)

Outcome This indicator will be tailored to national and sub-national context and “communities” or “groups” will depend on a specific national or sub-national drivers of under-immunization or zero-dose.

HMIS or survey

Depends on country tailoring

Annual

IR 1.1: Improved service readiness to provide quality MNCH/FP/RH interventions in public and private sectors, including emergency care


1.1.1 Percent of M-RITE-supported districts (or facilities) reporting full stock availability at service delivery level during re-supply cycle or reporting period.

Initial outcome

Full stock availability is defined as the absence of any stock-out for vaccines and supplies.

LMIS, HMIS, or other stock databases

IA2030 LMIS, HMIS, or other stock databases

Annual

1.1.3 Percent of health facilities with evidence of utilization of an up-to-date and quality microplan to identify at-risk or vulnerable communities and children.*

Initial outcome

"Up-to-date" will be defined according to the national recommendations and context. “At-risk” will be defined according to national and sub-national context and M-RITE’s assessment.

Project records

MCSP process indicators

Country, district, SDP type

Annual

1.1.6 Percent of health workers in M-RITE-supported facilities who can explain how to identify and reach zero-dose and under-immunized children.*

Initial outcome

Project records

Country, district, SDP type, gender of health worker

Annual

1.1.7 Percent of M-RITE-supported health facilities who improve their readiness to reach zero-dose and under-immunized children with high-quality immunization services. (T)

Outcome “Readiness” will be defined in country PMPs based on needs identified through M-RITE’s baseline assessment.

Project records

Country, district, SDP type, public/private

Annual

IR 1.3: Increased demand for and utilization of quality MNCH/FP/RH interventions and care by individuals, families, and communities

1.3.1 Percent of M-RITE-supported areas with activities led by the EPI program designed and implemented to overcome specific vaccine demand- or utilization-related barriers to immunization in high-risk communities.*

Initial outcome

"High risk" will be defined at the level of planning and implementation, ideally district or facility through M-RITE’s co-creation process.

District-level EPI program workplan reviews, rapid telephone surveys

IA2030 Country, district, type of SDP (as relevant)

Annual

Result 2: Capacity of host-country institutions, local organizations, and providers to deliver evidence-based, quality MNCH/FP/RH services improved, institutionalized, measured, documented, and responsive to population needs

2.0.2 Percent of national and sub-national immunization stakeholders whose capacity* was strengthened** by M-RITE-supported capacity strengthening activities. (T) *Capacity will be defined and tailored to specific country contexts

Output Capacity building interventions are defined as any forms of interventions or activities that improve the ability of a person, group, or organization to meet objectives or to perform better. This indicator counts those stakeholders who were reached by such interventions.

Project records

Country, stakeholder type

Annual


IR 2.1: Increased effectiveness of country institutions and systems to sustainably plan and manage quality services/care, including anticipating and responding to crises, as well as long-term demographic and epidemiologic transitions

2.1.3 Evidence of increased use of sub-national (e.g. district-level) processes and tools* to regularly review and address immunization performance and equity, including gender equity, and to anticipate and respond to change. (T)

Initial outcome

*Regular review processes and tools will be defined and tailored to country context but may include: quarterly review meetings, data review meetings, effective planning processes, uptake of new global guidance, microplanning, use of costing or budgeting tools.

Project records

Country Annual

IR 2.2: Enhanced capacity of country governing bodies to align policies and guidelines with international standards and evidence-based, innovative, and promising practices

2.2.2 Number of global, national, and sub-national policies (including strategies, guidelines, operational plans)

developed with M-RITE support related to immunization

services or MOMENTUM cross-cutting areas.

Initial outcome

Project records

MCGL Country, type of policy, topic (e.g., COVID-19 vs. non-COVID-19)

Annual

2.3.1 Percent of M-RITE-assisted local organizations with improved performance.*

Outcome Capacity is the ability of people,

organizations, and society as a whole to

manage their affairs. Capacity

development is the process of

strengthening and maintaining such

capacity. Capacity is a form of

potential; it is not visible until it is used.

Therefore, performance is the key

consideration in determining whether

capacity has changed. Organizations

with improved performance will have

undergone a deliberate process to

improve execution of organizational

mandates to deliver results its

stakeholders.

M-RITE will measure improvement

organization using already existing

tools such as university accreditation

Project records

USG CBLD-9

Country, Type of organization

Annual


process, ISO professional farmer

organization guidelines, OCA tool, etc.).

And for monitoring and measuring

changes on a key performance metric

(e.g., increased sales, reduced staff

turnover, increased efficiency of

service delivery, etc.).

Result 3: Adaptive learning and use of evidence in MNCH/FP/RH programming through sustained host-country technical leadership increased

3.0.1 Number of evidence products, tools, or data analyses produced by M-RITE that include actionable recommendations and an equity and gender equity lens.

Output "Actionable recommendations" will be measured through existing tools.

Project records

MKA Topic (e.g., COVID-19 or non-COVID-19), country

Semi-annual

3.0.4 Evidence of M-RITE adaptive learning or HCD approaches contributing to improved design or implementation of global, national, or sub-national immunization policies, programs, or interventions.

Outcome Case studies n/a Annual

3.0.5 Evidence of increased use of equity-related data and evidence by M-RITE-supported EPI programs.


IR 3.1: Increased appropriate and timely availability and use of data for decision making in MNCH/FP/RH policy and programs at the global, regional, and sub-national country levels

3.1.7 Evidence of increased use of routine immunization data for EPI-program and policy decision-making.


IR 3.2: Increased knowledge-generation, translation, and management strategies adopted to support best practices in MNCH/FP/RH policies and programs expanded at the global, regional, national, and sub-national country levels


3.2.2 Number of individuals reached through M-RITE co-creation, dissemination, knowledge exchange, and knowledge translation activities and events.

Initial outcome

Individuals are counted in “reach" if they were the active participant in an M-RITE co-creation or knowledge-related event. Active participation could be defined as those people who actively have to choose to engage in content - by attending meetings, opening e-mails, logging into webinars, etc.

Project records

Similar to MKA

Organization, gender (where possible), country

Semi-annual

IR 3.3: Testing and adoption of innovative practices, including the use of digital health technologies, to improve MNCH/FP/RH outcomes increased.

3.3.1 Number of innovations supported through M-RITE assistance.*

Output "Supported" refers to technical, in-kind, or financial support from M-RITE related to the introduction, testing, or adoption of innovations.

Project records

USG STIR10

Country, topic, new or ongoing support, innovation phase

Annual

IR 3.4: Contributions to USAID’s global technical leadership in MNCH/FP/RH by USAID-funded partners and USAID-supported countries increased

3.4.2 Number of global and regional documents (guidelines, strategies, tools, etc.) that M-RITE contributes to.

Initial outcome

Project records

Topic (e.g., COVID-19 or non-COVID-19), new or ongoing

Semi-annual

Result 4: Cross-sectoral collaboration and innovative partnerships between MNCH/FP/RH and non-MNCH/FP/RH organizations increased

4.0.2 Number of multi-sectoral collaborations formed or strengthened at global, regional, national, and sub-national levels as a result of M-RITE support to address immunization coverage and equity issues.*

Initial outcome

A "collaboration" is a relationship between two organizations working together with the goal of jointly producing an output or contributing to a common goal.

Project records, case studies

Collaboration type (aligned with IRs), topic, country, new (formed) or existing (strengthened), level

Annual

4.0.3 Number of partnerships formed or strengthened at global, regional, national, and sub-national levels to address immunization coverage and equity issues.*

Initial outcome

Project records, case studies

Collaboration type (aligned with IRs), topic, country, new (formed) or existing (strengthened), level

Annual

Table notes: ± "M-RITE-supported areas" will be defined based on the scope of M-RITE's engagement in a country and that country's specific administrative units.

*Indicators dependent on scope of country awards.


DATA MANAGEMENT AND REPORTING

DATA MANAGEMENT AND QUALITY ASSURANCE

M-RITE will implement a high-quality data management system that offers consistent data collection,

systematic data cleaning, and secure data storage. The project will develop and maintain a project database

on a DHIS-2 instance that is interoperable with the MKA XM data system. The system will follow the

standards of the data exchange process between MKA and M-RITE to enable data exchange and integration.

These data will be entered manually but the exchange will be automatic, as guided by the document. The M-

RITE database is also being designed based on lessons from the development of the MKA data system. The

project database will include baseline values for some performance indicators, target values for each

indicator once the final AMELP is approved, and definitions of the indicators. For service delivery-related

indicators, including those from HMIS or logistic management information systems (LMIS), the unit of

observation in most cases will be a health district (or similar administrative unit), the lowest level of data

aggregation that is most readily available in most countries’ HMIS. Activity leads will input aggregate data

from various sources into the database quarterly.

To ensure that information gathered is credible and usable, it must be extracted from a high-quality reporting

system. Data quality measures necessitate that data are accurate, reliable, timely, relevant, and complete.

Such parameters require placement of data quality measures throughout the data flow and management

system. M-RITE will design a system that ensures internal data quality checks. We will design standard

operating procedures for data collection, entry, and storage. In addition, a list of internal data quality checks

will be built into the data storage system to ensure consistency across each indicator. We will pay special

attention to additional data quality issues because of COVID-19 interruptions. M-RITE’s senior management

team will review data for completeness on a quarterly basis, and conduct spot checks to ensure data quality.

Data validation dashboards will be used to catch reporting-level errors.

M-RITE will continually consider known data quality issues in immunization. M-RITE and XM indicators

related to vaccine coverage suffer common data quality issues in many countries. One is that population

denominators (i.e., the estimated target population) are often inaccurate, leading to inaccurate coverage

estimates. To avoid this, M-RITE will report both the number of vaccines administered (numerator) and

coverage (numerator/denominator). Where applicable, we will apply denominator correction formulas

developed during MCSP.5 To avoid misinterpreting late or under-reporting into DHIS-2 during COVID-19, M-

RITE will track data reporting indicators to develop country- or district-level approaches to reporting and

interpreting data. M-RITE will also support data quality assessments and improvement plans where feasible

and where data quality and use is a significant bottleneck not only to M-RITE performance monitoring, but to

equitable immunization coverage more generally.

REPORTING

The M-RITE data system will be used to compile data on the indicators above at the frequency drafted in the

indicator table. Indicator reporting to the XM database will be guided by MKA guidelines and timelines. Data

systems will be developed to ensure automated and ongoing transfer of reports to the MKA system.

Quarterly reports will be drafted and sent to USAID in narrative and presentation format within 30 days of

the end of the quarter. The report format was designed in collaboration with the agreement officer team at

USAID. The quarterly report will provide M-RITE activities and key accomplishments and challenges under


each result area. Along with reporting performance indicators, the report will also report on learning for each

quarter. Finally, additional USAID-requested reports will be written as needed.

DATA ANALYSIS AND USE M-RITE’s data analysis and use approaches and practices aim to serve three main purposes:

1. Track progress toward stated results by comparing observed to intended changes (difference

between target and baseline). We will have a project-wide agreement on performance and a

communication tool to discuss challenges and opportunities to inform adaptive project

management.

2. Support adaptive learning by integrating contextual data and insights with quantitative data so we

can identify why and how we are achieving anticipated results or lagging behind.

3. Identify equity gaps by performing sub-group analyses, particularly by known equity

bottlenecks/dimensions like wealth quintile, to ensure we get closer to our goal of increasing access

to services for zero-dose children and at-risk populations.

STRENGTHENING M-RITE CAPACITY AND CULTURE FOR DATA ANALYSIS

AND USE

As part of Result 3, M-RITE will apply strategies to improve data use in the programs where we work. We will

also apply these strategies to our own data practices to facilitate adaptive learning and improve the overall

effectiveness of M-RITE support. These strategies aim to strengthen capabilities, improve behaviors and

motivations, and create better opportunities through updated systems and processes to use data. M-RITE’s

data system will include built-in dashboards with decision-support cues to make information more available

and actionable. All project staff will be trained to use these tools and will self-assess their data use and

interpretation skills in order for the MEL team to offer capacity strengthening where needed. M-RITE will

explore the use of behavioral nudges to increase analysis, use, and adaptive learning with the team’s

behavioral science and HCD partners.

USING NOVEL ANALYTIC APPROACHES

M-RITE’s emphasis on equity and zero-dose children will require analytic approaches that use meaningful

disaggregation and different measurement approaches.6,7 Performance monitoring indicators will be

disaggregated by variables with known equity dimensions, as noted above, in the project data dashboards

and all reports to ensure we are closing equity gaps. We will measure the IA2030 indicator for geographic

coverage inequality through district-level disaggregation and will track progress on narrowing geographic

inequalities by tracking cohorts of low-coverage districts over time. We will use district-level data to

identify districts with the greatest number or proportion of zero-dose children to initially identify where

M-RITE should work. Because neither survey nor administrative data offer a complete set of variables

related to equity, we will triangulate both when feasible and look for proxy indicators in existing or novel

data collection platforms.8 Where child-level data are available (e.g., through EIRs), we will leverage them

for more granular and real-time monitoring of performance indicators and their equity disaggregation.9

When available, we will use EIR data to measure immunization timeliness during and after COVID-19 and

to inform efforts to identify and catch-up missed children.


Our approach to equity analysis and disaggregation extends to process indicators in addition to outcome

indicators; for example, disaggregating service readiness indicators such as vaccine supply availability by

rural/urban and facility type as part of our focus on improving the equitable distribution of vaccines and

supplies. Where sex disaggregated data are available, we will use them to measure gender inequity in

coverage rates, particularly in countries that have historical evidence of coverage differences between girl

and boy children; or where these differences have emerged during COVID-19 (e.g., in Pakistan).

COLLABORATING, LEARNING, AND ADAPTING All aspects of M-RITE’s MEL and implementation approaches align with USAID’s CLA practices to inform

project strategy and implementation to accelerate sustainable progress toward improved immunization

coverage and equity. In the context of COVID-19, CLA is more important than ever as it can help M-RITE and

its stakeholders prepare for and respond to shifting context, priorities, resources, and partners. M-RITE uses

CLA and adaptive learning practices and processes along the program cycle, as illustrated in Figure 3.

FIGURE 3: SIX ESSENTIAL PROCESSES ENABLE ADAPTIVE LEARNING AT THE PROJECT AND

ORGANIZATIONAL LEVELS, FROM MKA’S DRAFT ADAPTIVE LEARNING TOOLKIT

During activity design, M-RITE uses co-creation, supports use of evidence and data, and enables evaluative

thinking to identify critical assumptions. For example, the M-RITE MEL team will support activity leads to

use causal link monitoring as part of activity design, particularly for activities that involve many

assumptions or for which the evidence base is weaker.4

During routine monitoring, our choice of indicators and data management systems ensures that the right

data are collected and presented at the right time for decisions on program strategy, scale up, and adapting

to other locations and contexts. These data and insights will be discussed in quarterly data review and action

meetings with MEL and activity staff. M-RITE will use these meetings to assess progress and situation, adjust

or adapt strategies and priorities, reflect on learning questions, and propose alternatives to achieve better

results. We will encourage participants at all levels to reflect on qualitative information and consider the

interrelationships, perspectives, and boundaries within the system that may affect implementation and how

these affect the proposed approaches and questions.

Other global, regional, and national learning fora will include pause and reflect sessions, learning cafes, and

virtual conferences to share real-time as well as synthesized insights, emerging evidence, and knowledge. As

M-RITE begins to work in multiple countries, we will identify opportunities for cross-country knowledge

sharing and learning.

M-RITE will communicate results from knowledge management and communication activities through a

variety of activities.

LEARNING AGENDA AND EVALUATION The primary objective of M-RITE’s learning agenda is to improve program design and implementation

through ongoing analysis and reflection on key questions. The aims of M-RITE’s learning agenda mirror


those of the XM learning agenda:

● Capture relevant, prioritized learning that cuts across the suite of awards.

● Test and explore assumptions and hypotheses laid forth by the M-RITE and XM TOC.

● Fill knowledge gaps critical for the MOMENTUM suite to make informed decisions, including

course corrections.

● Generate evidence for MOMENTUM and its partners.

● Help fill gaps in knowledge important to the USAID portfolio and to the larger body of

stakeholders at country and global levels.

Alignment with MKA’s aims and specific learning questions will facilitate XM learning. M-RITE will

participate in the XM MEL working group and learning sub-group to share learnings and coordinate efforts

to gather and synthesize information. M-RITE will work with Gavi to identify concrete opportunities to

partner on specific learning questions and lessons for applying an adaptive learning approach more

broadly.

Our learning questions are organized under the following themes:

● Equity (including zero-dose/under-immunized).

● COVID-19.

● Health systems, resilience, and life-course/integration. ● Demand, trust, and community. ● Co-creation/HCD/adaptive learning/innovation. ● Capacity strengthening/human resources. ● Partnerships.

The full list of learning questions is in Appendix C. M-RITE’s learning agenda includes: 1) externally relevant questions that aim to fill some of the most important knowledge gaps in immunization today, and 2) internally relevant questions to assess M-RITE’s contributions and the validity of the underlying M-RITE and XM TOC. M-RITE’s learning agenda reflects our consortium’s expertise and considers where we can contribute best to global, regional, and national conversations related to evidence.

We anticipate answering many of the external-facing questions through workplan activities using

approaches such as systematic and realist reviews and triangulation of secondary data. For internal-facing

questions, we will largely use causal link monitoring and other process evaluation tools embedded in case

studies to understand why and how a given project strategy works. The MEL team will adapt existing tools

such as ‘most significant change’ and realist frameworks to synthesize emerging learnings. M-RITE will

discuss and reflect on insights, evidence, and knowledge during the learning fora listed in the CLA section.

EVALUATION

We define evaluation as a systematic way to generate evidence related to whether, how, and why

something works. M-RITE’s approach to adaptive learning draws on many approaches from

developmental and process evaluation, noting that a range of approaches can be used to generate

evidence for decision-making. However, we will transition to a more traditional evaluation mindset for

certain learning questions and evaluation topics when the evidence threshold to move forward with the

decision in question is high (e.g., there is extensive uncertainty about an intervention), or there is a high

risk of bias with less systematic learning approaches. We will identify these priorities and needs with key


stakeholders and propose developmental evaluation approaches where possible to strengthen adaptive

learning capacity in countries.

M-RITE anticipates that the evaluation of innovations will be a priority for many stakeholders, and where

feasible we will propose quasi-experimental approaches to evaluating innovations as a less expensive

alternative to experimental designs. We will draw from the discipline of process evaluation and recent

advances in the evaluation of complex interventions and/or complex systems to understand why and how

something is effective or not. As needed, we will incorporate economic evaluation to support countries’

Journeys to Self-Reliance.

MEL ROLES AND RESPONSIBILITIES The MEL advisor, working closely with other MEL staff as appropriate, will be responsible for AMELP

implementation. Within the project, s/he will work in partnership with the project senior management team,

including the MEL, immunization technical, and results leads; senior knowledge management strategist;

project director; and team members. The MEL lead will engage with MKA MEL team members, project

stakeholders, and USAID to implement the AMELP across all project activities while responding to activity

learning and changes in USAID priorities, strategies, and external events.


APPENDIX A. CONTEXTUAL INDICATORS

Indicator Data source Disaggregation Reporting

frequency

Demographic

Total population World Bank WDI Total Annual

Total births World Bank WDI Total Annual

Mortality World Bank WDI Total Annual

Number under-5 child deaths WHO Age, sex Annual

Under-5 mortality rate WHO Age, sex Annual

Number neonatal deaths WHO Age, sex Annual

Neonatal mortality rate WHO Age, sex Annual

Number infant deaths WHO Age, sex Annual

Infant mortality rate WHO Age, sex Annual

COVID-19 DATA


# of COVID-19 cases in previous three months HMIS/COVID-19 tracking

(secondary data)

N/A Quarterly

# of COVID-19-related deaths in previous three

months

HMIS/COVID-19 tracking

(secondary data)

N/A Quarterly

# of COVID-19 cases in health workers in previous

three months


(secondary data)

Cadre Quarterly

# of COVID-19-related deaths among health

workers in previous three months


(secondary data)

Cadre Quarterly

COVID-19 policy stringency score Oxford COVID-19

Government Response

Tracker

N/A Annual

National recommendations related to the delivery

of immunization services during COVID-19

COVID-19 Policy Tracker N/A Annual

Reach

Number of districts with M-RITE support for EPI

programs

Program records N/A Annual

Total population in areas with M-RITE support for

EPI programs

Estimate/program

records

N/A Annual


Number of health facilities with M-RITE support

for EPI programs

Program records Type of SDP Annual

Number of organizations receiving M-RITE support Program records Type of organization (government,

parastatal, private [CSO, NGO, for-

profit]), type of USAID partner (new,

existing)

Annual

Context

Number of instances of stresses and shocks in [1]

M-RITE-supported areas

Program records Type of stress or shock Quarterly

%/# of M-RITE-supported health facilities with no

[2] stockouts of any tracer stock for vaccine in the

past three months

HMIS/LMIS/survey/

program records

Type of SDP; urban/rural;

numerator/denominator

Quarterly

Distance from nearest health facility DHS Urban/rural; sub-national unit; wealth

quintile; maternal education

Annual

Number of measles cases VPD surveillance Sub-national unit Annual

Gender and equity

% of women who report shared or full ability to

have the final say in decisions about their child

DHS Urban/rural; sub-national unit; wealth

quintile; maternal education

Annual


Difference in DTP3 coverage by wealth, maternal

education, and rural/urban

DHS Annual

Service organization and health systems

% of vaccine doses that are delivered through

fixed-sites (e.g., facility) versus outreach-based

services

Country multi-year plan Facility, outreach-based services Annual

Daily vaccination clinics available? Country multi-year plan Type of SDP, if available Annual

Number of antigens in the national EPI schedule Joint Reporting Form N/A Annual

% of health budget spent on immunization JRF N/A

% of health care providers who received a

supervision visit in the past quarter

SPA/SARA Type of provider; SDP type; sub-

national unit; provider gender

Annual

% of health care providers who were trained on

immunization safety in last 2 years

SPA/SARA Type of provider; SDP type; sub-

national unit; provider gender

Annual

Nurses per capita World Bank WDI Annual

Demand and community


% of caregivers with knowledge about vaccines

and the recommended schedule


quintile; maternal education;

maternal parity

Annual

% of caregivers who trust the safety and efficacy

of vaccines



maternal parity

Annual

% of children whose mothers intend to vaccinate

their child



maternal parity; child sex

Annual


APPENDIX B. PRIORITIZED PERFORMANCE INDICATOR

REPORTING SHEETS

Indicator 1.0.1 Dropout between first dose (DTP1) and third dose (DTP3) of DTP-containing vaccines in MOMENTUM-supported areas

Rationale XM core indicator. Recommended by IA2030. This is an established indicator that is one measure of the construct of "under-immunization" and a proxy measure system accessibility and service quality.

Precise definition The difference between the number of percent of children<12 months who

received DTP1, but not DTP3 vaccine.

Data source(s) HMIS

Data disaggregation SDP, country, district, urban/rural

Data collection and reporting frequency

Annual

PY1 target TBD upon country buy-in

Individual responsible for reporting data

Reporting by in-country M&E leads

Indicator 1.0.2 Number or percent of children <12 months who received DPT3/Penta 3 vaccine in M-RITE-supported areas

Rationale

This is an established indicator for vaccination coverage, included in IA2030 and XM lists. M-RITE acknowledges that while it is the most commonly used indicator for coverage, it does not represent full coverage of all antigens, nor does it reflect effective coverage of DPT-containing vaccines.

Precise definition Number/percent of children <12 months who received DPT3/Penta 3 vaccine

in M-RITE-supported areas.

Data source(s) HMIS




Annual




Indicator 1.0.3 Number or percent of surviving infants who received 1st dose of measles-containing vaccine in M-RITE-supported areas

Rationale XM core indicator begins to measure vaccination coverage across the life-course as 1st dose of measles-containing vaccine is administered at 9 months of age.

Precise definition Number/percent of surviving infants who received 1st dose of measles-

containing vaccine in M-RITE-supported areas.

Data source(s) HMIS



Annual




Indicator 1.0.4 Number or percent of surviving infants who received 2nd dose of measles-containing vaccine (MCV2) in M-RITE-supported areas

Rationale XM core indicator measures vaccination coverage across the life-course as MCV2 is administered at 18 months of age.

Precise definition Number/percent of surviving infants in previous year’s birth cohort who

received MCV2 in M-RITE-supported areas.

Data source(s) HMIS




Annual




Indicator 1.0.5 Number or percent of surviving infants who did not receive 1st dose of DPT/Penta vaccine in M-RITE-supported areas

Rationale

This is the definition of zero-dose children used by IA2030. It aims to measure how many children did not receive the first dose of a DTP3-containing vaccine. It could indeed be the case that they received birth doses of BCG or HepB, or oral polio. It is calculated by subtracting the # of children who received DTP1 from the estimated surviving birth cohort to arrive at the number who did not receive it.

Precise definition Number/Percent of surviving infants who did not receive 1st dose of

DPT/penta vaccine (DTP1) in M-RITE -supported areas

Data source(s) HMIS



Annual




Indicator 1.0.7 Number or Percent of live births who received a birth dose of BCG (BCG0) in M-RITE-supported areas.


Rationale

This indicator measures the administration of BCG vaccine within the first 24 hours of a newborn’s life. Because it is delivered in maternity wards, it is not traditionally the responsibility of the EPI program, but it is important to measure and track for advocacy purposes. M-RITE will not be held accountable for performance on this indicator.

Precise definition

Numerator: Number of newborns receiving BCG in the first 24 hours of life

Denominator: Number of live births

Definitions may vary by country as this is a less-established routine M&E

indicator in many national HMIS systems.

Data source(s) HMIS



Annual




Indicator 1.0.8 Equity DTP3 coverage among historically unreached communities or group

Rationale This tailored indicator will measure changes over time in DTP3 coverage among communities or groups that M-RITE identifies as being under-served, socially marginalized, or at-risk of being under-vaccinated for other reasons.

Precise definition

This indicator will be tailored to national and sub-national context and

“communities” or “groups” will depend on a specific national or sub-national

drivers of under immunization or zero-dose.

Data source(s) HMIS, Survey depending on country

Data disaggregation Country, district, urban/rural



Annual



Reporting by in-country M&E leads.

Indicator 1.1.1 Percent of M-RITE-supported districts (or facilities) reporting full stock availability at service delivery level during re-supply cycle or reporting period

Rationale Stockout is a common driver of under-vaccination, so this indicator will identify areas in need of supply chain related support, and measure progress in stock availability over time.

Precise definition Full stock availability is defined as the absence of any stock-out for vaccines

and supplies.

Data source(s) LMIS, HMIS, or other stock databases

Data disaggregation Country, district


Annual




Indicator 1.1.3 Percent of health facilities with evidence of utilization of an up-to-date and quality microplan to identify at-risk or vulnerable communities and children

Rationale

A microplan defines how to reach clients, how many people should be targeted for services in the area, and how frequently high-quality services are provided. A microplan is developed by all stakeholders at each level. An effective micro-plan will support health facilities and district teams to: 1) identify target population; 2) design data and graphic mapping; 3) prioritize plans to reach target population; 4) define realistic actions; 5) reduce inequity and improve the quality of immunization services.


Precise definition

"Up-to-date" will be defined according to the national recommendations and context. “At-risk” will be defined according to national and sub-national context and M-RITE’s assessment.

Data source(s) Project records

Data disaggregation Country, district, SDP type, public/private


Annual




Indicator 1.1.6 Percent of health workers in M-RITE-supported facilities who can explain how to identify and reach zero-dose and under-immunized children.

Rationale

This indicator measures health worker capabilities related to the specific behaviors needed to identify and reach zero-dose children. In some countries we may tailor it to measure direct observation of the application of these capabilities (e.g. by observing their use of facility register data to identify which communities need outreach). These capabilities are a necessary but not sufficient condition to actually reaching zero-dose and underimmunized children, yet important and measurable steps in the results chain.

Precise definition Percent of health workers in M-RITE-supported facilities who can explain how to identify and reach zero-dose and under-immunized children.





Annual




Indicator 1.1.7 Percent of M-RITE-supported health facilities who improve their readiness to reach zero-dose and under-immunized children with high-quality immunization services. (T)

Rationale

Building from Gavi’s theory of change to reduce the number of zero-dose children, M-RITE recognizes the important step of reaching zero-dose children once they have been identified. Through M-RITE’s baseline assessments, we will support efforts to define the behaviors and actions of health care workers, as well as necessary processes, to reach zero-dose children and convert those into a readiness checklist. These may include constructs such as having an up-to-date microplan, implementing community engagement activities, and having sufficient operational budget for outreach activities.

Precise definition “Readiness” will be defined in country PMPs based on needs identified through M-RITE’s baseline assessment.




Annual





Indicator 1.3.1 Percent of M-RITE-supported areas with activities led by the EPI program designed and implemented to overcome specific vaccine demand- or utilization-related barriers to immunization in high-risk communities.*

Rationale

This aligns with the IA2030 indicator to measure actions to overcome demand- and utilization-related barriers. When providing in-country TA, M-RITE will assess the baseline status of activities to overcome demand- and utilization-related barriers and determine if current activities and plans are sufficient, or require updating.

Precise definition

"High risk" will be defined at the level of planning and implementation, ideally district or facility through M-RITE’s co-creation process.


Data disaggregation Country, district, type of SDP (as relevant)


Annual




Indicator 2.0.2 Percent of national and sub-national immunization stakeholders whose capacity* was strengthened** by M-RITE-supported capacity strengthening activities. (T)

Rationale

This indicator will be further updated and tailored to reflect specific capacity building needs in countries where M-RITE works. This indicator measures the initial outcomes of M-RITE support capacity strengthening activities, such as trainings, coaching, peer-learning, etc.

Precise definition

Numerator: National and sub-national immunization stakeholders whose capacity was strengthened by M-RITE support capacity strengthening activities, as measured by a post-test, direct observation, or other feasible and appropriate approaches


Denominator: National and sub-national immunization stakeholders who participated in M-RITE supported capacity strengthening activities Capacity building interventions are defined as any forms of interventions or activities that improve the ability of a person, group, or organization to meet objectives or to perform better. We will tailor the definition of “capacity,” but it may include skills, knowledge, behaviors, or actions taken.


Data disaggregation Country, stakeholder type, gender of stakeholder


Annual




Indicator 2.2.2 Number of global, national, and sub-national policies (including strategies, guidelines, and operational plans) developed with M-RITE support related to immunization services or MOMENTUM cross-cutting areas.

Rationale This indicator measures the initial outcomes of M-RITE's technical assistance toward improved policy for immunization.

Precise definition

Number of global, national, and sub-national policies (including strategies,

guidelines, and operational plans) developed with M-RITE support related to

immunization services or MOMENTUM cross-cutting areas.


Data disaggregation Country, type of policy, topic (e.g., COVID-19 vs. non-COVID-19)


Annual

PY1 target 1 (DRC)



Reporting by result leads and in-country M&E leads.

Indicator 2.3.1 Percent of M-RITE-assisted local organizations with improved performance

Rationale This indicator measures whether USG-funded capacity-development efforts have improved performance in supported organizations.

Precise definition

This is the USG indicator CBLD-9:

https://www.usaid.gov/sites/default/files/documents/cbld-

9_pirs_final_fy20_10.1.2020.pdf

Capacity is the ability of people, organizations, and society as a whole to

manage their affairs. Capacity development is the process of strengthening

and maintaining such capacity. Capacity is a form of potential; it is not visible

until it is used. Therefore, performance is the key consideration in

determining whether capacity has changed. Organizations with improved

performance will have undergone a deliberate process to improve execution

of organizational mandates to deliver results its stakeholders.

M-RITE will measure improvement organization using already existing tool

such as university accreditation process, ISO professional farmer

organization guidelines, OCA tool, etc.) and for monitoring and measuring

changes on a key performance metric (e.g., increased sales, reduced staff

turnover, increased efficiency of service delivery, etc.).


Data disaggregation Country, type of organization


Annual




Indicator 3.0.1 Number of evidence products, tools, or data analyses produced by M-RITE that include actionable recommendations and an equity and gender equity lens.

https://www.usaid.gov/sites/default/files/documents/cbld-9_pirs_final_fy20_10.1.2020.pdf

https://www.usaid.gov/sites/default/files/documents/cbld-9_pirs_final_fy20_10.1.2020.pdf


Rationale

This indicator measures M-RITE’s evidence outputs. This indicator is worded to ensure that M-RITE produces evidence outputs that are more likely to be used (e.g. by including actionable recommendations) and that will accelerate the achievement of equitable immunization (e.g. by including an equity and gender equity lens).

Precise definition

The term "actionable recommendations" is measured using a checklist from

FAO. We will assess equity and gender lens using our M-RITE process

indicators related to pro-equity/gender-sensitive activity design and

implementation.


Data disaggregation Topic (e.g., COVID-19 or non-COVID-19), country


Semi-annual

PY1 target 9


Reporting by each result lead and in-country M&E leads.

Indicator 3.2.2 Number of individuals reached through M-RITE co-creation, dissemination, knowledge exchange, and knowledge translation activities and events.

Rationale This indicator measures reach of M-RITE knowledge products and knowledge translation/dissemination activities.

Precise definition

Individuals are counted in "reach" if they were the active participant in an M-RITE co-creation or knowledge translation/dissemination-related event or activity. Active participation could be defined as those people who actively have to choose to engage in content - by attending meetings, opening e-mails, logging into webinars, etc.


Data disaggregation Organization, gender (where possible), country


Semi-annual


PY1 target 500



Indicator 3.3.1 Number of innovations supported by M-RITE assistance.

Rationale

Particularly during COVID-19, innovations in products, processes, tools, approaches, and service delivery models are needed to maintain immunization services. M-RITE will contribute to the identification and implementation of these innovations and track this output.

Precise definition

‘Innovations’ are products, processes, tools, approaches, service delivery

models, and/or other interventions (broadly defined) that have the potential

to achieve significant (not incremental) improvements in development

outcomes versus existing alternatives and are intended to improve the lives

of ultimate beneficiaries. Here, innovations are defined not by their novelty,

but by their potential to achieve significant improvements in development

outcomes versus existing alternatives. (From STIR10 PIRS.) "Supported"

refers to technical, in-kind, or financial support from M-RITE related to the

introduction, testing, or adoption of innovations.


Data disaggregation Country, topic, new or ongoing support


Annual

PY1 target 2



Indicator 3.4.2 Number of global and regional documents (guidelines, strategies, tools, etc.) that M-RITE contributes to.

Rationale This indicator measures M-RITE's active participation and thought leadership in global and regional technical conversations.


Precise definition Number of global and regional documents (guidelines, strategies, tools, etc.)

that M-RITE contributes to.


Data disaggregation Topic (e.g., COVID-19 or non-COVID-19), new or ongoing


Semi-annual

PY1 target 4



Indicator 4.0.2 Number of multi-sectoral collaborations formed or strengthened at global, regional, national, and sub-national levels as a result of M-RITE support to address immunization coverage and equity issues.

Rationale This indicator measures M-RITE's contributions to forming or strengthening multi-sectoral collaborations, a necessary step in the causal chain toward improved immunization coverage and equity.

Precise definition

We define "collaboration" as a relationship between two organizations

working together with the goal of jointly producing an output or

contributing to a common goal. In this case, a multi-sectoral collaboration

must occur between two organizations from different sectors (e.g.,

education and health, finance and health). We will define "formation" as a

new relationship, and "strengthening" as a new goal on an existing

collaboration relationship.


Data disaggregation Collaboration type, topic, country, new (formed) or existing (strengthened)


Annual

PY1 target 2




Indicator 4.0.3 Number of partnerships formed or strengthened at global, regional, national, and sub-national levels to address immunization coverage and equity issues.

Rationale This indicator measures M-RITE's contributions to forming or strengthening multi-sectoral partnerships, a necessary step in the causal chain toward improved immunization coverage and equity.

Precise definition

We will define "partnership formation" during the partnership landscaping activity and based on existing concepts that partnerships can be organic or mandated, but typically share common goals. We will measure partnership strength at baseline and end line of related activities using a validated partnership survey tool that measures perceived effectiveness, efficiency, and legitimacy of the partnership, as well as its composition.


Data disaggregation Partnership type, topic, country, new (formed) or existing (strengthened)


Annual

PY1 target 1




APPENDIX C. ILLUSTRATIVE LEARNING AGENDA THEMES AND

QUESTIONS

Theme Proposed learning questions to be answered over M-RITE life-of-project (PY1 focus questions indicated).

1. Equity (including

zero-dose, under-

immunized,

gender)

1.1 What are the information-related obstacles and opportunities to identifying, targeting, and reaching zero-dose and under-immunized children? What are promising information systems, tools, or practices to help with identifying, targeting, and reaching zero-dose and under-immunized children? (Similar to Gavi learning question 1.6: What are the evidence gaps at national/sub-national level to monitor and measure for zero-dose and have Gavi data investments contributed to identifying and quantifying zero-dose children and missed communities?) [PY1]

1.2 What strategies can overcome barriers to adequate financing that most significantly affect zero-dose children? [PY1]

1.3 How can improved governance increase vaccination among zero-dose and under-immunized children? [PY1]

1.4 How can supply chain interventions improve the equity of immunization coverage, specifically by overcoming obstacles to reaching zero-dose children (e.g., geographic access barriers among rural/remote communities)? [PY1]

1.5 Where and who are zero-dose children, and missed communities? Why are they being missed? (From Gavi learning agenda question 1.1)

1.6 What are the most effective and cost-effective strategies to overcoming entrenched obstacles to reaching zero-dose children? Under-immunized children?

1.7 What strategies did M-RITE use to improve immunization equity? How did approaches mitigate inequities in demand and access? (Aligned with XM learning question)

1.8 What are some of the most relevant innovations for improving identification of, and targeting, and reaching zero-dose and under-vaccinated children? How did M-RITE leverage these innovations? (Aligned with Gavi learning question 5.1: How do Gavi’s investments in innovation contribute to improved immunization outcomes? Incl. innovation in digitization of data and monitoring of zero-dose children and missed communities? And enable identification and scaling of these across products, services, and practices?)

1.9 What is the cost-effectiveness and effectiveness of gender-sensitive, -responsive, or -transformative strategies? How can this evidence be used to inform policy change?

1.10 Whether, how, and why do various approaches to micro-planning improve the identification of and targeting, and reaching zero-dose and under-immunized children?

1.11 How did M-RITE achieve its objectives related to improving immunization coverage and equity? (Aligns with XM learning question)


2. COVID-19 2.1 What are the most effective strategies to mitigate the effects of COVID-19 on

the organization, access, and use of RI services in different settings and

contexts? What are the equity implications of these strategies for vulnerable

communities (e.g., rural remote, urban poor, children in conflict settings)?

Which of these strategies or adaptations should be recommended for

ongoing implementation? [PY1] (Aligned with Gavi learning question 11.8:

What approaches were selected and were they effective at re-establishing RI,

and identifying and reaching zero-dose children and missed communities?

Including the role of innovations. Aligned with MKA learning question: How

did COVID-19 affect health systems, how did the response affect

MNCHN/FP/RH services delivery and access, and what was done by

programs/countries to mitigate impact?)

2.2 What can we learn from the introduction of other life-course vaccinations

(e.g., MCV2, RTSS, Td extended schedule, HPV [youth], meningitis A) and

platforms to reach non-standard populations for COVID-19 vaccine

introduction? [PY1]

2.3 How did the COVID-19 pandemic influence short- and longer-term changes to

the financing, planning, organization, delivery, and monitoring of

immunization services? How did it influence client trust, demand, and

satisfaction with immunization services and the health system more broadly?

2.4 How did the contextual shock of the COVID-19 pandemic influence the

implementation of M-RITE and the results it achieved? How did M-RITE adapt

its overall project approach to the COVID-19 context?

2.5 [New in version 2] What are the bottlenecks to effective and equitable

vaccination against COVID-19 at the sub-national level, and what approaches

were effective in overcoming these bottlenecks.

3. Health systems,

resilience, and

life-course/

integration

3.1 What are valid, feasible indicators to measure integration of immunization

and other primary health care (PHC) services?

3.2 How has integration with other PHC services been used to reach zero-dose

children and missed communities? What has worked well, or not, and why?

(Aligns with Gavi learning question 10.4)

3.3 How did M-RITE contribute to strengthening health systems resilience? What

capacity strengthening efforts (whose and what dimensions of capacity) were

successful to strengthen resilience of health systems? (Aligns with XM

learning question)


4. Demand, trust,

community

4.1 What are effective or high-potential partnership models for underserved

communities? How have models shifted during COVID-19? [PY1] (Aligns with

Gavi learning question 1.3: What are effective ways to engage with other

partners to reach the marginalized, missed communities and zero-dose

children (including integration with PHC?)

4.2 What are valid and feasible indicators and data collection approaches to

measure respectful care during an immunization encounter? [PY1]

4.3 What community engagement strategies are most effective at reducing the

number of zero-dose children?

5. Innovation/ HCD/

co-creation/

learning

5.1 What is the added value of a “co-creation” approach to designing and

implementing M-RITE country projects? What does the ideal co-creation

approach entail, and how can it be adapted to different contexts and needs?

[PY1]

5.2 How was active use of data and adaptive learning improved for

immunization? (Includes capacity-building at community, facility,

subnational, and national levels) (Aligned with XM learning question)

5.3 What evidence is there of institutionalization or sustainable change in use of

CLA, adaptive learning, and HCD approaches in MOMENTUM countries?

What contributed to these successes? (Aligned with XM learning question)

5.4 What is the added value of an HCD approach to identifying and overcoming

entrenched obstacles, compared to more traditional approaches? How did

M-RITE’s HCD approach contribute to its achievement of results?

5.5 What are the most effective ways to scale innovations that reduce the

number of zero-dose children? How do we ensure innovation does not

increase inequity?

6. Capacity

strengthening

6.1 What capacity-building strategies/interventions (or combination of

strategies) were effective in increasing capacity at the individual,

organization, community, and system levels? (Aligned with XM learning

question)

6.2 Were any associations seen between strengthened capacity and

improvements in equitable immunization coverage? (Aligned with XM

learning question)

7. Partnerships 7.1 How have partnerships contributed to strengthening immunization

programs, and what is the untapped potential of strategic partnerships for

improving equitable immunization coverage? [PY1]

7.2 How can strategic partnerships contribute to planning, introduction, and

acceptance of COVID-19 vaccine? [PY1]

7.3 What strategic partnerships (and under what conditions) were successful

(feasible, acceptable)? (Aligned with XM learning question)


7.4 Which program strategies were most effective in building or strengthening

cross-sectoral collaboration and partnerships (e.g., health and non-health,

health and education, PPPs, private sector)?

7.5 What is M-RITE’s approach to measuring the contribution of partnerships to

improving immunization equity?


APPENDIX D. SUPPLEMENTARY INDICATORS

Indicator Code

Indicator Name

Indicator Type

Indicator Definition Numerator Denominator Data source (e.g. HMIS)

Indicator source/alignment (e.g. Gavi)

Disaggregation

S1.0.1 Number of programs where M-RITE-supported the development or improvement of evidence-based and funded plans to address vaccine coverage of high-risk communities.

Initial outcome

Number of M-RITE-supported EPI national or subnational programs with evidence-based and funded plan to address vaccine coverage of high risk communities. "High risk" should be defined on a country-by-country basis. When providing in-country TA, M-RITE will assess the current status of equity plans and guidelines, and will assess whether the current equity plan is sufficient, or requires updating. For decentralized countries, the assessment and updating may occur at a lower level (e.g. province, county).

Number of M-RITE-supported national or subnational EPI programs that have an evidence-based and funded plan to address vaccine coverage of high-risk communities.

n/a Project records

IA2030 Country, province or county where applicable

S1.0.2 Percent of M-RITE-supported districts (or facilities) reporting full stock availability at service delivery level during re-supply cycle or reporting period.

Initial outcome

Percent of M-RITE-supported districts (or facilities) reporting full stock availability at service delivery level during re-supply cycle. Full stock availability is defined as the absence of any stock-out for vaccines and supplies.

Number of M-RITE-supported districts (or facilities) reporting full stock availability at service delivery level during re-supply cycle.

Total number of M-RITE-supported districts (or facilities).

LMIS, HMIS, or other stock databases.

IA2030 Country, district, SDP, urban/rural


S1.0.3 Number of programs where M-RITE-supported the development or improvement of policies or standard operating procedures in place to strengthen delivery of immunization services integrated with primary health care, across the life-course.

Initial outcome

Number of M-RITE-supported national or subnational EPI programs with policies or standard operating procedures in place to strengthen delivery of immunization services integrated with primary health care, across the life course. When providing in-country TA, M-RITE will assess the whether sufficient and updated integration policies/SoPs exist.

Number of M-RITE-supported programs with policies or standard operating procedures in place to strengthen delivery of immunization services integrated with primary health care, across the life-course.

n/a Project records

IA2030 district, country

S1.0.4 Percent of health facilities in M-RITE-supported areas that received a high-quality supportive supervision visit for immunization in the last quarter.

Initial outcome

Percent of health facilities in M-RITE-supported areas that received a high-quality supportive supervision visit for immunization in the last quarter. At the outset of a country buy-in M-RITE will use existing checklists to assess the quality of existing supportive supervision visits and through workplan activities will strengthen quality and frequency of supervision, noting where contextual bottlenecks such as inadequate resources to hire supervisors limit progress on this indicator. The timeframe (e.g. "last quarter") may be adjusted according to national standards and norms.

Number of health facilities in M-RITE-supported areas that received a high-quality supportive supervision visit for immunization in the last quarter.

Total number of health facilities in M-RITE-supported areas

Supportive supervision report, rapid telephone facility survey, or health worker interviews.


Country, district, SDP type, urban/rural

S1.0.5 Percent of health workers in M-RITE-supported areas who can explain how to use immunization

Initial outcome

Percent of health facilities in M-RITE-supported areas with up-to-date immunization monitoring charts or other data use tool.

Number of health facilities in M-RITE-supported areas with up-to-date immunization monitoring charts or other data use tool.

Total number of health facilities in M-RITE.

Supportive supervision report, rapid telephone facility survey, or health


Country, district, SDP type, urban/rural


monitoring charts or other data use tools and the meaning of the available information.

worker interviews.

S1.0.6 Percent of clients who report being treated with respect during their last immunization visit in M-RITE-supported facilities.

Initial outcome

Percent of clients who report being treated with respect during their last immunization visit in M-RITE-supported facilities. The definition for "respect" will build from WHO's definition of respectful maternity care, but will be adapted for immunization and validated with clients in DRC.

Number of clients who report being treated with respect during their last immunization visit in M-RITE-supported facilities.

Total number of clients served.

Caregiver exit surveys and rapid phone surveys.

ACCELERATE behavioral indicators

Country, district, SDP, provider type, sex of caregiver

S1.0.7 Percent or number of potential vaccination opportunities that were missed (MOV).

Initial outcome

Percent or number of an individual (child or person of any age) making a contact with health services who is eligible for vaccination (e.g. unvaccinated or partially vaccinated and free of contraindications to vaccination), which does not result in the person receiving one or more of the vaccine doses for which he or she is eligible (WHO).

Missed = Number of individuals (child or person of any age) making a contact with health services who is eligible for vaccination (e.g. unvaccinated or partially vaccinated and free of contraindications to vaccination), which does not result in the person receiving one or more of the vaccine doses for which he or she is eligible.

All opportunities = Number of individuals (child or person of any age) making a contact with health services who is eligible for vaccination (e.g. unvaccinated or partially vaccinated and free of contraindications to vaccination)

MOV surveys (M-RITE will explore more routine data collection approaches)

WHO MOV


S1.0.8 Number of M-RITE-supported districts/provinces that are implementing community accountability mechanisms (e.g. score cards, community consultations, community conversations, community charters, community health committees, civil society hearings etc.) to support women’s, children’s and adolescents’ health, including immunization

Initial outcome

Community accountability mechanisms include score cards, community consultations, community conversations, community charters, community health committees, civil society hearings etc.) to support women’s, children’s and adolescents’ health, including immunization.

Number of M-RITE-supported districts/provinces that are implementing community accountability mechanisms (e.g. score cards, community consultations, community conversations, community charters, community health committees, civil society hearings etc.) to support women’s, children’s and adolescents’ health, including immunization.

n/a. Project records

MCGL District, province, country, type of mechanism

S1.0.9 Percent of health facilities that meet with community members and discuss performance of immunization activities (either by themselves or through participation in broader RI

Initial outcome

Percent of health facilities that meet with community members and discuss performance of immunization activities (either by themselves or through participation in broader RI meetings) in M-RITE-supported areas.

Number of HFs that meet with community members and discuss performance of immunization activities (either by themselves or through participation in broader RI meetings).

Total number of health facilities in M-RITE-supported areas.

Supportive supervision report

MCSP process indicator

Country, district, type of SDP


meetings) in M-RITE-supported areas.

S1.10 Number of M-RITE-supported programs with activities implemented to engage fathers in immunization services.

Initial outcome

Number of M-RITE-supported programs with activities implemented to engage fathers in immunization services.

Number of M-RITE-supported programs with activities implemented to engage fathers in immunization services.

N/A Project Records

Related to proposed ERG gender indicator

country, district, type of activity

S1.11 Proportion of health facilities that received supportive supervision (last quarter) in M-RITE-supported areas.

Initial outcome

Proportion of health facilities that received supportive supervision (last quarter) in M-RITE-supported areas.

Number of health facilities that received supportive supervision (last quarter) in M-RITE.

Total number of health facilities that received supportive supervision (last quarter) in M-RITE-supported areas.

Supportive supervision records


S1.12 Proportion of health facilities with an updated immunization monitoring chart (last month).

Initial outcome

Proportion of health facilities with an updated immunization monitoring chart (last month).

Number of health facilities with an updated immunization monitoring chart (last month).

Total number of health facilities with an updated immunization monitoring chart (last month).

Facility charts


Country, district


S1.13 Proportion of health facilities that met with community members to discuss immunization (last quarter).

Initial outcome

Proportion of health facilities that met with community members to discuss immunization (last quarter).

Number of health facilities that met with community members to discuss immunization (last quarter).

Total number of health facilities that met with community members to discuss immunization (last quarter).

Facility charts


Country, district

S1.14 Proportion of health facilities with at least one qualified and trained vaccine provider (last quarter).

Initial outcome

Proportion of health facilities with at least one qualified and trained vaccine provider (last quarter).

Number of health facilities with at least one qualified and trained vaccine provider (last quarter).

Number of health facilities with at least one qualified and trained vaccine provider (last quarter).

Facility charts


Country, district

S 2.0.1 Number of M-RITE-supported countries that use a reliable method for tracking/monitoring total government routine immunization expenditures at either national or subnational levels.

Initial outcome

Number of M-RITE-supported countries that use a reliable method for tracking/monitoring total routine immunization expenditures at either national or subnational levels. "Reliable method" will be defined during the PY1 activity on this topic, and will include the method that M-RITE produces. Total government routine immunization expenditures comprise the total value of funds from government sources spent on routine vaccines (including any co-financing payments) and immunization-specific system-level costs (e.g., injection safety, immunization supply chain, costs of RI outreach, transport, wages of fully-dedicated immunization staff, etc.) across all administrative levels.

Number of M-RITE-supported countries that use a reliable method for tracking/monitoring total government routine immunization expenditures at either national or subnational levels; depending on direction of work, could be disaggregated by evidence of this process at national or subnational levels (though would need to define if those categories would be mutual exclusive).

n/a Project records

Country, district


S 2.0.2 Number of M-RITE-supported countries with at least one implemented recommendation from a NITAG or other relevant independent technical advisory group.

Outcome Number of M-RITE-supported countries with at least one implemented recommendation from a NITAG or other relevant independent technical advisory group.

Number of M-RITE-supported countries with at least one implemented recommendation from a NITAG or other relevant independent technical advisory group.

n/a Project records

IA2030 Country, district

S 2.0.3 Evidence of increased capacity among immunization program managers in M-RITE-supported programs to plan and implement activities to reach zero-dose and under-immunized children.

Initial outcome

Evidence of increased capacity among immunization program managers at national and sub-national levels in M-RITE-supported programs to plan and implement activities to reach zero-dose and under-immunized children. We will measure management capacity using established tools, adapted to ensure that capacities/competencies related to 'managing for equity' are included.

Number of M-RITE-supported countries and districts with increased capacity to identify and reach zero-dose and under-immunized children.

n/a Project records

n/a Country, district

S 2.0.4 Number of M-RITE-supported countries with legislation in place that is supportive of immunization and commits the government to finance all immunization

Initial outcome

Number of M-RITE-supported countries with legislation in place that is supportive of immunization and commits the government to finance all immunization program functions at all levels.

Number of M-RITE-supported countries with legislation in place that is supportive of immunization and commits the government to finance all immunization programmed

n/a Project records

n/a Country


program functions at all levels.

functions at all levels.

S 2.0.5 Number of improved global, national, and sub-national policies (including strategies, guidelines, and operational plans) developed with M-RITE support that are gender sensitive or gender transformative.

Initial outcome

Number of improved global, national, and sub-national policies (including strategies, guidelines, and operational plans) developed with M-RITE support that are gender sensitive or gender transformative.

Number of improved global, national, and sub-national policies (including strategies, guidelines, and operational plans) developed with M-RITE support that are gender sensitive or gender transformative.

n/a Project records

MCGL Country, district, type of policy

S 2.0.6 Percent of MoH / UNEPI Technical Working Group coordination meetings held where routine immunization was discussed in M-RITE-supported districts.

MoH/UNEPI Technical Working Group coordination meetings are organized by UNEPI and EPI partners to share experiences and challenges in relation to routine immunization to develop strategies for improvement as a country. Routine immunization refers to free immunization services readily available by public and private health facilities as per the immunization schedule. These are different from one-off campaigns aimed at reaching bigger populations and age groups other than the target age of less than one year.

Number of MoH / UNEPI Technical Working Group coordination meetings held with EPI technical partners where routine immunization was discussed.

The number of planned MoH / UNEPI Technical Working Group coordination meetings (the target for UNEPI is to conduct 1

Meeting minutes

MSCP


meeting per month).

S 2.0.7 Number of national level guidelines, manuals, and tools in which concepts are incorporated (IP custom).

This refers to national level / MoH-EPI documents/manuals e.g. Immunization in Practice and operational level training materials in which REC-QI concepts are incorporated.

Number of national level guidelines, manuals, and tools in which concepts are incorporated.

Copies of UNEPI national level documents

MSCP Type of document

S 2.0.8 Percent of MoH / EPI Technical Working Group coordination meetings held where routine immunization was discussed.

Number of MoH / EPI Technical Working Group coordination meetings held with EPI technical partners where routine immunization was discussed.

The number of planned MoH / EPI Technical Working Group coordination meetings (The target for UNEPI is to conduct 1 meeting per month).

Meeting minutes

MSCP N/A


S.3.0.1 Number of M-RITE-supported activities that use an adaptive learning approach to design and implementation.

Output We will assess whether the activities used "adaptive learning" based on the MKA Adaptive Learning toolkit.

Number of M-RITE-supported activities that use an adaptive learning approach to design and implementation.

n/a Project records

MCGL Country, topic

S.3.0.2 Evidence of M-RITE evidence products or data analyses contributing to decisions affecting host country priorities, strategies, programs, policies, or shifts in resource allocations made as a result of MKA capacity building or technical assistance activities.

Outcome We will assess the extent to which M-RITE evidence outputs contribute to decisions using case studies, and applying a validated scale for evidence uptake (Landry).

n/a n/a Case studies MKA Topic (e.g. COVID or non-COVID), country

S.3.0.3 Number of M-RITE-supported review and reflection sessions (e.g. data review meetings, root cause analysis workshops)

Output Number of M-RITE-supported review and reflection sessions (e.g. data review meetings, root cause analysis workshops) where data and evidence are discussed critically.

Number of M-RITE-supported review and reflection sessions (e.g. data review meetings, root cause analysis workshops) where data and evidence

n/a Project records

Similar to MCGL

Country, SDP type


where data and evidence are discussed critically.

are discussed critically.

S.3.0.4 Number of national and sub-national EPI programs that developed or implemented a data improvement plan with M-RITE support.

Output A "data improvement plan" should reflect the most recent WHO, CDC, or Gavi guidance.

Number of national and sub-national EPI programs that developed or implemented a data improvement plan with M-RITE support.

n/a Project records

IA2030 Country

S.3.0.5 Number of national and sub-national EPI programs that strengthened health information systems with M-RITE support to improve the identification of zero-dose and/or under-immunized children.

Output We will define the requirements of systems to identify and reach zero-dose children in the information ecosystem landscaping activity.

Number of national and sub-national EPI programs that strengthened health information systems with M-RITE support to improve the identification of zero-dose and/or under-immunized children.

n/a Project records

n/a Country


S.3.0.6 Evidence of strengthened skills and capacity of immunization stakeholders at national and sub-national levels to take action on equity-related data.

Initial outcome

We will measure skills to take action on equity-related data using an M-RITE checklist adapted from MEASURE tools, but adapted for equity-related data use actions.

n/a n/a Case studies (interviews or surveys with health workers and managers to measure skills and capacity, direct observation of data review and planning meetings to observe skills and capacity)

n/a n/a

S.3.0.7 Evidence of strengthened systems and processes at global, regional, national, and sub-national levels to produce, make available, and take action on equity-related data.

Initial outcome

Systems and processes include technological and management systems such as data review meetings, the use of data entry clerks, the use of dashboards or other decision-support tools, incentives to use data, etc.

n/a n/a Project records

n/a n/a

S.3.0.8 Number of unique visitors to M-RITE web page on MOMENTUM website.

Initial outcome

Number of unique visitors to M-RITE web page on MOMENTUM website.

Number of unique visitors to M-RITE web page on MOMENTUM website.

n/a Project records

Similar to MKA

Country


S.3.0.9 Number of global and regional technical working groups that M-RITE contributes to.

Output Number of global and regional technical working groups that M-RITE contributes to.

Number of global and regional technical working groups that M-RITE contributes to.

n/a Project records

n/a Topic (e.g. COVID or non-COVID)

S.3.10 Number of individuals from M-RITE-supported country programs who contribute to global and regional working groups with M-RITE support.

Output Contribution could be in various ways that include document review, participation in working group meetings, and other events.

Number of individuals from M-RITE-supported programs who contribute to global and regional working groups with M-RITE support.

n/a Project records

n/a Topic (e.g. COVID or non-COVID)

S.4.0.1 Number of new, high-potential, non-immunization organizations engaged to address immunization coverage and equity.

Initial outcome

"New partners" will be defined during the stakeholder landscaping/framework activity. The purpose of this indicator is to measure M-RITE's steps to facilitate the entrance of new, high-potential partners to solve entrenched obstacles related to immunization coverage and equity.

Number of new, high-potential, non-immunization organizations engaged to address immunization coverage and equity.

n/a Project records

n/a Stakeholder type

S.4.0.2 Number of core immunization partners engaged in the planning, design, and implementation of M-RITE

Initial outcome

"Core partners" will be defined during the stakeholder landscaping/framework activity, but are likely to include WHO, UNICEF, Gavi, and CDC. "Co-creation" is defined as a participatory, engaging process that leverages the unique voices of all stakeholders. The purpose of this indicator is to measure M-RITE engagement of these core partners in our activities, which will

Number of core immunization partners engaged in the planning, design, and implementation of M-RITE activities through a co-creation approach.

n/a Project records

n/a Stakeholder type


activities through a co-creation approach.

increase the relevance and usefulness of our outputs.


REFERENCES

1 USAID. Series 200 References. Last updated October 28, 2020. https://www.usaid.gov/who-we-are/agency-policy/series-200/references-chapter

2 USAID. Evaluation Policy. Last updated May 09, 2020. https://www.usaid.gov/evaluation/policy 3 USAID Learning Lab. Collaborating, Learning, and Adapting Toolkit. https://usaidlearninglab.org/cla-toolkit 4 Britt, Heather, Richard Hummelbrunner, and Jacqueline Greene. 2017. Causal Link Monitoring.

http://www.betterevaluation.org/resources/overview/Causal_Link_Monitoring 5 USAID. Maternal and Child Survival Program Mozambique Program Brief Addressing the Denominator

Conundrum for Maternal and Child Health Programs: A New Methodology. https://www.mcsprogram.org/resource-search-results/?_sfm_resource_country=mozambique&_sfm_resource_language_2=english

6 Arsenault, Catherine, et al. 2016. “An Equity Dashboard to Monitor Vaccination Coverage.” Bulletin of the World Health Organization vol. 95,2:128-134. doi:10.2471/BLT.16.178079

7 Restrepo-Mendez, Maria Clara, Godwin Mindra, Alyssa Sharkey, and Richard Duncan. Are the Current Equity Metrics Used by Immunization Programmes Adequate? Equity Reference Group for Immunization. May 2018.

8 Victoria, Cesar and Tove Ryman. March 2018. Potential Approaches to Better Measure and Track Equity in Immunization Using Survey and Administrative Data, and Data Triangulation. Equity Reference Group for Immunization

9 Dolan, Samantha, et. al. March 2019. “Redefining Vaccination Coverage and Timeliness Measures Using Electronic Immunization Registry Data in Low- and Middle-Income Countries.” Vaccine vol. 37, issue 19: 1859-1867. https://doi.org/10.1016/j.vaccine.2019.02.017

https://www.usaid.gov/who-we-are/agency-policy/series-200/references-chapter

https://www.usaid.gov/who-we-are/agency-policy/series-200/references-chapter

https://www.usaid.gov/evaluation/policy

https://usaidlearninglab.org/cla-toolkit

http://www.betterevaluation.org/resources/overview/Causal_Link_Monitoring

https://www.mcsprogram.org/resource-search-results/?_sfm_resource_country=mozambique&_sfm_resource_language_2=english

https://www.mcsprogram.org/resource-search-results/?_sfm_resource_country=mozambique&_sfm_resource_language_2=english

https://doi.org/10.1016/j.vaccine.2019.02.017

activity monitoring, evaluation, and learning plan

Documents