evaluation of unicef zambia’s capacity development ... · final report for the evaluation of...

Evaluation of UNICEF Zambia’s

Capacity Development

Interventions 2016–2018 Final Report

OCTOBER 2019

Contents Page

Acronyms ......................................................................................................................................... i

Executive Summary .........................................................................................................................1

1.0 Background ................................................................................................................................4

1.1 Growing Emphasis on Capacity Development ......................................................................4

1.2 Capacity Development in the Zambian Context ....................................................................4

1.3 Objectives and Scope of the Evaluation ................................................................................5

2.0 Selection of CD Interventions ....................................................................................................7

3.0 Evaluation Design ......................................................................................................................8

4.0 Conceptual Framework ..............................................................................................................9

4.1 Evaluation Methodology ......................................................................................................11

4.2 Qualitative Analysis .............................................................................................................12

4.3 Sampling Approach .............................................................................................................13

4.4 Methodological Limitations .................................................................................................15

4.5 Ethical Considerations .........................................................................................................16

5.0 Findings....................................................................................................................................17

5.1 Relevance .............................................................................................................................17

5.2 Effectiveness ........................................................................................................................20

5.3 Efficiency .............................................................................................................................28

5.4 Perceived Impacts ................................................................................................................32

5.5 Equity ...................................................................................................................................36

5.6 Sustainability .......................................................................................................................38

6.0 Lessons Learned.......................................................................................................................42

6.1 Challenges ............................................................................................................................42

6.2 Opportunities .......................................................................................................................43

7.0 Conclusions and Recommendations ........................................................................................43

7.1 Relevance .............................................................................................................................44

7.2 Effectiveness and Perceived Impacts ...................................................................................44

7.3 Efficiency .............................................................................................................................45

7.4 Equity ...................................................................................................................................45

7.5 Sustainability .......................................................................................................................46

References ......................................................................................................................................50

Annex A. TOR .................................................................................................................................1

Annex B. Protocols ..........................................................................................................................1

Annex C. Evaluation Matrix ............................................................................................................1

Annex D. Program Summaries ........................................................................................................1

Figures Page

Figure 1. Overview of CD Interventions Included in the Study ......................................................6

Figure 2. Research Questions ..........................................................................................................8

Figure 3. Conceptual Framework ..................................................................................................10

Figure 4. Evaluation and Research Design Overview ...................................................................12

Tables Page

Table 1. CD Interventions and Data Collection Levels ...................................................................7

Table 2. Sample for Each CD Intervention ....................................................................................14

Table 3. Description of Needs Assessments by Intervention ........................................................18

Table 4 Summary of Relevance Findings ......................................................................................19

Table 5. Summary of Effectiveness Findings ................................................................................26

Table 6. Summary of Efficiency Findings .....................................................................................31

Table 7. Summary of Perceived Impacts Findings ........................................................................35

Table 8. Summary of Equity Findings ...........................................................................................37

Table 9. Summary of Sustainability Findings................................................................................40

Table 10. Recommendation Priority, Timeframe, and Accountability ..........................................46

Table D1. Summary of HMISS CD Intervention and Outcomes ............................................... D-2

Table D2. Summary of SCT CD Intervention and Outcomes .................................................... D-5

Table D3. Summary of WASH CD Intervention and Outcomes ................................................ D-7

Table D4. Summary of Catch Up CD Intervention and Outcomes ............................................ D-8

Table D5. Summary of Strengthening of the Alternative Care System CD Intervention and Outcomes.............................................................................................................................. D-9

Final Report for the Evaluation of UNICEF Zambia’s Capacity Development Interventions

i

Acronyms

AIR American Institutes for Research

ACC Area Coordinating Committee

CD Capacity Development

CLTS Community-Led Total Sanitation

CWAC Community Welfare Assistance Committee

CHW Community Health Workers

DEBS District Education Board Secretary

DSESO District Senior Education Standards Officer

DHIS2 District Health Information System 2

DWAC District Welfare Assistance Committee

EHT Environmental Health Technicians

FGD Focus Group Discussion

GRZ Government of the Republic of Zambia

HMIS Health Management Information System

HMISS Health Management Information System Strengthening

ICT Information and Communications Technology

IRB Institutional Review Board

KII Key Informant Interview

M&E Monitoring and Evaluation

M2W Mobile to Web

MCDSS Ministry of Community Development and Social Services

MIS Management Information System

MoCTA Ministry of Chiefs and Traditional Affairs

MoGE Ministry of General Education

MoH Ministry of Health

MoLG Ministry of Local Government

MWDSEP Ministry of Water Development, Sanitation and Environmental Protection

ODF Open Defecation Free

RTM Real Time Monitoring

SCT Social Cash Transfer

TaRL Teaching at the Right Level

ToT Training of Trainers

UNICEF United Nations Children’s Fund

WASH Water, Sanitation and Hygiene

ZSHP Zambia Sanitation and Hygiene Programme


1

Executive Summary

Introduction

Over the last several years, UNICEF Zambia has supported numerous capacity development (CD)

interventions in various sectors, including trainings, mentorship and professional development,

and community outreach activities. Despite positive overall feedback from stakeholders and

participants, there is little empirical evidence on what has worked well, where improvements could

be made, and whether these CD interventions have led to the desired improvements in the skills

and knowledge of local government officials, professionals, and communities (UNICEF, 2018a).

Moreover, there is little evidence concerning the long-term sustainability of activities.

To generate empirical evidence on what CD activities have worked well and where adjustments

could potentially be made to improve effectiveness, UNICEF Zambia contracted the American

Institutes for Research (AIR) to evaluate a small portfolio of UNICEF Zambia’s recent CD

interventions implemented between 2016 and 2018. The purpose of the formative evaluation was

to examine the relevance, effectiveness, efficiency, perceived impacts, equity, and sustainability

of CD interventions within five programmes in five different sectors: (1) health—the Health

Management Information System Strengthening (HMISS) intervention; (2) social protection—

scale-up of the Social Cash Transfer (SCT) programme; (3) WASH—scale-up of the Mobile to

Web (M2W) tool for real-time monitoring, and scale-up of the Community-Led Total Sanitation

(CLTS) intervention; (4) education—scale-up of the Catch Up/Teaching at the Right Level

(TaRL) intervention; and (5) child protection—strengthening the alternative care system through

improved case management of vulnerable children and adolescents.

UNICEF Zambia commissioned this formative evaluation to learn what is working well and what

could potentially be improved in their capacity development programming. While UNICEF

Zambia is the primary intended user of this evaluation, other key stakeholders include

collaborating ministries within the Government of the Republic of Zambia (GRZ). One of the main

goals of the GRZ’s 7th National Development Plan is to implement a decentralised approach to

organisational management, technical operations and data review processes in order to reduce the

gaps in service delivery across sectors within the GRZ (UNICEF Zambia, 2017b). UNICEF

Zambia implemented several capacity development interventions through partnerships with

appropriate ministries to strengthen systems-building and to improve quality of services and care

through decentralized approaches and enhanced sustainability (UNICEF Zambia, 2017b; UNICEF

Zambia, 2018b).

Key Findings

The CD interventions under the five programmes included in our evaluation were largely perceived

to achieve their intended objectives. Specifically, we found that the CD activities contributed to

improved access to accurate data, improved data management, and increased motivation among

community volunteers. We identified government and community buy-in as key facilitating factors

in the implementation of CD activities and insufficient funding for information and

communications technology (ICT) and transportation as the primary constraining factors. There

appears to be room for improvement in effectiveness and relevance of CD interventions for

marginalized groups, especially women and people with disabilities. We saw some examples of


2

CD activities that were male-dominated (such as CLTS trainings led by community champions)

and some in which it was difficult for elderly or disabled people to apply what they learned during

trainings.

Close collaboration between UNICEF Zambia and the GRZ and needs assessments conducted

prior to programme rollout ensured that the content of CD interventions was relevant and well

aligned with national priorities. Respondents indicated that CD interventions were co-created with

relevant ministries and, in some cases, multiple models of the intervention were piloted prior to

scale-up to determine the most appropriate implementation model. However, greater attention

could be paid to district- and community-level circumstances during the needs assessment phase,

which appears to have focused heavily at the national level.

UNICEF Zambia, GRZ, and partners implemented CD activities in a timely manner apart from a

few reported delays receiving necessary inputs for training at the district and community levels.

We found several examples of CD activities using resources in creative and economical ways,

such as relying on locally available materials (TaRL), creating WhatsApp groups to troubleshoot

technical challenges virtually (HMISS), and convening community champions in zones for M2W

trainings to eliminate participant housing costs. In addition, UNICEF worked with MCDSS to

develop standardised training materials and modules on case management for children rather than

convening expensive workshops to train staff on the specifics of case management for children. It

is evident from the data that UNICEF and partners make considerable efforts to maximise available

resources and avoid unnecessary costs, and it seems unlikely that comparable CD support could

be delivered at significantly lower cost.

Respondents across the five programmes noted few barriers to participating in CD activities apart

from the inevitable resource constraints related to transportation, mobile network access, and

limited number of days available for training. Across the board, programmes made considerable

efforts to include comparable numbers of men and women in CD activities, although these efforts

achieved more success in some cases than in others. For example, respondents indicated that

community champions conducting CLTS trainings were disproportionately male, a fact that may

undermine the participation of women in CLTS activities and the effectiveness of the intervention.

Further, some vulnerable populations faced difficulty applying what they learned through

trainings. For example, elderly participants in CLTS trainings experienced challenges constructing

toilets following the training, and low-resource communities lacked the necessary supplies to

construct toilets.

Financial resources signify the primary challenge related to sustaining CD efforts, as there is often

limited funding available to sustain interventions beyond the initial period of performance. Despite

broad support among local implementers for sustaining CD activities, they lack concrete plans for

continued financing by GRZ. Respondents regularly mentioned the challenge of maintaining

funding for programming with government funds alone and spoke instead of seeking additional

funding partnerships with donors such as USAID. Several CD interventions rely heavily on unpaid

volunteers, too, which presents a risk to sustainability when those volunteers face a choice between

continuing their support for monitoring of a CD initiative versus pursuing an income-generating

activity. Finally, personnel turnover and insufficient plans to train new staff also threaten the

sustainability of capacity building activities.


3

Illustrative Recommendations

The final section of this report contains detailed recommendations stemming from our research

findings on the relevance, effectiveness, efficiency, perceived impacts, equity, and sustainability

of CD interventions. Table 10 in the final section of this report presents all recommendations and

assigns them a level of priority (high, medium, or low), a timeframe (immediate, medium term, or

long term) and designates the person(s) or organisation(s) responsible for implementing each

recommendation. For example, one of our high priority, immediate recommendations is that during

the initial needs assessment, greater consideration should be given to the potential barriers to

participation and uptake of CD activities for vulnerable groups such as the elderly. Creating user

personas (Dam & Siang, 2019) could help ensure that all potential barriers are considered during

programme design. Another high priority recommendation is to develop strategies and instruments

to assess gender equity in CD programming. A third high priority, immediate recommendation is

to develop plans to co-finance CD programming with GRZ partners from the outset to maximise

the sustainability of CD interventions.


4

1.0 Background

1.1 Growing Emphasis on Capacity Development

In the last two decades, international development sector actors have increasingly recognized that

delivering foreign aid without investing in the capacities1 of local governments, policymakers,

national institutions and civil society organizations, as well as communities and individuals, is less

likely to lead to sustainable and effective outcomes (Zaltsman, 2006). The United Nations General

Assembly (UNGA) has emphasized that capacity development is one of the key strategies to

achieve broader Millennium Development Goals, and called upon UN organizations to “provide

further support to the efforts of developing countries to establish and/or maintain effective national

institutions and to support the implementation and, as necessary, the devising of national strategies

for capacity building” (UNICEF, 2010, p. 2).

Despite the enthusiasm about CD interventions, research cautions that not all CD leads to desired

outcomes and sustainable results. First, if CD interventions only focus on formal policies and

legislation, they may not lead to effective enforcement and significant changes for the end

beneficiaries (Mittler, 2005; Peters, 2007). Second, programming to build up staff capabilities may

not lead to uptake and scale-up (Akerberg, 2001; Organisation for Economic Co-operation and

Development [OECD], 1991). Third, not all existing CD interventions account for local culture

and interests to a sufficient extent (Kalyanpur, 1996; Chaudhry & Owen, 2005). Finally, some CD

interventions have not proven sustainable and have dissipated with the loss of international

consultants and turnover of local staff (Kisanji, 1998).

Ideally, monitoring and evaluation (M&E) is incorporated within CD interventions (when there

are sufficient resources), so researchers can help donors and stakeholders explore: 1) the process,

or how CD interventions have been implemented; 2) the missing parts, or what else needs to

happen to ensure better performance; and 3) the outcomes, or whether interventions have led to

the desired outcomes and improved capabilities. Ensuring that CD interventions incorporate proper

M&E ideally begins at the planning stage and includes input from stakeholders involved in the CD

intervention. As researchers and stakeholders embark on an evaluation, it is necessary to clarify

who needs to use the M&E data, recognizing that different actors will use the findings for different

purposes.

1.2 Capacity Development in the Zambian Context

In 2017, the GRZ implemented the 7th National Development Plan (NDP) for a period of four

years from 2017 to 2020, with a focus on the implementation of a decentralised approach to

development in Zambia (UNICEF Zambia, 2017b). To meet the NDP’s goal of “accelerating

development efforts toward vision 2030 leaving no one behind” and a goal of forming an integrated

approach, UNICEF Zambia implemented strategies to build sustainable and efficient government

capacities across national and subnational levels (UNICEF Zambia, 2017b). For example, the

Ministry of Water Development, Sanitation, and Environmental Protection (MWDSEP) had gaps

1 Capacity can be defined as “the ability to carry out stated objectives” (LaFond & Brown 2003, p. 3; see also Goodman et al.,

1998). LaFond and Brown (2003) define capacity development or capacity building as a “process that improves the ability of a

person, group, organization or system to meet objectives or to perform better” (p. 5). They explain that evaluating capacity

development is “normally more complex than monitoring and is conducted to gain understanding of the relationship between

capacity-building interventions and capacity outcomes, or the links between capacity and performance variables” (p. 5).


5

in infrastructure and resource management capacity; the Ministry of Health (MoH) had gaps in

technical and management capacity due to the presence of under-developed information systems,

weak governance, and centralized decision-making efforts; and the Ministry of Community

Development and Social Services (MCDSS) needed financial and technical support to strengthen

the Alternative Care regulatory framework (African Development Fund, 2006; MWDSEP, 2019;

UNICEF Zambia, 2017b).

To address some of the gaps within the GRZ noted above, UNICEF Zambia aims to strengthen

national systems through the development of the government’s technical effectiveness as outlined

in the Terms of Reference (TOR) (2018) for this evaluation (see Appendix A). UNICEF Zambia

partnered with several ministries to improve the GRZ’s service delivery, emergency preparedness

response, and systems-building by enhancing capacity development. Further, the capacity

development approach is based on participatory training; communication for development (C4D)

strategies; and strengthening the capacities of government systems for data collection,

management, and usage (UNICEF Zambia, 2018b).

To implement the above strategies and programmes to enhance capacity development, UNICEF

Zambia invested a significant portion of the USD 120 million programmed for the UNICEF

Zambia Country Programme (UNICEF TOR, n.d.). For example, for the HMISS CD intervention,

UNICEF Zambia invested a total of USD 3.8 million; for capacity development through C4D,

which includes the CLTS CD intervention under the WASH sector, UNICEF Zambia invested

USD 275,192; and for the institutional strengthening of the national system under the Child

Protection sector, UNICEF Zambia invested close to USD 7.5 million (UNICEF, 2019a). Further,

although in 2018 the GRZ continued to scale up the Social Cash Transfer (SCT) programme after

the main 2017 scale-up without the support of UNICEF Zambia, in 2017 UNICEF Zambia helped

develop the review tool, SCT system development, and expansion through an investment of close

to USD 5.3 million (UNICEF Zambia, 2017b; UNICEF Zambia, n.d.).

1.3 Objectives and Scope of the Evaluation

Within the last several years, UNICEF Zambia supported numerous capacity development (CD)

interventions across different programmes in the form of trainings, professional development, and

community outreach activities. These CD interventions aimed to improve the capacities of

government officials, relevant province/district-level officers, and community-level professionals

and staff to use technology and software in a decentralized manner; to enter, clean, and utilize data

management systems in their decision-making; and to better serve the needs of their beneficiaries.

Despite overall positive feedback from stakeholders and participants, there is currently scarce

empirical evidence on what works and whether CD interventions led to the desired improvements

in the skills and knowledge of local government officials, professionals, and communities

(UNICEF, 2018a). Moreover, little evidence exists about the uptake and sustainability of these CD

activities.

UNICEF Zambia contracted with the American Institutes for Research (AIR) to evaluate UNICEF

Zambia’s CD interventions across various programme areas. The purpose of the evaluation was to

learn from and provide recommendations about UNICEF Zambia’s capacity building strategy. The

specific objective of the formative evaluation was to examine the relevance, effectiveness,

efficiency, perceived impacts, equity, and sustainability of five CD interventions implemented

between 2016 and 2018, particularly to highlight good practices and identify lessons learned across


6

initiatives. Figure 1 provides a visual overview of the CD interventions and their components, and

Table 1 following details the levels at which data on specific CD interventions were collected.

Following the inception meeting in Lusaka with UNICEF Zambia stakeholders (February 27,

2019), AIR and UNICEF agreed to evaluate five CD interventions implemented within the 2016–

18 period across five sectors: health; social protection; water, sanitation and hygiene (WASH);

education; and child protection. UNICEF supported these interventions, reflecting a shift in focus

to increasing investment in the capacities of local governments, national institutions, and members

of civil society.

Drawing on our conceptual framework, we collected data at three levels: the institutional/system level,

the province/district level, and the community level. The research objectives per level include:

• At the institutional/system level, we explored how trainings delivered to officers and

employees of relevant ministries contributed to improved capacities in allocating and

managing resources, using new technology/software, and coordinating systematic adoption of

new skills by other stakeholders in the lower chains of command.

• At the province/district level, we examined how trainings may have improved the capacities

of relevant employees and staff to manage information systems, enter and clean data, and use

improved data management systems.

• At the community level, we explored whether the training participants (e.g., health committee

members, members of the Community Welfare Assistance Committee [CWAC]) share an

understanding of the training goals, and whether trainings improved their capacities to perform

tasks and deliver services to participants.

Figure 1. Overview of CD Interventions Included in the Study


7

Table 1. CD Interventions and Data Collection Levels

Health:

HMISS

Social

Protection:

Scale-Up of

the SCT

Programme

WASH:

Scale-Up of the

M2W Tool and

CLTS

Intervention

Education:

Scale-Up of the

Catch Up

Intervention

Child

Protection:

Strengthened

Alternative

Care System

Institution/system level ✓ ✓ ✓ ✓ ✓

Province/district level ✓ ✓ ✓ ✓ ✓

Community level ✓ ✓ ✓

The research team conducted a desk review that included 45 key informant interviews (KIIs) with

UNICEF programme officers and CD intervention implementers and participants; and nine focus

group discussions (FGDs) with community-level stakeholders, including programme participants.

We collected data in Lusaka as well as in Copperbelt, Eastern, and Southern provinces.

This report is organised as follows: We begin with the existing literature on capacity-building

activities. We then present the five selected CD interventions, the evaluation context, and the

conceptual framework; and then we summarise the research design, including research questions,

methodology, data collection methods, and sampling approach, before laying out the findings from

the evaluation across the six thematic areas of relevance, effectiveness, efficiency,

perceived impacts, equity, and sustainability. We conclude with a discussion of findings and

recommendations.

2.0 Selection of CD Interventions

The Learning Network on Capacity Development advises that the starting point for any evaluation

of capacity-building activities is to clarify the purpose of CD interventions (Pearson, 2011).

Clearly defining the purpose of interventions helps identify the key questions to be answered.

During the inception visit, the evaluation team discussed within UNICEF the scope of CD

interventions and the goals UNICEF wanted to accomplish through these interventions. We later

reviewed a document that mapped CD interventions in Zambia for the period 2014–18.

Drawing from the literature review and UNICEF Zambia’s original terms of reference, our team

communicated a set of criteria for selecting the CD interventions for the evaluation:

• Interventions that involved stakeholders at different levels, ranging from national stakeholders

to province/district stakeholders and community members

• Interventions that were implemented within the 2016–18 period

• Mature interventions

• Interventions that incorporated more than one activity, such as trainings and a technology

dimension

With these criteria, AIR and UNICEF Zambia agreed to evaluate CD interventions in five

programmes: (1) health—the Health Management Information System Strengthening (HMISS)

intervention; (2) social protection—scale-up of the Social Cash Transfer (SCT) programme; (3)


8

WASH—scale-up of the Mobile to Web (M2W) tool for real-time monitoring and scale-up of the

Community-Led Total Sanitation (CLTS) intervention; (4) education—scale-up of the Catch Up

intervention; and (5) child protection—strengthening the alternative care system through

improved case management of vulnerable children and adolescents. Annex D provides more detail

on the implementation of each CD intervention.

3.0 Evaluation Design

Across the selected CD interventions, we analysed how the implementation processes, activities,

and training content used gender inclusive, rights-based, and equitable approaches. For example,

were the training activities implemented at a time and location that enabled all participants,

irrespective of gender and ability, to attend trainings? Figure 2 contains the research questions that

guided our study, which are organised according to the OECD-DAC criteria.

Figure 2. Research Questions

Relevance:

1. To what extent do the intended outcomes and the relevant outputs address the national priorities as

indicated in the National Strategies that culminate into the realization of vision 2030.

2. To what extent are the key strategies in capacity initiatives aligned with UNICEF’s global/regional

priorities?

3. Has UNICEF been able to adapt to the changing contexts- social, political and economical to address

capacity/ systems strengthening needs in the country?

4. Has investment in capacity and systems strengthening across the programmes been relevant to the

needs of children, women, and the other marginalized groups?

Effectiveness:

5. To what extent have outcomes been achieved, are there any additional outcome(s) being achieved

beyond the intended outcome? Did the capacity building/system strengthening interventions lead to

major unexpected results—both positive and negative?

6. How have the results, both at outcome and output benefited men and women? Have they benefitted

equally?

7. Is the current set of indicators, both outcome and output indicators, effective in informing the progress

made towards the outcomes? If not, what indicators should be used?

8. Were the assumptions underlying the programme intervention strategy correct? Which factors, internal

and external to the programme strategies, explain the extent to which results have been achieved?

9. What were the facilitating and constraining factors? What are the challenges to achieving the outcomes?

10. What are consistent patterns and good practices across capacity building and system strengthening

initiatives?

Efficiency:

11. Were the resources and inputs converted to outputs in a timely and cost-effective manner? Did ZCO

use resources in the most economical manner to achieve expected equity-focused results? If so how?

12. Would it have been possible to achieve the same results at lower costs? If so how? What alternative

models exist to achieve the results at the lower costs?

13. What were the most important cost drivers in the delivery of the programme, were costs contained

without compromising results? If so how?


9

14. Were the programme strategies implemented in a timely manner? If not why?

15. Has there been any duplication of efforts among UNICEF’s own interventions and interventions

delivered by other organizations or entities in contributing to the outcomes?

16. How well did the programme coordinate with other, similar programme strategies within the Country

Office for synergy and in order to avoid overlaps/duplication?

Perceived Impacts:

17. Have the capacity building strategies so far contributed or is likely to contribute to medium to long-

term social, economic, technical, changes for individuals, communities, and institutions?

18. What positive/negative, intended or unintended outcomes have the programmes strategies for systems

strengthening contributed to?

Equity:

19. Did the programme interventions reach the worse off individuals or communities like it did for the

other groups?

20. What key barriers-political, economic, social hindered all the affected populations, communities and

institutions?

Sustainability:

21. How strong is the level of ownership of the results by the relevant government entities and other

stakeholders?

22. What is the level of capacity and commitment from the Government and other stakeholders to ensure

sustainability of the results achieved?

23. What could be done to strengthen sustainability?

4.0 Conceptual Framework

We developed a conceptual framework to guide this study based on a review of the literature and

initial materials provided by UNICEF Zambia programme officers. The conceptual framework

broadly maps out the primary activities of the CD interventions. Figure 3shows the broader

conceptual framework for the evaluation, including the initial conditions under which CD

interventions were designed, the levels of CD that UNICEF Zambia programmes sought to build,

activities that were incorporated across CD interventions of interest, and the expected high-level

outcomes. We provide more detail on the conceptual framework below.


10

Figure 3. Conceptual Framework

Initial conditions: The first column in the framework indicates the justification for CD

interventions, including who and what contextual factors and assumptions motivated and enabled

CD implementation (Milstein & Cotton, 2000; Preskill & Boyle, 2008). Capacity building should

result from a needs assessment that identifies existing gaps in service delivery for a particular

programme. The CD interventions identified needs at the policy level, including ensuring that

interventions are implemented using a rights-based approach; in knowledge and capacity for

service delivery among staff at all levels; and the potential for sustained delivery.

CD activities by level: The second column illustrates the activities across all CD interventions for

the three levels at which CD was implemented (Brown, LaFond, & Macintyre, 2001):

1. At the institutional/system level, UNICEF programme officers worked with national-level

ministry officials to design the CD interventions according to both UNICEF and Government

of Zambia priorities in the sectors. In addition, UNICEF contracted external implementers to

deliver a ToT that catalysed CD efforts at provincial, district, and community levels.

2. At the provincial and district levels, the CD interventions trained officials and staff to either

become master trainers or to receive the ToT.

3. At the facility/community level, ministry staff delivered services to the community after

receiving capacity development training. Chefs monitored activities using the technologies

provided by the CD interventions. Ultimately, community members should receive enhanced

services from staff as a result of the CD trainings.

CD outputs: The conceptual framework assumes that CD activities produce the following outputs,

which become the new conditions as a result of the interventions:


11

1. Staff are better able use technology/software.

2. Staff are better able to deliver services to programme participants.

3. Community-level outreach about service availability is both enhanced and increased.

To understand these activities, we looked at how the CD activities took place, including: (a) the

strategies for capacity development (e.g., management skills, analytical skills, knowledge of a new

technology or software); (b) the substance and content of CD activities (e.g., materials that were

used, sources); (c) the delivery methods used for various activities (e.g., ToT, participatory

methods); and (d) the regularity and length of activities. All of these factors are important both

for descriptive purposes and because they mediate effects on outcomes (Durlak & DuPre, 2008;

Rapkin & Trickett, 2005).

Outcomes: The fourth column describes how outcomes are similar across levels. Overall, the CD

interventions are expected to result in policy changes and mainstreaming of enhanced processes

(Boyle, Lemaire, & Rist, 1999; Preskill & Boyle, 2008), including rights-based approaches;

enhanced management, service delivery, and technology capacity; improved capacity in data usage

and decision-making; and improved communication with end users.

Further, depending on the CD intervention and the communication between facility-level ministry

staff and community volunteers, community members are expected to have improved knowledge

and awareness of evidence-based practices such as best WASH practices to meet the goal of Zambia

attaining an ODF status. Tables D1–D5 (annexed) detail the specific outcomes by intervention. It is

important to note that the outcomes for this study are particular to duty-bearers, rather than children,

who were the ultimate beneficiaries but were not the focus of the CD interventions.

Development Assistance Committee (DAC) criteria: Finally, the framework illustrates the

importance of assessing the relevance, effectiveness, efficiency, equity, perceived impacts, and

sustainability of the CD interventions throughout the life of the interventions (final row, Figure 3).

While relevance and effectiveness questions will examine how and why CD interventions came

about and which levels they targeted, perceived impact and sustainability questions are important

for specific interventions and their outcomes.

4.1 Evaluation Methodology

We used qualitative methods to examine the relevance, effectiveness, efficiency, perceived

impacts, equity, and sustainability of the CD interventions. After a desk review of relevant

materials, we conducted 45 KIIs and nine FGDs at the institutional/system, province/district, and

community levels. While the institution/system-level interviews happened in Lusaka, other KIIs

and FGDs took place in Copperbelt, Eastern, and Southern provinces.2

A key informant is a person who possesses comprehensive knowledge about the scope and

processes of the targeted CD interventions. The team conducted KIIs with national, provincial,

and district-level officials from MoH, MCDSS, MoLG, MoCTA and MWDSEP, and members of

DWAC, members of ACC, the RWSS focal person, master trainers and external implementers of

the HMISS and SCT interventions. FGDs with heads of communities, members of community-

2 We discuss our selection of these locations in the “District Selection and Sampling Approach” section of this report.


12

based volunteer committees and groups, and beneficiaries aimed to facilitate a deeper

understanding of the usefulness of CD interventions.

Given the study’s research questions, we examined how the process of designing a CD

intervention was arranged, how those activities were implemented and whether they led to specific

changes. The qualitative methods helped illuminate strengths and challenges associated with

implementing an intervention, as well as how beneficiaries experienced the programme and

translated lessons learned into practice. Figure 4 provides an overview of the evaluation.

Figure 4. Evaluation and Research Design Overview

4.2 Qualitative Analysis

The team audio-record KIIs and FGDs, some of which were in English (in Lusaka) and some of

which were in local dialects. We transcribed and translated all data into English. Transcripts and

relevant documents were uploaded and analysed using the qualitative data analysis software

NVivo. We developed a preliminary coding outline, based on the research questions, interview

protocols, and themes that emerged during data collection. This outline served as a tool for

organizing and subsequently analysing the information.

The team refined themes, categories and theories that emerged throughout analysis. During this

iterative process of data analysis, reduction and synthesis, researchers assessed the prevalence of

responses, examined differences, and identified key findings and themes related to the research

questions.

*Researchers collected data for the formative evaluation of CD interventions under the Education and Child

Protection Section only at the national level.


13

4.3 Sampling Approach

We used purposeful sampling driven both by theory and the geographic scope of CD interventions.

Since each intervention differs in participant size and geographic scope, the number of respondents for

each intervention varied. However, we believe the numbers we allocated for each CD intervention and

each method allowed us to reach saturation, obtain rich information from each category of respondent,

and gather only limited new information from each additional respondent. A qualitative approach

requires sacrifices in terms of generalizability and comparability, and a small sample size (often non-

randomized and purposeful selected to allow researchers to explore and understand the experiences,

opinions and perspectives of their informants in greater depth). Anthropologist Russell Bernard (2011)

notes that “there is growing evidence that 10–20 knowledgeable people are enough to uncover and

understand the core categories in any well-defined cultural domain or study of lived experience” (p.

154). Lusaka-based AIR Zambia staff were trained to adhere to cultural norms when collecting data

and engaging with communities to ensure that our research did no harm to the participants or the

surrounding community.

We selected three provinces to collect data at the province/district level and community level for

the three key CD interventions, using purposive sampling based on our theoretically informed

understanding that provinces and districts which are further away from financial and

administrative centres like Lusaka are more likely to face challenges with accountability,

management of resources and systems, and growing their local staff’s capabilities.

For the HMISS CD intervention we selected Copperbelt province, and within it Masaiti district.

Copperbelt was one of the two provinces where this CD intervention was implemented. For the scale-

up of the SCT programme we selected Eastern province, and within it Katete district. We collected

data for the ZSHP programme in Southern province, and within it Gwembe district. These districts

met our selection criteria while also ensuring efficient, cost-effective data collection.

Finally, we used purposeful sampling to select respondents at each of the following levels:

• Institutional/system level: Interviews with UNICEF programme officers and relevant

ministry officials provided information on how CD interventions were conceived and

designed. These interviews explored how GRZ stakeholders defined needs, and how they

partnered with UNICEF and other donors to select priority areas, organizations, and

individuals to be targeted.

• Province/district level: Interviews with officials and staff members who participated in

trainings, as well as implementers of CD interventions, investigated how the CD interventions

were rolled out and how newly acquired knowledge and skills from trainings was being used

in practice. They also gave insight into how ToTs were delivered, the approaches used in

trainings on new technology/software, what skills and knowledge were expected to improve,

and what outcomes were achieved.

• Individual/Community level: FGDs with training participants and end users of services

explored perceived gains from the CD activities, as well as perceptions about changes in

behaviours, skillsets, knowledge, data use, and decision-making capacities among local

stakeholders. They explored individual and community experiences with service delivery and

mechanisms through which changes have occurred as a result of CD interventions, as well as

unanticipated outcomes and consequences associated with the programme.


14

For each of these levels, we disaggregate our presentation of the findings for men and women only

when responses differed by gender. We trained data collection team members specifically on how

to conduct FGDs to ensure full participation of women. Table 2 summarises the sample for each

CD intervention including the gender breakdown of respondents.

Table 2. Sample for Each CD Intervention

Level Informant

Method and

N Respondents

Gender

Disaggregation

Health—Health management information system strengthening

Institution/system UNICEF programme officer

Central health officers, MoH

M&E Unit officers

1 KII

2 KIIs

2 KIIs

1 Male

2 Males

1 Male,1 Female

Province/district Province-level senior health information officers

District health information officers

1 KII

1 KII

Male

Male

Community Facility-level staff

Community-level health workers

EHTs

Beneficiaries receiving services from health workers

and EHTs

2 KIIs

1 FGD

3 KIIs

2 FGDs

2 Females

6 Females,

1Male

1 Female,

3 Males

6 Males,

9 Females

Social protection—Technical support to develop and expand the government’s cash transfer system

(beneficiary selection)


Officials from the SCT Unit, MCDSS

M&E Unit officers

1 KII

1 KII

2 KIIs

Male

1 Female

1 Male,

1 Female

Province/district Master trainers, district government officials

District social welfare officers

Members of DWAC

Members of ACC or assistant district officers

1 KII

1 KII

2 KIIs

1 KII

1 Female

1 Female

1 Female,

4 Males

1 Female,

3 Males

Community Members of CWAC

Chair of CWAC

Secretary of CWAC

Enumerators

Beneficiaries receiving SCTs

2 KIIs

1 KII

1 KII

1 FGD

2 FGDs

2 Females,

2 Males

1 Female,

2 Males

2 Female,

2 Males

1 Female,

5 Males

14 Females,

8 Males

WASH—M2W for real-time monitoring of WASH interventions and CLTS


Officials from MoH & MWDSEP

1 KII

2 KIIs

1 Male

1 Male,

1 Female


15

Level Informant

Method and

N Respondents

Gender

Disaggregation

Province/district Province-level officials who received training

RWSS focal person

Members of the health promotion team from the district

community

Medical Office

3 KIIs

1 KII

2 KIIs

2 Males

1 Female

2 Males

Community EHTs

Chiefs

Village headmen or headwomen

Community champions/community volunteers

Beneficiaries who receive CLTS services

2 KIIs

2 KIIs

1 KII

1 FGD

2 FGDs

1 Female,

1 Male

2 Males

1 Male

1 Female,

7 Males

8 Female,

8 Males

Education—Catch-up using TaRL approach


Officials from MoGE

1 KII

2 KIIs

1 Female

1Male,

1 Female

Child protection—Strengthening the alternative care system


Officials from MCDSS

1 KII

2 KIIs

1 Male

2 Males

Total: 45 KIIs

9 FGDs

4.4 Methodological Limitations

This study contains several limitations to its methodological approach. First, we cannot determine

programme impacts as a result of the qualitative design of this study. Second, the study cannot

measure outcome indicators at the population level, meaning that our findings are not generalizable

for all of the population where CD interventions took place. Third, like many other retrospective

studies, the study may be limited due to recall bias, or the fact that some respondents may have

difficulty remembering events or factors that shaped their experience, and some sensitive and

socially censured subjects may be distorted (Fenton et al 2001). Last, since the research team

recorded all interviews and focus groups, respondents may have been reluctant to share their honest

opinions.

Despite these limitations, qualitative and retrospective studies can assist in improved understanding

of what works and why it works (Bryman 2012). Qualitative data is valuable in examining the

dynamics of how an intervention/programme worked. Although qualitative indicators may be

limited in establishing causal connections, they can improve our understanding of how stakeholders

perceive the benefits or disadvantages of a programme (Miller & Daly 2013).

To address the limitations, AIR research team collected data from different stakeholders using

several qualitative methods to triangulate methodologically and across different informants. We

explored respondents’ perceptions of how their work may have changed as a result of trainings.

For the three CD interventions in health, social protection, and WASH sectors, we collected data

at institutional, district/province and community levels to achieve greater depth on how


16

interventions were implemented and whether they facilitated needed improvements in target

groups’ skills and knowledge.

4.5 Ethical Considerations

AIR is registered with the Office of Human Research Protection as a research institution and

conducts research under its own Federalwide Assurance. The AIR IRB follows the standards set

forth by the American Evaluation Association Guidelines and the Joint Committee on Standards

for Educational Evaluation. AIR follows the Code of Conduct of UNEG that requires both a

conflict- and gender-sensitive approach to research and adherence to the “do no harm” principle,

among other key principles. AIR respects and adheres to the UN Declaration of Human Rights,

the UN Refugee Convention, the Convention on the Elimination of all forms of Discrimination

Against Women, as well as other human rights conventions and national legal codes that respect

local customs and cultural traditions, religious beliefs and practices, personal interaction, gender

roles, disability, age, and ethnicity.

We received approval from AIR IRB and we also obtained approval local IRB from Zambia-based

research ethics board: ERES Converge IRB, which is governed by the National Health Research

Ethics Committee. AIR’s researchers and data collectors were trained and certified on research

ethics from relevant national-level ethics board such as the National Institutes of Health (NIH)

Office of Extramural, USA. Further, a lead researcher from AIR USA was in Zambia in June 2019

and trained local data collectors on the importance of consent and maintaining participant

confidentiality. Local data collectors gained informed consent from participants only after sharing

all important information as outlined in the IRB approved consent form. Verbal informed consent

was also obtained from each participant only after vital information on voluntary participation,

confidentiality and the research study were shared and understood by the participant. The informed

consent procedures complied with both the local Zambian ethics board and AIR’s consent

requirements. AIR handled all data in accordance with the procedures and protocols approved by

both IRBs and all AIR computers are encrypted and password-protected.

Throughout the evaluation, the AIR research team maintained its independence and impartiality

and can therefore attest to the credibility of all findings presented in this report. No members of

the research team had any personal affiliations or past connections to the projects or project staff

included in the evaluation.


17

5.0 Findings

This section presents findings from qualitative research across the five CD interventions related to

their relevance, effectiveness, efficiency, perceived impacts, equity, and sustainability. Although

organized by these categories, many of the findings are crosscutting.

5.1 Relevance

The evaluation showed that each of the CD interventions resulted from some type of needs

assessment that helped ensure the relevance of the activity within the sector. Specifically, UNICEF

and the GRZ worked together in an ongoing manner to ensure that the interventions aligned with

the priorities of both organisations. Although the needs assessments conducted prior to each

intervention helped ensure that some priorities were addressed—primarily at the national level—

incorporating additional needs assessments with community members may have increased the

programmes’ relevance for marginalized groups. This section presents findings about the

relevance of the CD interventions to national strategies, UNICEF priorities, and the social,

political, and economic context of the country.

• To what extent do the intended outcomes and the relevant outputs address the national

priorities as indicated in the National Strategies that culminate into the realization of

vision 2030?

• To what extent are the key strategies in capacity initiatives aligned with UNICEF’s

Global/Regional priorities.

CD Approaches Aligned With UNICEF and GRZ Priorities

UNICEF CD approaches purposefully aligned with existing GRZ systems, which made CD

interventions appropriate and relevant for UNICEF priorities and GRZ National Strategies for

Vision 2030 (National Planning Commission, 2013), which are described for each programme in

Annex D. The five CD interventions selected for this study focused on the integrated and improved

use of technology for decision-making and ensuring that the technology could be streamlined into

existing national systems. Incorporating electronic monitoring into existing government data

systems made data collection and training more relevant in light of existing systems. Integration

also potentially increased the likelihood that the capacity building will have a continued influence

on the accuracy and usefulness of data systems from the programme. A ministry representative

working in child protection described how staff were planning to create an integrated system to

track child protection indicators across various sectors:

“That call centre will store data from the national registry, anti-corruption commission,

drug enforcement commission, and the Zambia Police. It will be like our central data

centre, meaning that even the manner in which we report these cases may improve such

that we’ll have to report cases electronically. There will be an improvement in that,

because whatever information regards to child protection unit … they’ll be able to provide

it to you electronically.”

One representative from UNICEF also indicated that the organisation aimed to “Generate

evidence that the ministry can start using to transform their priorities in terms of what they put in

the midterm expenditure framework.”


18

• Has UNICEF been able to adapt to the changing contexts—social, political, and

economical—to address capacity/systems strengthening needs in the Country?

• Has investment in capacity and systems strengthening across the Programmes been

relevant to the needs of Children, Women, and the other marginalized groups?

Needs Assessments Ensured CD interventions Addressed Relevant Gaps

The focus of each of the CD interventions was highly relevant to real needs in each sector and, in

most cases, what based on a formal research process that identified those needs. For example, the

TaRL programme selected pilot districts for CD based on level of need as indicated by low

performance on national assessments. UNICEF indicated that it also conducted some action

research for TaRL, which enabled teachers to design their own lessons and assess their

performance. The programme also piloted three different models and conducted a process

evaluation based on research in other countries to assess which model would work for Zambia.

Table 3 describes the extent of the needs assessments conducted to inform the design of each

intervention.

Table 3. Description of Needs Assessments by Intervention

Intervention Description of Needs Assessment

Health—Health management information

system strengthening

Identified gaps in management information using HMIS

assessments conducted by other organizations.

Social protection—Technical support to

develop and expand the government’s cash

transfer system (beneficiary selection)

UNICEF-administered global needs assessment identified

technical misunderstanding and lack of use of MIS.

WASH—M2W for real-time monitoring of

WASH interventions and CLTS

During implementation of Zambia Sanitation and Hygiene

Program, identified need for timely monitoring data.

Education—Catch Up using TaRL approach Utilized study by J-PAL and Pratham showing importance of

TaRL for improving literacy and numeracy performance; carried

out a process evaluation of three models to test approach in

Zambian context.

Child protection—Strengthening the

alternative care system

UNICEF supported Ministry of Child Development to conduct

nationwide assessment of status of children in residential care.

A ministry official who works on TaRL explained that continuing assessment would help the

programme to understand how to continue making cost-effective investments:

“The fiscal space is very tight, so that would be a challenge. We … haven’t started, but are

thinking of generating evidence around public financing for children. [We] want to do an

assessment—of the workload, of officers in Ministry of Community Development (MCDSS)

insofar as providing alternative care and prevention of family separation. [We] hope to do

a budget expenditure review to see trends over past years and do a cost analysis to see the

cost of providing alternative care services as well as family separation and response

services.”

A representative from the UNICEF Child Protection section said that CD efforts had stemmed

from needs identified by the Ministry:


19

“We support to see gaps and help them start the journey. We are not at that point where

we can let go, [the Ministry] still needs UNICEF to support in this area. After a few years

we will get to a point where we revive the practice of case work as method for providing

services to children in need. [We] have continued the dialogue, and [the Ministry is] seeing

the tools being used and data being generated. There are talks of having an integrated MIS

at the national level. All these are building blocks to where we want to go.”

Finally, UNICEF representatives at all levels indicated the importance of co-creating the CD

interventions with relevant ministries from the GRZ based on assessment of current need and

planning for the future.

Focus on Community Level During Needs Assessment Could Increase Relevance of

Interventions for Marginalized Groups

Although each of the CD interventions assessed the greatest needs to design their programmes—

for example, through extant education data and Ministry-level priorities—data from the

community level indicated that increasing the focus on local-level realities and needs from the

beginning could increase the relevance of the programming, especially for marginalized groups

such as women and children (see Table 4). We provide examples of community-level challenges

that could be easily addressed throughout the report, such as ensuring that female trainers are

available to female community members and ensuring that training takes place at a time that is

convenient for women. Troubleshooting potential challenges to implementation stemming from a

lack of resources or lack of movement at the higher levels could increase communities’ ability to

continue activities. In some instances across all sectors, participants indicated that marginalized

groups such as women, people with disabilities, and elderly people did not realise the same level

of benefits from the CD interventions.

Table 4 below summarises the main findings related to relevance.

Table 4 Summary of Relevance Findings

Key Relevance Findings

CD Intervention and Respondent-Level

Evidence Comes From

1. CD approaches were aligned with UNICEF and GRZ

priorities such as the push to incorporate mobile technology in

data collection and management.

• HMISS (central)

• SCT (central)

• CLTS/M2W (central)

• TaRL (central)

• ICM (central)

2. Needs assessments helped identify gaps in capacity that

informed CD interventions but the assessments themselves

varied in comprehensiveness and rigour.

• HMISS (central)

• SCT (central)


• TaRL (central)

• ICM (central)

3. Greater focus on community-level perspectives during needs

assessment could increase the relevance of interventions,

particularly those targeting marginalized groups.

• HMISS (community)

• SCT (community)


20

5.2 Effectiveness

Overall, our analysis indicated that the CD interventions under the health, WASH, social

protection, and education sectors largely achieved intended outcomes as described in the objectives

of each of the CD interventions. Specifically, analysis showed that the CD contributed to improved

access to accurate data, improved data management, and motivation among community volunteers.

Facilitating factors included government and community buy-in, though inadequate financial

support for ICT and transportation hindered some aspects of implementation. CD interventions

could also improve effectiveness for marginalized groups, especially women and people with

disabilities. This section details our findings on the effectiveness of the five CD interventions.

• To what extent have outcomes been achieved? Are there any additional outcome(s) being

achieved beyond the intended outcomes? Did the capacity building/system strengthening

interventions lead to major unexpected results—both positive and negative?

CD Interventions Improved Access to Accurate Data

Respondents such as an MWDSEP official, a district-level WASH coordinator, EHTs, a Provincial

Real Time Monitoring Coach on WASH (RTM WASH) from MoLG, a Provincial Social Engineer

from MoLG, and a District Sanitation Focal Person who received the RTM/M2W CD intervention

trainings under the WASH sector said they had increased access to accurate, complete, ground-

level data from different districts and wards and that this access increased their capacity to monitor

sanitation and hygiene-related data and make real-time data-driven decisions simultaneously. A

social engineer from the MoLG said, “If I go in the system I’ll be able to know the progression

rate in terms of ODF status, the number of users [within] the sanitation coverage and also I will

be able to guide the district on which wards are not performing well and help with the location of

resources.” A district-level WASH coordinator shared the same sentiment, “Just the aspect of

having the real-time data as the name suggests has had quite some beneficial aspects to the district,

you can make a remote supervision from the office and you’ll be able to get the expected results.”

Across respondents, the availability of ground-level data through M2W RTM (through the DHIS2

platform) was a key positive outcome as it relates to community monitoring and surveillance of

sanitation and hygiene interventions.

HMISS CD Intervention Improved Data Management

Under the HMISS CD intervention, EHTs from health facilities reported that they had increased

capacity at the facility level to use the DHIS 2 tool to check for data quality, manage data accurately,

assess data appropriately, and use data to make decisions that serve the community and improve

service delivery. Health staff at the province, district, and facility levels also used the updated DHIS2

tool, H1A4B form (an updated form used in HMIS system) and the scorecard app to effectively

analyse data. These enhanced capacities helped facilities to self-assess performance, as well as to

assess the performance of other facilities. Further, the HMISS CD intervention seems to have

improved the capacity of community health workers to make data-driven decisions to meet the needs

of users. For example, a staff from a council health facility stated that both facility staff and

community volunteers were effectively using HMIS:

“Apart from just us members of staff, the community has also learnt to honour data that

they submit. It is no longer only a matter of routine to just enter data. For example, if it is


21

the neighbourhood health committees […] see that the reports show lots of diarrheal cases,

they are able to bring that report to the facility and state that this is a problem in the

community. With the improved knowledge on data analysis and management, they are able

to link it with a facility within that particular zone with high incidence of diarrheal cases

and direct specific support to the facility to respond to the cases.”

The same council health facility staff member suggested that improved reporting helped staff at

the council facility to realize high disease burden and prioritize responses to reduce the burden.

For example, when data indicated that the incidence of malaria cases was high within a community,

Malaria Task Force Committees were developed, and community health workers directly assessed

the risk factors for the disease burden and responded accurately. However, an M&E officer from

the MoH pointed out that the use of the updated DHIS2 tool through the HMISS CD is only one

factor among several that is perceived to have led to improved health delivery. The officer said the

DHIS2 tool particularly helped monitor performance and create demand for service improvements.

Another positive result of the HMISS CD intervention was the significantly improved timeliness

of data reporting at the facility level.

CD Helped Increase Motivation Among Community Volunteers

Community champions and Community Health Workers (CHWs) said the RTM/M2W, CLTS

triggering, and HMISS CD trainings helped increase motivation to ensure data quality, report data

on time, and input and analyse data because it informed decisions at the ministry levels. For

example, a district-level coordinator for the RTM CD training said that the provision of telephones

to community champions to enter data provided them with motivation to respect their work. The

respondent said, “The understanding by community [champions] that the information that they

report on is entered into a system for every Zambian to see … has motivated them to respond to

what [the CD trainings] taught them.” A council health facility staff also said HMISS trainings

helped the CHWs to respect the data they submit and equipped the staff at the council to speak

confidently on the data, “It has also given us the confidence and also ownership to say, ‘we are

the owners of this data so let’s work hard’.”

• How have the results, both at outcome and output, benefited men and women? Have they

benefitted equally?

Participation Not Fully Gender-Inclusive

The CD training for scale-up of the TaRL and CD trainings for the strengthening of the alternative

care system under CP included training to use a gender-inclusive participant selection criterion.

For example, a child protection officer stated that if the gender balance of the recipients was not

met, the training workshops were not approved. In relation to the gender balance for the TaRL CD

trainings, due to the presence of a larger number of female teachers at the primary school level and

the presence of more women as District Education Board Secretary (DEBS) and District Senior

Education Standards Officer (DSESO) at the district level of the MoGE, gender balance was not

attained. Despite this trend, an MoGE official said mothers insisted on attending training

workshops:

“The ladies who had small children would come with a nanny to the training, so they would

be attending training and outside the hall, the nannies would be taking care of the babies


22

and then when there was an opportunity during break they would go and breast feed, then

continue with the trainings so it was a flexible way that made sure that everyone

participated fully.”

However, the Ministry’s efforts to encourage mothers to attend trainings did not speak to those

who were unable to afford or make arrangements for childcare during trainings.

Community champions, an EHT, RTM DWASH coordinator personnel, and a national-level staff

from the MWDSEP said there were few or no female community champions. Further, mostly men

attended CLTS triggering and sensitization meetings delivered by community champions. An EHT

from a health facility said, “When you have a community meeting, you find that only men

participate and I don’t know if women feel uncomfortable, [but] they just can’t stand when there

are men around so [even the answers to the questions you ask] come from men.” The primarily

male community champions and the delivery of CLTS concepts to mostly male community

members indicates a gap where females do not have opportunities to ask questions that meet their

needs, such as on menstruation hygiene, and female community members do not have enhanced

knowledge on WASH best practices and CLTS concepts.

• Is the current set of indicators, both outcome and output indicators, effective in informing

the progress made toward the outcomes? If not, what indicators should be used?

Emphasis on Top-to-Bottom Monitoring Visits

Across the CD interventions, most monitoring activities were conducted via the top-to-bottom

approach where, for example, facility monitoring would be conducted by provincial-level or

national-level staff. National-level staff across the five CD interventions stated that they monitor

the district and community level facilities and units such as the CPU at the district level and health

facilities in the community. The monitoring of CPU staff and teachers’ capacity and application

of CD knowledge was conducted through occasional site visits and evaluation of the quality of

reports received at the provincial level from the district-level staff. For example, a UNICEF officer

for education stated that monitoring visits were conducted as frequently as possible. Similarly, a

child protection officer reported that monitoring was conducted through a “provincial oversight

visit,” and the application of knowledge gained from the CD training was evaluated based on trends

indicated by reports. A CPU officer explained,

“Sometimes when there is an increase in the number of cases recorded it means that the

members of the public have been informed, that they have been given the knowledge that if

there is an abuse which has been suspected- they are free to report to the police. Sometimes

when there is an increase in the number of cases solved, it means the officers are doing a

very good job whereby the officers are booking all the perpetrators of child violence

against children.”

Scorecard Tool and Appraisals Allowed for Internal Monitoring

Monitoring of district-level MoH staff was also conducted via report analysis. Health status was

monitored by health indicators, and service delivery by facilities was monitored by the use of

performance assessment tools. The health indicators were tied to the MDGI initiative and

specifically to maternal and child health service indicators. Some of the indicators such as


23

immunization status, number of skilled deliveries, number of institutional deliveries, incidence of

teenage pregnancies were used to help monitor the status of the quality improvement (QI) projects

as described above. Further, the scorecard tool within the DHIS2 was used by both facilities and

provincial-level staff to monitor the progress of facilities and compare indicators over time. A

district-level health information officer said, “For indicators that were not performing well, the

score cards would tell you that you are in the red zone, which indicates that you are not performing

well. And as you are in the red zone, you can even intensify your efforts to improve”. The scorecard

tool was also used by district-level staff from the MoGE to provide progress reports to the

provincial levels on local schools that had implemented the Catch Up intervention. Some of the

indicators that were used to monitor the reports were the teacher performance index using TaRL

initiative, school assessment, the number of students who were taught through the TaRL approach,

and others. A national-level MoGE official said that monitoring of teachers who were trained on

TaRL and schools that had implemented the Catch Up intervention were not only internally

monitored by mentors and senior teachers who were trained as part of the CD training on TaRL

but also by provincial-level MoGE officials.

Another way that M&E was conducted was through the use of annual performance appraisals of

staff, which informed not only the monitoring of staff capacity changes after CD training but also

future needs assessment and gaps in CD training content. For example, under the CD training for

SCT beneficiary selection, a social welfare officer said, “So these assessments also help in terms

of needs assessments for capacity, it’s not only for punishing people, you may want to know as to

why people did not do certain activities and if they did certain activities very well—you may also

want to know why they did so well so that you can learn from that. It could be one area where

officers are not performing well, then you identify that area.” It is important to note that some

recipients such as an EHT from a health facility and other staff who delivered and received CD

training for WASH M2W/RTM CD training reported that there was no specific M&E framework

present. Similarly, one social welfare officer from the national level stated that the M&E

framework was currently being designed.

• What were the facilitating and constraining factors? What are the challenges to achieving

the outcomes? What are consistent patterns and good practices across capacity building

and system strengthening initiatives?

• Were the assumptions underlying the programme intervention strategy correct? Which

factors, internal and external to the programme strategies, explain the extent to which

results have been achieved?

Government Buy-In and Commitment Important Facilitating Factors

Government commitment and buy-in helped facilitate CD trainings. A UNICEF WASH sector

officer said it was important that CD interventions were perceived as part of existing government

structures and strategies instead of as independent donor programmes. For example, according to

the UNICEF WASH officer, the official recognition of the DHIS2 tool as an information

management tool by the Zambian government after the implementation of the HMISS and

M2W/RTM CD trainings continues to pave the way for the allocation of resources and trainings

on DHIS2. Similarly, the integration of the M2W/RTM training as part of the national ODF

strategy facilitated the implementation of the M2W/RTM training. To further inform sustainability


24

of the use of the DHIS2 tool for M2W/RTM of sanitation and hygiene data, future implementation

of the DHIS2 tool is a part of the government’s 2020 plan, which indicates strong government

commitment and buy-in. An example of government commitment to ensure that the HMISS CD

training continues to be applied is the allocation of a national-level MoH staff stationed only to

respond to the challenges faced by provincial-, district-, and facility-level staff regarding DHIS2

tool use for HMIS, thus facilitating continuous learning and application of HMISS CD teachings.

In addition, the overlap of the child protection strategies with the government’s national child

development policy and its integration into the national framework has facilitated the trainings to

strengthen case management for alternative care services.

Generating District- and Community-Level Demand Facilitated CLTS CD

Districts created demand for the M2W/RTM and CLTS trainings, thus building an enabling

environment for these CD trainings and facilitating the implementation of these trainings. A

DWAC coordinator explained this demand from districts:

“I think one of the motivating factors was the need from the district. Especially for CLTS,

a district would request either through UNICEF or AKROS that they need this kind of

training because they feel the ODF status is low. And from the office of the WASH

coordinator, they would also request that we need a DHIS2 orientation or training because

of such kind of needs assessments… It was a bottom-up kind of approach that motivated

trainings [to] really have meaning and attain some desired goals.”

Continued focus on CLTS and WASH best practices was also attributed to the desire and demand

of community members to see positive changes within their communities. For example, a District

Sanitation Focal Person said,

“If individuals were not interested in changing their lives, we would have not gone

anywhere…but the desire started with the community members, even the village headmen

were motivated when they were given this concept. We had meetings at the Chief’s place,

the headmen were trained, so they also got motivated, they wanted to really see that change

as well.”

Further, a Chief’s representative also emphasized the importance of having buy-in from the

headmen to ensure that community members attend sensitization meetings and apply the

knowledge they received from the CLTS triggering.

ICT Equipment Hindered Implementation at Times

Technical difficulties were a common constraining factor across the CD interventions

implemented under the health, WASH, and social protection sectors. Since these CD interventions

included scale-up through the enhanced use of technological tools such as DHIS2, MIS, Form 2

and H1A4B forms, the presence of faulty or unreliable ICT equipment such as desktops, laptops,

and airtime/Internet network issues proved an obstacle to continuous data entry and management.

Community-level respondents such as community champions, community health workers and

enumerators said they experienced difficulties reinstalling apps used for data entry. In addition,

WASH-related community champions, CHWs and SCT enumerators relied on airtime or data

bundles to enter data into the phone in a timely manner. The provision of airtime from the


25

government was even more important as community champions and CHWS were volunteers, who

received no stipends or payments for their work.

Some district-level respondents who received the M2W/RTM training and the DHIS2 training

under HMISS reported that they were no longer able to log-in to the system to view the data at the

facility level, reducing their ability to effectively monitor the facilities. Similarly, a DWASH

coordinator said there were gaps in capacity building for staff at the district level through the scale-

up of the M2W/RTM CD training. The district-level WASH coordinator explained this

shortcoming:

“So, whilst the Ministry would want WASH coordinators to do something; one such thing

would be [the] addition of a village. As a WASH coordinator, we have not been given the

log-in rights to add a village but there are villages which continue to be created by Chiefs.

I feel these new villages need to be added by people on the ground, which would be the

district staff.”

The coordinator elaborated that the provision of log-in rights to provincial-level partners only

prevented district-level staff from capturing essential ground-level data that inform national-level

sanitation and hygiene data.

Finally, respondents from three health facilities confirmed that after the CD trainings, computers

were no longer connected to the Internet. This challenge prevented staff from using the DHIS2

tool, thus reducing their ability to apply knowledge and skills gained to achieve intended outcomes.

In addition, not all facilities received the appropriate ICT equipment. One respondent who was

transferred and who is currently in charge of a council health facility said, “Here we are using

paper, we are using hardcopies. At the facility where I was, we were using the system and then

coming [back] here to hardcopy… I’m not using much of the skill and knowledge that I got from

the training.” An EHT from a council health facility said, “Here, I don’t [use the DHIS2 tool], I

can easily forget what I was doing but if I went to another facility [with a computer and accessible

DHSI2 tool] … I could continue, but it’s now almost a year and I haven’t done such kind of things.”

Another EHT from a district health facility said that since February of 2019, the computers had

been ‘locked’ and could not be used, forcing them to go back to the paper-based system. Further,

not having access to the DHIS2 tool prevented staff from continuing to compare their facility’s

performance across the district and self-monitor to develop important improvement plans.

Unreliable Transport Led to Inadequate Outreach and Screening of Community

Members

Unreliable transport and lack of transport to rural areas constrained the implementation of CD

training activities and, potentially, the application of the knowledge gained from the CD trainings.

Enumerators who received training for the scale-up of the SCT beneficiary selection said they did

not have enough reliable transportation vehicles such as motor vehicles or bicycles to reach rural

villages, which were at least 15–20 kilometres from the main point of drop-off. Enumerators said

the long distances meant they were unable to visit and select potential beneficiaries at far-off

villages and could only assess 4–5 households in a day.

Similarly, community champions who were trained in M2W/RTM reported that they found it

difficult to capture ground-level data and sensitize all households within a village due to lack of


26

working bicycles. A community champion said the issue of transport is crucial to ensure

sustainability of CLTS triggering: “We don’t just monitor the villages close to us but far villages,

so if the issue of transport is not properly worked on, then this programme will not continue

because we will just be monitoring the villages close to us. I see [the issue of transport] as a major

challenge to the programme not continuing.”

Table 5. Summary of Effectiveness Findings

Key Effectiveness Findings


Evidence Comes From

1. The CD interventions improved provincial staff’s access

to ground-level data and enhanced their capacity in M&E

and ability to make decisions based on real-time data.

• M2W/CLTS (province, district,

community)

• HMISS (district)

• SCT (district)

2. The HMISS CD intervention helped improve data

management through the use of improved data entry and

data analysis tools such as the DHIS2 system and scorecard

application.

• HMISS (central, province, district)

3. Trainings on CD for community volunteers helped

increase volunteers’ motivation.


• M2W/CLTS (district, community)

4. Some CD interventions were not delivered to a gender-

balanced group of recipients.

• HMISS (central, district)

• M2W/CLTS (central, district,

community)

• TaRL (central)

• ICM (central)

5. Across the five CD interventions, top-to-bottom

monitoring and supervision visits took place, thus

keeping district-level and facility-level staff accountable.

• HMISS (central)

• M2W/CLTS (central)

• SCT (central)

• ICM (central)

• TaRL (central)

6. Internal monitoring of staff was enhanced by the data

analysis tools such as scorecards


• M2W/CLTS (district)

• TaRL (central)

• SCT (central, district)

7. CD interventions that had strong government buy-in and

which were incorporated into existing government

strategies functioned smoothly.


• TaRL (central)


• ICM (central)

8. Implementation of CLTS CD intervention was driven by

district-level demand.

• M2W/CLTS (province, district)


27

Key Effectiveness Findings


Evidence Comes From

9. ICT and technical challenges such as lack of funds for

data bundles, poor wireless connectivity, and faulty ICT

equipment hindered the application of skills learnt

through some CD interventions such as using mobile

technology for data entry.

• HMISS (district, community)

• SCT (district, community)



28

5.3 Efficiency

While the CD activities evaluated under this study were largely delivered efficiently and on

schedule, some common challenges existed regarding the timely provision of necessary inputs.

Further, the cost of transportation for participation in and monitoring of CD activities is

considerable. While the ToT model is perceived as cost effective, it can be deceptively expensive

and time-consuming. This section presents detailed findings related to efficiency.

• Were the resources and inputs converted to outputs in a timely and cost-effective manner?

• Were resources used in the most economical manner to achieve expected equity-focused

results? If so how?

Resources Used Efficiently; Primary Challenge is Transportation

Qualitative data suggest that the primary challenge related to efficiency across the five CD

interventions is the high cost of transportation. This is not to say that UNICEF and partners are

spending money unnecessarily or inefficiently on transportation, but simply that transportation is

a major (if inevitable) drain on resources. We identified several ways UNICEF and partners

consistently used resources and inputs efficiently in CD programming, such as maximising the use

of locally available materials, developing reusable training modules, carefully assessing the

number of inputs required for a given CD activity and convening trainings in zones to eliminate

the need to house participants overnight. We did not collect quantitative data on programme costs

or programme impacts as part of this study, so our assessment of cost effectiveness is based solely

on our analysis of perceptions reported by qualitative respondents. Below, we discuss the

perceived resource efficiencies and inefficiencies in greater detail. The issue of equity is addressed

at length in a later section of this report.

While the high cost of transportation and limited options for transportation to remote areas was

the most prevalent efficiency concern across the five programmes, other perceived inefficiencies

include the high cost of convening workshops, delays with fund distribution for capacity building

activities, limited mobile network availability, and equipment such as laptops breaking down.

Respondents reported that funding delays interfered with training delivery under TaRL and

training for CWACs under the SCT. In the case of TaRL, however, the respondent from MoGE

shared that funding delays were often due to late submission of funding requests for trainings and

workshops. Respondents involved with M2W repeatedly noted challenges with laptops not

working, suggesting that equipment quality may be a source of inefficiency in the M2W rollout.

Resources Used in Economical Manner

Across the five programmes, several examples of using resources in the most economical manner

emerged. For TaRL, the methodology itself relied on the use of locally available materials rather than

more expensive learning materials available for purchase. While trainings for CLTS community

champions required expensive inputs (mobile phones), district-level official reported conducting a

careful assessment of exactly how many phones were needed to train community champions:

“…we have laboured to just do a needs assessment in terms of how many phones are

needed, each community champion would want a new phone but we have stuck to who


29

really needs a phone for the training to be done properly and phones that are being used

for practicals are the same phones that champions have gone away with.”

In addition, community champions were called to zones within their respective districts for the

M2W training, which eliminated the cost of lodging during training. Similarly, for ICM

strengthening efforts, UNICEF worked with MCDSS to develop standardized training materials

(including a module) on case management for children rather than convening expensive workshops

to train staff on the specifics of case management for children. Finally, it should be noted that there

were exogenous factors that improved programme efficiency over time, such as the wider use of

smartphones, which respondents reported has made it much easier to train community champions

on M2W.

• Would it have been possible to achieve the same results at lower costs? If so how? What

alternative models exist to achieve the results at the lower costs?

• What were the most important cost drivers in the delivery of the programme? Were costs

contained without compromising results? If so how?

No Clear Way to Deliver Comparable CD Support at Lower Cost

Respondents across the five programmes did not suggest lower cost alternatives to delivering the

CD support offered through their programmes, and there do not appear to be ways to provide

comparable capacity building at a reduced cost. As discussed in the previous section, programmes

already use several approaches to maximise available resources and limit unnecessary spending.

In addition to these examples of using resources efficiently, an informant from the MoH noted the

existence of a WhatsApp group for DHIOs that served as a cost-effective model for

troubleshooting common problems with data entry and management. Future CD programming—

particularly CD activities involving technology—could potentially consider WhatsApp groups as

an alternative to in-person support for troubleshooting.

As discussed in the section on sustainability, it is possible that transitioning programmes to

government implementation will lead to reduced cost and greater efficiency. For example, an

informant from MoGE noted potential efficiencies to be gained (at least in terms of time) once

TaRL is “mainstreamed in the Ministry of Education curriculum and the time tabling,” but noted

that the programme is still very much in the pilot phase. Similarly, the cost of involving an external

organisation such as Akros for M2W/CLTS training should diminish over time.

Transportation is a Key Driver of Costs

Transportation is a key driver of costs for capacity building activities across the five programmes.

Speaking of the costliest aspects of delivering TaRL trainings and follow-up support, a MoGE

informant lamented the considerable transportation costs: “…at the moment [TaRL] is…limping in

terms of funding because they need transport, they need basic resources to make those outreach

programmes…currently we’re struggling to fund them.” UNICEF noted the limited transportation

resources available to TaRL mentors, particularly to conduct monitoring at the district level.

Respondents also commented on the high cost of HMISS trainings, despite the adoption of the ToT

model. While not explicitly part of capacity building efforts, respondents also referenced the high

cost of data collection associated with the five programmes included in this study.


30

• Were the programme strategies implemented in a timely manner? If not, why?

CD Support Delivered Relatively Promptly

The five programmes delivered capacity building support in a relatively timely manner; but some

respondents reported delays in the delivery of inputs required for training and the training itself

taking longer than anticipated. Respondents reported that delays receiving required inputs (such as

mobile phones and laptops, in the case of the SCT, M2W, and HMISS trainings) led to late delivery

of trainings. To this end an SCT Master Trainer stated, “Phones were not yet in by the time we

wanted them to be in, we were a bit delayed.” In addition to technological equipment, some

respondents such as this SCT Master Trainer reported delays receiving training allowances:

“…we were a bit delayed and the worst was the monies issues, we were training people

who were supposed to be paid, but by, in fact, that training; ok it was a general problem it

wasn’t just for us in Western, all training sessions were experiencing the same…I don’t

know for what reasons monies were not given in time so that really affects the participants’

motivation levels also.”

Respondents also mentioned funding delays impeding TaRL and ICM trainings, and UNICEF

noted that contracting delays slightly impeded the HMISS CD work that otherwise ran smoothly.

According to one respondent from UNICEF, while the ToT approach is “more cost effective,” it

presented a challenge in terms of the time required because “there were not enough experienced

staff to provide quality training, so the trainings tended to take longer within the districts and

wards compared to the timeframe we had earlier thought.” Finally, poor network coverage led to

minor delays in delivering the practical components of the M2W and HMISS trainings.

• Has there been any duplication of effort among UNICEF’s own interventions and

interventions delivered by other organisations or entities in contributing to the outcomes?

• How well did the programme coordinate with other, similar programme strategies within

the Country Office for synergy and in order to avoid overlaps/duplication?

Limited Duplication of Effort, but Opportunities Exist to Enhance Coordination

UNICEF’s preferred strategy of delivering CD support through existing structures at the national,

provincial, and district levels minimised duplication of effort between UNICEF and the respective

GRZ line ministries. This strategy was achieved particularly well with M2W/CLTS, where ToT

trainings followed a logical flow from the national level to the provincial water and sanitation team

and down to the district, ward, and village levels. While Akros was involved to support the

trainings, ministry staff (from MWDSEP and the MoCTA) and RTM coaches reported bearing

primary responsibility for delivering trainings. Respondents also agreed that the appropriate

individuals were involved in M2W trainings: the provincial water and sanitation team was

multisectoral, with representatives from the MoLG, the MoH, MoE, and MoCTA. Further, one

district-level respondent involved with M2W/CLTS noted that provincial officers in the WASH

sector sought to integrate ground-level activities such as community triggering on CLTS and data


31

capturing with community health workers who work under the HMISS intervention to save costs

but still have the opportunity to reach out to communities on CLTS concepts.

At a micro level, there were a few examples of CD activities that could have been better

coordinated. In the case of HMISS mentorship, for example, a DHIO recounted the following:

“Sometimes there’s maybe antenatal and they are quite many outside waiting so you can’t say

stop attending to these people we have come for mentorship. We just make another appointment

and make another time.” Several other respondents mentioned difficulty scheduling TSS or

mentorship visits, suggesting that it might be possible to better plan these visits to avoid

particularly busy days, such as antenatal clinic days. Respondents from all five programmes spoke

of other programmes with similar or complementary objectives, but not of shared mentorship,

training, or capacity building efforts across programmes. This finding suggests there may be

opportunities to streamline CD across programmes—and perhaps even across sectors—to

maximise resources.

Table 6 below summarises the main findings related to efficiency and what information sources

led us to draw these conclusions.

Table 6. Summary of Efficiency Findings

Key Efficiency Findings


Evidence Comes From

1. There does not appear to be an obvious way to deliver

comparable CD support at lower cost, and the five CD

interventions included in our study incorporate

measures to minimise unnecessary costs and maximise

resources.


• ICM (central)

• SCT (district)

• TaRL (central)

• M2W/CLTS (central, district)

2. Transportation is perceived as the key driver of costs in the

implementation of CD activities. • HMISS (central, district)

• M2W/CLTS (central, district)

• SCT (district)

• TaRL (central)

3. CD support is consistently delivered promptly, but at times

there have been delays deploying equipment, funds, and

personnel to support trainings and other CD

interventions at the district and community levels.


• ICM (central)

• M2W/CLTS (district)

• SCT (district)

• TaRL (central)

4. There appears to be limited duplication of effort relative to

CD activities, but further opportunities exist to enhance

coordination across programmes and sectors.


• CLTS (district)


32

5.4 Perceived Impacts

This study cannot determine actual programme impacts, as qualitative data cannot be used to

establish causal connections. However, qualitative indicators can inform the understanding of how

different stakeholders perceived the benefits or disadvantages of a programme (Miller & Daly,

2013). Overall, respondents perceived improved capacity to use technology, understand data,

inform decision-making, and understand important concepts for the interventions in each sector.

In addition, we observed some unexpected results from the CD interventions, including medium-

term changes in staff’s ability to use appropriate language and their perceived safety as a result of

the technology. This section details these primary findings on how respondents perceived the

positive and negative impacts of the CD interventions.

• Have the capacity building strategies so far contributed or are likely to contribute to

medium- to long-term social, economic, and technical changes for individuals,

communities, and institutions?

Improved Capacity to Use Technology

All levels of staff who participated in CD interventions in the health, WASH, and social protection

sectors welcomed the use of technological supports to enter data. Analysis indicates that the CD

interventions were perceived to be successful in increasing staff capacity to use technological tools

(i.e., mobile technology, DHIS2 tool for HMIS and RTM, the M2W tool for RTM, and electronic

forms), thus contributing to medium-term technical changes for individuals and within the

facilities where they work. Staff at the facility and community levels said the use of electronic data

entry prevented the loss of data and made data entry more accurate and efficient. Timely

submission of data also enabled district and province staff at to accurately monitor facilities. For

example, staff at health facilities who received HMISS training—including the use of DHIS2

tools—and community champions who received training on M2W/RTM of CLTS, reported that

electronic data entry allowed them to double-check for data quality, therefore increasing data

accuracy. The use of an electronic system also created an environment in which facilities were

able to self-assess performance and make data-based decisions on key areas for improvement.

Further, enumerators who received the CD intervention on SCT beneficiary selection reported that

the use of the updated electronic Form 2 and m-tech tool to assess residents’ eligibility for SCT

also increased accuracy and transparency of beneficiary selection.

Improved Ability to Understand Data for Informed Decision-Making

One key technical change was in the increased capacity of facility-level staff to manage and

analyse data and capture how data analysis can improve services. Staff at the health facilities who

received the HMISS CD intervention said having access to accurate data improved data

aggregation at the facility level, allowing them to manage data and assess their own performance.

Depending on performance levels, community-level facilities and district-level health centres

developed action plans termed as ‘Situation Analysis’ to improve their health service delivery.

Further, a Provincial Health Information officer said the increase in technical capacity enabled

ground-level staff to better understand how data can be used to inform decisions: “The

decentralisation of the data management from the DHO to the facilities, just submitting a report


33

was their duty but now they have to appreciate; if I enter this figure what does it mean? Having

moved from data entry clerks to data analysers.”

The increased ability of community champions to enter real-time data through the m-tech tool

reduced the lag time of when districts and provinces can monitor data by at least 2–3 months.

Therefore, with the presence of real-time data entry, a provincial-level social engineer from the

MoLG reported that staff at provinces were able to monitor WASH-related data in real time and

thus to provide facilities with immediate support if the data indicated that facilities were not yet

conducive to the overall goal of maintaining high sanitation and hygiene standards in communities

and to reach the target of achieving ODF status by 2030. In addition, the technical capacity that

Chiefs have been empowered with from the CD trainings, such as the use of tablets to monitor

their chiefdoms’ ODF status, helped communities to respond to the triggering efforts because there

was community buy-in.

Increased Community Understanding

Some SCT enumerators who received CD on beneficiary selection said the information on the

different categories of life-course vulnerabilities enhanced their understanding of vulnerabilities.

Both SCT enumerators and CWAC members said the CD trainings improved their ability to

appropriately screen beneficiaries and increased their knowledge on how to identify people who

are disabled, as well as how to assess a resident’s social vulnerability status. One District Welfare

Assistance Committee (DWAC) secretary said, “We used to not know how to categorize … the

difference between disabled and non-disabled persons, so this training assisted us to know who is

a disabled person.”

Further, the evaluation of the M2W CD training for RTM and the CLTS triggering intervention

indicated that increasing community members’ awareness on CLTS concepts will likely contribute

to long-term social changes with respect to WASH practices at the community and individual

levels. The CD on CLTS triggering elicited positive responses from community members, who

said the community sensitizations helped them understand the benefits of building toilets and using

new and existing toilets. For example, one resident said receiving information from community

champions helped them understand the social, health, and individual benefits of using the toilet

and contributing to the nation’s aim to achieve ODF status. The resident said,

“I believe that from the time we were born, we didn’t know about the importance of having

a toilet. Until last year when I came to know about sensitization meetings, it was very

difficult for me to comply because from childhood I knew of using the bush. Even if we

were asked to build toilets, some of us just built simple structures to avoid being taken to

court; but later on, we all accepted and built good toilets which was so wonderful for us

who herd cattle because we could no longer see feces in the bush, it is now clean.”

A community champion also described the growth of sanitation and hygiene-related social change

in the communities: “In the beginning only a few people had toilets but now almost everyone has.

Before the training, those who had toilets they would still go in the bush to defecate, but after they

received knowledge, they are able to use their toilets.”


34

Perceived Improvements in Health Status

Our analysis indicates that the HMISS CD intervention is perceived to have improved maternal,

new-born, child and adolescent survival, and health and nutrition outcomes as a result of the

development of the Quality Improvement (QI) projects. Staff at provincial and district levels used

data to inform areas to improve, assessed the baseline data and developed appropriate interventions

to achieve health outcomes. For example, a provincial health information officer provided an

example of a QI project in Luanshya town that focused on registering pregnant women within the

first trimester of their pregnancy and facilitated a rise in registration from 13% to 34% according

to monitoring data.

• What positive/negative, intended or unintended outcomes have the programmes strategies

for systems strengthening contributed to?

Perceived Changes in Interactions With Beneficiaries

The CD interventions enhanced some respondents’ capacity to use appropriate language and a

respectful approach when speaking to a vulnerable individual and ensuring the security of staff

under the SCT programme. For example, an officer under the child protection unit indicated that

the knowledge and skills disseminated as part of the Basic Qualification Child Care training helped

him to assess a child protection case suitably. The respondent stated:

“Because [recipients] have undergone a lot of training, there will be that tenderness and

care that this is just a child who needs care and I will bring in the NGO language but for

those guys [who have not received the training] they will just say that these children are

troublesome and they are criminals. We would try to change the language and try not to

criminalize the child, so we see that there is a difference when it comes to this approach.

People who don’t have the training, they will be very rough but me—I will say that there

are some factors that led this child to be on the street. I will try to ask some questions in a

tender way and get that information which will help me solve the case that the child is

facing or to know which institution I am going to refer the child to”.

Similarly, a few SCT enumerators said the information they received on life-course vulnerabilities

equipped them with the confidence and knowledge to politely approach and explain to a

community member why he or she is not eligible for the SCT. Community Champions trained on

CLTS triggering also said the trainings helped them gain knowledge on how to appropriately

trigger communities on sanitation and hygiene, improving their ability to communicate the benefits

of toilets.

SCT Staff Felt Safer Using Electronic System

SCT enumerators and CWAC members said the use of the electronic system to select SCT

participants was an important strategy to ensure independence and protect their safety. Prior to the

use of m-tech and electronic Form 2 to select residents, community members were required to

validate if their fellow residents were truly eligible for a SCT. A DWAC secretary said that at times,

CWAC members were threatened by ineligible residents, jeopardizing their safety; but now, with

the use of an electronic system, the community members know that the CWAC member and

enumerators do not determine eligibility. The DWAC Secretary said, “One of the major impacts that


35

I’ve seen is that there are no quarrels, no conflicts. Before [there] were a lot of conflicts regarding

who has been selected and who has not been selected; this time, those [CWAC members] are safe,

they can move freely, they can interact with their beneficiaries freely.”

Table 7 below summarises the main findings related to perceived impact and what information

sources led us to draw these conclusions.

Table 7. Summary of Perceived Impacts Findings

Key Perceived Impact Findings


Evidence Comes From

1. The CD interventions appear to have improved staff’s

technical capacity to enter data electronically, increased

their efficiency in monitoring data quality and

reporting.


community)

• HMISS (provincial, district)


• TaRL (central)

2. The use of improved technical tools for data entry,

management, and analysis increased staff’s ability to

understand the data and make informed decisions on

performance, service delivery, and outreach.


community)

• HMISS (central, provincial, district)


3. The knowledge disseminated through some of the CD

trainings increased training recipients’ awareness of

certain issues. For example, SCT enumerators reported

improved knowledge on life-course vulnerabilities and

disabilities.



• ICM (central)

4. The HMISS CD intervention equipped staff at health

facilities to conduct accurate assessment of service

delivery and health outcomes, and thus enabled them to

develop quality improvement projects with the aim of

improving community health status.

• HMISS (central, provincial, district)

5. The CD trainings were perceived to improve staff’s

skills in communication and engagement with

vulnerable individuals. Further, the increased capacity of

SCT enumerators to use an electronic system to screen

beneficiaries enabled them to feel protected and safe when

interacting with beneficiaries.


• SCT (central, provincial, district,

community)

• ICM (central)


36

5.5 Equity

• Did the programme interventions reach the worst-off individuals or communities as they

did for the other groups?

• What key barriers (political, economic, or social) hindered the affected populations,

communities, and institutions?

Programmes and CD Support Equitably Distributed; Concerted Efforts to Achieve

Gender Parity

Although we did not collect or analyse quantitative data on programme participation as part of this

study, we do see qualitative evidence that UNICEF-supported CD interventions consistently

sought to include women and men equally. Further, the ultimate beneficiaries (both individuals

and communities) of the five CD interventions are indeed among the worst off and most vulnerable

in Zambia. That said, we found evidence that certain vulnerable groups—such as the elderly in the

case of the CLTS—may have faced added difficulty adopting the practices suggested during

trainings.

Across all five CD interventions, stakeholders made concerted efforts to provide men and women

with equal training opportunities. According to one child protection officer involved in

TRANSFORM trainings, “when it comes to workshops, they would not approve it if it’s not

[gender] balanced.” Similarly, a DWAC secretary reported that they made conscious efforts to

ensure gender balance during CWAC recruitment and training, stating,

“In areas where…we discovered that there was that problem [of gender imbalance] we

guided them to say you need women, this committee is male dominated [and] we need

women to be part of this. Where we see that the chair is male the vice becomes male we

advise to say no, I think one of the two should be the chair either male or female then the

one who comes later, there was that kind of gender mixing.”

Despite efforts to achieve gender balance, two CD interventions (M2W/CLTS and TaRL) struggled

to achieve gender parity. While respondents agreed that both men and women serve as community

champions for M2W/CLTS, a few noted that community champions were mostly men. Respondents

also mentioned that more men attended community meetings than women and that men were more

likely to ask questions during these meetings. MoGE made efforts to ensure equal representation of

men and women during TaRL trainings, but according to one respondent it was difficult to achieve

gender balance because most teachers of younger primary grades are women.

Some Barriers Exist to Participation and Acceptance of CD Interventions

Respondents across the five programmes noted few barriers to participating in CD activities apart

from the inevitable resource constraints related to transportation, mobile network access, and

limited time available for training. One CD activity, however, faced a particular challenge in

having to overcome deeply entrenched social norms: Numerous recipients of the CLTS trainings

noted that established social norms around urinating and defecating in the bush initially hindered

acceptance of CLTS messaging and adoption of CLTS practices, most notably toilet construction.

To this end, one focus group participant said, “Ah, this was new to us and [we] didn’t want to

accept it. We didn’t even want to see or have anything with anyone who had to do with toilets”;


37

while another recollected, “People would stand against [the toilet] saying that ‘from the time we

were born up to this age we never used toilets, why should this be? And what will be the use of the

bush?’” Despite the initial resistance, respondents largely agreed that over time and after “serious

and continuous sensitizations” they began to use toilets.

Qualitative data suggest a few instances where vulnerabilities such as age or limited resources

inhibited uptake following CD interventions. For example, respondents reported that elderly

persons had difficulty building toilets following CLTS sensitisation. Numerous respondents also

mentioned that community champions under CLTS lacked resources to continue visiting villages

and uploading information without financial support. To this end, one community champion stated,

“…despite it being voluntary, it is becoming difficult for us to continue with work especially with

the drought we experienced this year, we would rather go for what will earn us a living with our

families that that something we are not paid for.”

Table 8 below summarises the main research findings related to equity.

Table 8. Summary of Equity Findings

Key Findings on Equity


Evidence Comes From

1. All five programmes included in our study and the CD

interventions within them appear to be equitably

implemented, and there are concerted efforts to achieve

gender parity. However. gender equity is not always

realised.

• CLTS (district, community)

• TaRL (central)

2. Some barriers exist to participation and uptake of CD

activities, particularly for vulnerable populations such as

the elderly or those with limited resources.

• CLTS (community)


38

5.6 Sustainability

The primary challenge related to sustaining CD efforts is financial, as there is often limited funding

available to sustain interventions beyond the initial period of performance. Indeed, we heard from

several examples. In addition, personnel turnover and insufficient plans to train new staff also

threaten the sustainability of capacity building activities. This section details the results under

sustainability for the five CD interventions.

• How strong is the level of ownership of the results by the relevant government entities and

other stakeholders?

Staff Motivation Facilitates Continued Implementation of CD Interventions

A common facilitating factor across the three CD interventions in health, social protection, and

WASH sectors was staff motivation, especially at the community level, to apply the knowledge

they gained from the CD trainings. Local-level staff continued implementing lessons from CD

independently, even past the period of implementation in some cases. For example, community

champions and CHWs who triggered communities and entered sanitation and health data under

the M2W/RTM and CLTS and HMISS continued to reach out to communities and capture essential

data despite being unpaid. They were primarily motivated by knowing that their work informed

national ministries.

For the three CD interventions for which we interviewed community implementers, local-level

staff indicated a strong feeling of continuous ownership of programme results. For example, SCT

enumerators said they continued to engage with communities on questions they had past the period

of implementation. One enumerator said that despite no longer having access to mobile phones

after the end of the programme, “We still interact, meet, and then go ahead with advising people.”

Ownership Through Mentors is Key to Sustainability in Current Model

Government staff at various levels recognized the need to continuously equip mentors and

implementers with ongoing trainings that reinforce lessons from CD interventions. A national-

level ministry official who works on TaRL described continuous mentor support as a key element

for sustainability and scaling, “Mentors need to be well equipped. They need to understand

programme and provide support. Even district resource coordinators need to be on top of their

game and be involved.”

Holding refresher courses after the implementation of the initial CD trainings was one way that

staff at provincial levels looked to ensure sustainability in the application of the knowledge

disseminated through the CD trainings. Refresher and online trainings were offered as part of the

CD implemented for the scale up of TaRL, the M2W/RTM for CLTS, and the use of DHIS2 for

HMIS. According to an officer from the MoGE, the ministry already had a donor to provide

refresher trainings on TaRL approach. Further, a one-day refresher training on CLTS triggering

for community champions is scheduled for October 2019.


39

• What is the level of capacity and commitment from the Government and other stakeholders

to ensure sustainability of the results achieved?

Unresolved Logistical Challenges Stall Implementation

As discussed in earlier sections of this document, lack of recourse for community-level implementers

relative to logistical challenges (such as broken or lost data collection phones and scarce funding for

transport) make the interventions difficult to sustain. For example, a community-level respondent

who participated in the WASH CD said that some of the phones used for data collection had broken

and had not been replaced, making it impossible to continue with mobile data collection activities.

A chief in a community that received WASH CD also said of the mobile phones used for data

collection, “Are we sure those gadgets which people have there are still there, or even completely

damaged or lost? We can’t account for those tools maybe anymore because there isn’t any system

to follow up on the same.” Aside from challenges with mobile technology, another community-level

WASH responded added, “We cannot manage to issue forms because they require funds.”

Delay in Approvals for Financing From Higher Levels Hinders Implementation

A province-level WASH employee described a seemingly common challenge confronting local-

level offices; that is, even if employees incorporate plans to continue activities that resulted from

the capacity development, they require higher level approval and distribution of funds to support

programme implementation. The employee said, “Yeah, plans are there, but [we do not know]

from headquarters … what amount of money they allocated [for the DHIS2].” District- and

community-level respondents also indicated, in the case of WASH, that funding discontinued after

Akros ended its support.

• What could be done to strengthen sustainability?

Lack of Plans for Integration and Co-Financing

Despite broad support among local implementers for sustaining the programmes, the government lacks

concrete plans for continued financing. Respondents were open about the challenge of maintaining

funding for programming with government funds alone. One district WASH coordinator said, “That

one is a serious question … We have the right personnel, but financially we are not ready … so much

that it has seemed like a donor-driven programme. The moment the donors pulled out, if you went

[around] in the district, the programme is almost in its dying phase.” A DWAC coordinator from the

SCT pointed to the challenge of sustaining activities that resulted from the CD interventions without

continuing funding allocated to maintaining capacities: “There was that funding but in the recent past

there has been nothing—unless there is something that we haven’t been notified as the DWAC through

this office.”

To strengthen the likelihood of sustainability, ministry-level stakeholders of some interventions

took it upon themselves to seek out additional funding partnerships. For example, a national

ministry official said that TaRL had engaged USAID in a partnership to continue supporting

schools with funding where needed. The representative also said the involved ministries are

actively looking at other donors to help continue supporting the programme activities. Similarly,

provincial officers under the WASH sector are also working to integrate ground-level activities


40

such as community triggering on CLTS and data capturing with CHWs who work under the HMIS

intervention to save costs but still continue to reach out to communities on CLTS concepts.

Similarly, a UNICEF officer for WASH indicated that another possibility for integration would be

to train EHTS and CHWs who are part of the MoH on the M2W and RTM of sanitation and

hygiene interventions.

New and Existing Staff Needed Continuous Trainings

In addition to the focus on equipping mentors with manuals and periodic re-training, the CD

interventions should incorporate a plan for training new staff to ensure knowledge maintenance in

each location. Staff turnover was a common challenge across the HMISS and M2W/RTM CD

training which contributed to gaps in achieving outcomes. Attrition rate of staff who received CD

trainings was high as they were transfers between facilities, districts and provinces due to rotations

within the MoH and WASH sector. A UNICEF HMISS officer said staff turnover was difficult to

control.

Reliance on Community Volunteers May Compromise Sustainability

The results achieved rely heavily on volunteers. Given that the volunteers are unpaid, however,

they may choose to drop their work at any time. Some volunteers expressly stated that their work

after receiving capacity development came second to any paid work they were able to obtain. One

community champion under WASH said, “Despite it being voluntary, it is becoming difficult for

us to continue with work, especially with the drought we experienced this year. We would rather

go for what will earn us a living with our families than [volunteer for] something we are not paid

for.”

Though volunteers learn the cost and other benefits of building latrines, a DWASH coordinator

suggested the model is not sustainable because GRZ would not be able to sustain allowances for

volunteers without the budget from UNICEF. Multiple respondents also suggested that volunteers

be paid.

Table 9 summarises the main findings related to sustainability.

Table 9. Summary of Sustainability Findings

Key Findings on Sustainability


Evidence Comes From

3. Building ownership through mentorship and continuous

training may contribute to the longer-term sustainability

of CD interventions.

• HMISS (central)


• TaRL (central)

4. Unresolved logistical challenges that impede

implementation make sustaining regular activities

difficult.

• HMISS (central)

• SCT (central, community)

• CLTS/M2W (community)

• TaRL (central)


41

Key Findings on Sustainability


Evidence Comes From

5. Concrete plans for integration and co-financing of CD

activities with the GRZ could increase the likelihood of

sustainability.


• SCT (central)

• CLTS/M2W (central, district,

community)

• TaRL (central)

• ICM (central)

6. Overreliance on community volunteers may jeopardise

the sustainability of CD interventions. • HMISS (community)

• CLTS/M2W (community)


42

6.0 Lessons Learned

This section presents some of the primary challenges and opportunities identified across the five

CD interventions and brings together common themes that emerged through our data analysis

according to the OECD-DAC criteria. We organise this section according to challenges and

opportunities; the following section provides recommendations that stem from these challenges

and opportunities.

6.1 Challenges

The research team noted several common challenges relative to implementing and sustaining CD

interventions, including the fact that these activities are driven by the central level, some rely

heavily on unpaid volunteers, and some use technology that is not sustainable over the long term.

Finally, some CD activities have struggled to achieve equitable participation in terms of gender

and those with certain vulnerabilities such as the elderly and the disabled.

Starting from the initial needs assessments, which do not consistently collect information about

community-level needs, CD interventions currently appear to be designed and implemented in a

top-down manner. As a result, there is sometimes a mismatch between the training approach and

what is most relevant and needed at the community level. For example, an assessment of

community-level needs could have ensured that training content and delivery under the CLTS

programme was more readily accepted by community members. Several key stakeholders and

community champions who provided the CLTS triggering activities and monitored subsequent

progress indicated that it was difficult for them to initiate a “shaming” component of triggering

because doing so required the trainers to show contamination of food with faeces—when

defecation was a taboo topic in the community.

Several CD interventions (most notably those within the HMISS and CLTS/M2W programmes)

rely heavily on unpaid volunteers to continue to implement and monitor the activities upon which

they are trained. Respondents in our study voiced concern about the sustainability of the volunteer

model given that the socioeconomic circumstances of volunteers are such that they must prioritise

paid work over volunteer duties when employment opportunities arise. Our analysis suggests that

further investigation into volunteer motivation and the long-term viability of volunteer-supported

interventions is needed.

Technology represents both an opportunity to increase the efficiency and accuracy of data

collection and a challenge in terms of effectiveness and sustainability. The CD interventions

included in this study that aimed to increase individual and organisational technological capacity

boasted impressive perceived impacts but also struggled to maintain technological equipment

(such as laptops) and troubleshoot technological issues over time. These challenges, in turn, posed

a risk to the long-term sustainability of CD efforts involving technology. Specifically, for CD

interventions that included scale-up through the enhanced use of technological tools such as

DHIS2, MIS, Form 2, and H1A4B forms, and faulty or unreliable ICT equipment such as desktops,

laptops, and network issues presented recurring obstacles to continuous data entry and

management.

Finally, while the five programmes included in our study each made considerable efforts to include

comparable numbers of men and women in CD activities, these efforts were more successful in


43

some cases than in others. For example, respondents reported that most community champions

(for CLTS) were male. In addition, some vulnerable populations experienced difficulty applying

what they learned through trainings. For example, elderly participants in CLTS trainings faced

challenges constructing toilets following the training, and low-resource communities lacked the

necessary supplies to construct toilets.

6.2 Opportunities

Across the five CD interventions, we noted several opportunities that could be leveraged in future

CD programming. For example, innovative approaches to saving money on CD activities could be

scaled up or applied going forward. There also appear to be opportunities to enhance government

participation—including co-financing—of future CD activities to increase sustainability. Finally,

there is enthusiasm at all levels (from the national level down to communities) relative to using

data for decision-making that present real opportunities to generate buy-in and support for CD

interventions related to data collection, management, and reporting.

This report presents several examples of innovative ways to save money on CD programming,

including capitalising on technology to reduce the cost of post-training support and

troubleshooting. For example, an informant from the MoH noted the existence of a WhatsApp

group for DHIOs that served as a cost-effective model for troubleshooting common data entry and

management problems. Future CD programming—particularly CD activities involving

technology—could potentially consider WhatsApp groups as an alternative to in-person support

for troubleshooting. Other opportunities to maximise cost effectiveness include developing

standardised modules. In the case of ICM strengthening efforts, for example, UNICEF worked

with MCDSS to develop standardised training materials (including a module) on case management

for children rather than convening expensive workshops to train staff on the specifics of case

management for children.

There are indications that GRZ ministries intend to take greater ownership of CD programming

moving forward, which could lead to greater efficiency and better prospects for the long-term

sustainability of CD interventions. For example, an informant from MoGE spoke of mainstreaming

TaRL in the MoGE curriculum; and the MoH has recruited staff at the national-level to respond to

challenges faced by provincial-, district-, and facility-level staff regarding use of the DHIS2 tool

under the HMISS CD intervention. The next step could be for donors to work closely with GRZ

ministries to implement needs assessments, design CD programmes, and agree to co-finance or

embed initiatives into existing government structures.

Finally, we noted evidence of enthusiasm at all levels (but especially at the community level)

relative to collecting and using data for decision making. Community champions and CHWs

reported that trainings on RTM/M2W, CLTS triggering, and the HMISS CD enhanced their

motivation to ensure data quality and report data on time because they knew the data were being

used to inform decisions at the ministry level. There may be opportunities to leverage this

excitement to generate buy-in for current and future CD programming.

7.0 Conclusions and Recommendations

In this section we summarise key findings from the six thematic areas (relevance, effectiveness,

efficiency, perceived impacts, equity, and sustainability) and present recommendations based on


44

these findings. The AIR research team presented evaluation findings and corresponding

recommendations to key stakeholders from UNICEF Zambia on October 2, 2019. During this

presentation, individuals from UNICEF offered feedback on prioritizing and further refining the

recommendations to make them more actionable. In response to this feedback, the research team

revised the recommendations and developed Table 10, below, which prioritizes the

recommendations and designates who is responsible for implementing each recommendation.

7.1 Relevance

The CD interventions included in this evaluation resulted from needs assessments that helped

ensure the relevance of activities in each sector. UNICEF and the GRZ worked together in an

ongoing manner to ensure that the interventions aligned with their respective priorities and, to the

extent possible, relied on existing structures and programmes rather than creating new ones for the

purposes of a single intervention. Although the needs assessments conducted prior to each

intervention helped ensure that some priorities were addressed—primarily at the national level—

incorporating needs assessments more focused on the needs of individuals districts and

communities may have increased the programmes’ relevance for marginalized groups.

• Recommendation: UNICEF and partners should consider incorporating ongoing needs

assessments with inputs from programme participants at the district and community levels.

Regular interaction with programme participants will help ensure that activities continue to be

relevant, increasing their likelihood of sustainability.

7.2 Effectiveness and Perceived Impacts

Our evaluation revealed that the CD interventions under the health, WASH, social protection, and

education sectors were largely perceived to have achieved their intended objectives. Specifically,

analysis showed that the CD contributed to improved access to accurate data, improved data

management, and increased motivation among community volunteers. Facilitating factors

included government and community buy-in, although inadequate financial support for ICT and

transportation hindered some aspects of implementation. CD interventions could also improve

effectiveness for marginalized groups, especially women and people with disabilities.

• Recommendation: Develop clear backup plans to troubleshoot ICT-related issues both during

and after the initial project period. For example, if people are trained to use phones or tablets

for data collection, incorporate resources and procedures to replace or repair the technology as

needed. This recommendation stems from a process gap we observed in CD related to the

HMISS intervention: EHTs from health facilities stated that they were unable to fix

technological issues with their laptops and were required to send them to the provincial-level

health offices for repair. District-level health staff deemed this an inefficient process and

reported that even after six months, they were still waiting on a status update and were unable

to retrieve their laptops.

• Recommendation: Following trainings on mobile data collection and data management,

establish clear performance benchmarks (for example, a threshold for data accuracy) with

district, provincial, and national staff during and after the initial project period to ensure that

these capacities are sufficiently monitored and sustained.


45

7.3 Efficiency

While the CD activities evaluated under this study were generally delivered efficiently and on

schedule, some common challenges existed regarding the timely provision of necessary inputs. In

addition, the cost of transportation for participation and monitoring CD activities is considerable

(that is, while the ToT model is perceived as cost-effective, it can be deceptively expensive and

time-consuming). We recommend the following to maximise efficiency during future CD

interventions:

• Recommendation: At the outset of all CD interventions, establish a clear process and timeline

for equipment and fund disbursements so that trainings are not delayed or participants

demotivated because of a lack of necessary equipment or funds.

• Recommendation: Carefully assess the quantity and quality of inputs required (such as mobile

phones or laptops) and the associated costs to ensure maximum efficiency in implementing

future CD activities.

• Recommendation: Minimise unnecessary transportation costs by convening workshops and

other CD activities locally (i.e., in zones within districts) to eliminate participant housing costs.

• Recommendation: When possible, explore the feasibility of developing standardised e-

learning modules on important topics that can be recycled in place of convening large groups

for in-person trainings.

• Recommendation: Where possible, use locally available materials (as in TaRL) or create

WhatsApp groups as cost-effective ways to troubleshoot technology issues (as done by

DHIOs).

• Recommendation: Prior to scaling up future CD interventions, conduct cost-effectiveness

studies to calculate the cost of achieving specific outcomes.

7.4 Equity

We found that most CD interventions were delivered in an equitable manner—with special

attention paid to achieving gender parity—and that programmes themselves targeted individuals

and communities that were among the neediest in Zambia. However, some vulnerable populations

had difficulty applying what they learned through trainings. For example, elderly participants in

CLTS trainings faced challenges constructing toilets following the training, and low-resource

communities lacked the necessary supplies to construct toilets.

• Recommendation: During the initial needs assessment, consider the potential barriers to

participation and uptake of CD activities for vulnerable groups such as the elderly. Creating

user personas (Dam & Siang, 2019) could help ensure that all potential barriers are considered

during programme design.

• Recommendation: Encourage local levels to recruit more women for positions that are likely

to interact with female participants.

• Recommendation: Develop strategies and instruments to assess gender equity in CD trainings

and community sensitisations after trainings.


46

7.5 Sustainability

The primary challenge related to sustaining CD efforts was financial, as there was limited initial

planning for how funding would continue to be available for interventions beyond the initial period

of performance. Several examples pointed to the impression at the local level that when an

implementing organization leaves, the programme has ended. Such impressions were exacerbated

by the fact that provisions for broken hardware and necessary materials were not available after

the end of the programme. In addition, personnel turnover and insufficient plans to train new staff

continue to threaten the sustainability of capacity building activities. In response to these

somewhat inevitable sustainability challenges, we recommend the following:

• Recommendation: Incorporate plans for continuing funding from the beginning to increase the

likelihood of sustainability. This should include an initial plan for co-financing programming

with the government as well as strategizing with GRZ to identify sources of future funding

other than donors.

• Recommendation: Develop and incorporate an ongoing protocol for training new staff and

retraining existing staff. Embedding information from CD interventions into existing

onboarding curricula and training for ministry staff could provide opportunities to streamline

the training. Embedding coaching for government to continue interventions could be

accomplished through regular meetings to discuss emerging issues and challenges.

• Recommendation: Integrate CD interventions across sectors and programs. For example, at

the community level, CHWs and Community Champions could be trained together on data

capture and community outreach to reduce cost.

• Recommendation: Develop a short-term approach for compensating volunteers (monetarily or

in-kind, such as free access to goods or services) to enhance recognition and motivation. Over

the long term, the GRZ should develop a plan to employ people to do the work currently done

by volunteers.

Table 10. Recommendation Priority, Timeframe, and Accountability

# Recommendation Priority

Level

Recommended

Timeframe for

Adoption

Person(s)/Organisation(s)

Accountable to

Recommendation

Relevance

1 UNICEF and partners should consider

incorporating ongoing needs assessments

with inputs from programme participants

at the district and community levels.

Regular interaction with programme

participants will help ensure that

activities continue to be relevant,

increasing their likelihood of

sustainability.

High

Immediate • UNICEF staff

supporting CD

intervention

implementation

• Implementing partners

• GRZ ministry staff at

national and district

levels

Effectiveness and Perceived Impacts

2 Develop clear backup plans to

troubleshoot ICT-related issues both

during and after the initial project period.

For example, if people are trained to use

High

Immediate • UNICEF staff

supporting CD

intervention


47


Level

Recommended

Timeframe for

Adoption


Accountable to

Recommendation

phones or tablets for data collection,

incorporate resources and procedures to

replace or repair the technology as

needed.

implementation

(including IT)




levels (including IT

departments)

3 Following trainings on mobile data

collection and data management,

establish clear performance benchmarks

(for example, a threshold for data

accuracy) with district, provincial, and

national staff during and after the initial

project period to ensure that these

capacities are sufficiently monitored and

maintained.

Medium Medium term • UNICEF staff

responsible for M&E

related to CD

interventions

• Implementing partner

M&E staff

• GRZ ministry staff

responsible for M&E at

national, district, and

community levels

Efficiency

4 At the outset of all CD interventions,

establish a clear process and timeline for

equipment and fund disbursements so that

trainings are not delayed or participants

demotivated because of a lack of

necessary equipment or funds.

High Immediate • UNICEF staff

supporting CD

intervention

implementation

(including finance)



the national level

responsible for

approvals and fund

disbursement

5 Carefully assess the quantity and quality

of inputs required (such as mobile phones

or laptops) and the associated costs to

ensure maximum efficiency in

implementing future CD activities.

Medium Immediate • UNICEF staff

supporting CD

intervention

implementation

(including procurement)



the national level

6 Minimise unnecessary transportation

costs by convening workshops and other

CD activities locally (i.e., in zones within

districts) to eliminate participant housing

costs.

Medium Medium term • UNICEF staff

supporting CD

intervention

implementation


the national level

7 When possible, explore the feasibility of

developing standardised e-learning

modules on important topics that can be

recycled in place of convening large

groups for in-person trainings.

Medium Long term • UNICEF staff

supporting CD

intervention

implementation


the national level


48


Level

Recommended

Timeframe for

Adoption


Accountable to

Recommendation

8 Where possible, use locally available

materials (as in TaRL) or create

WhatsApp groups as cost-effective ways

to troubleshoot technology issues (as

done by DHIOs).


supporting CD

intervention

implementation



the national and district

levels

9 Prior to scaling up future CD

interventions, conduct cost-effectiveness

studies to calculate the cost of achieving

specific outcomes.

Medium Long term • UNICEF staff designing

and commissioning CD

interventions

Equity

10 During the initial needs assessment,

consider potential barriers to participation

and uptake of CD activities for vulnerable

groups such as the elderly. Creating user

personas (Dam & Siang, 2019) could help

ensure that all potential barriers are

considered during programme design.

High Long term • UNICEF staff designing

and commissioning CD

interventions


(including programme

and M&E staff)


the national and district

levels

11 Encourage local levels to recruit more

women for positions that are likely to

interact with female participants.

High

Immediate • UNICEF staff managing

and selecting partners

for CD activities



levels


(including training staff

at district and local

levels)

12 Develop strategies and instruments to

assess gender equity in CD training and

community sensitisation after training.

High Medium • UNICEF staff

supporting CD

interventions; M&E

staff


national level


(including M&E staff)

Sustainability

13 Incorporate plans for continuing funding

from the beginning to increase the

likelihood of sustainability. This should

include an initial plan for co-financing

programming with the government as

well as strategizing future fund

generation outside of relying on donors

with GRZ.

High Immediate • UNICEF staff and other

donors collaborating

with GRZ (funding

should be contingent

upon having these long-

term co-financing plans)


49


Level

Recommended

Timeframe for

Adoption


Accountable to

Recommendation



level

14 Develop and incorporate an ongoing

protocol for training new staff and

retraining existing staff. Embedding

information from CD interventions into

existing onboarding curricula and

training for ministry staff could provide

opportunities to streamline the training.

Embedding coaching for government to

continue interventions could be

accomplished through regular meetings

to discuss emerging issues and

challenges.

High Immediate • GRZ staff at the national

and district level

15 Integrate CD interventions across sectors

and programs. For example, at the

community level, CHWs and Community

Champions could be trained together on

data capture and community outreach to

reduce costs.


supporting CD

interventions


national and provincial

levels


16 Develop a short-term approach for

compensating volunteers (monetarily or

in-kind, such as free access to goods or

services) to enhance recognition and

motivation. Over the long term, the GRZ

should develop a plan to employ people

to do the work currently done by

volunteers.

High Immediate and

long term • GRZ ministry staff at

national level

• UNICEF staff support

CD programming that

relies on volunteers


50

References

Akerberg, A. (2001). Human rights and persons with disabilities. Stockholm, SE: SHIA Human

Rights & Disability Network.

AKROS Inc. (2016). HMIS assessment 2016: Copperbelt and Lusaka provinces. Lusaka, ZM:

Author.

AKROS Inc. (2018). Health Management Information System Strengthening project: MDGi.

Lusaka, ZM: Author.

Bernard, R. (2011). Research methods in anthropology: Qualitative and quantitative approaches.

Oxford, UK: AltaMira Press

Boyle, R., Lemaire, D., & Rist, R. C. (1999). Introduction: Building evaluation capacity. In R. Boyle

& D. Lemaire (Eds.), Building effective evaluation capacity: Lessons from practice (pp. 1–

19). New Brunswick, NJ: Transaction.

Brown, L., LaFond, A., & Macintyre, K. (2001). Measuring capacity building. Measure

evaluation. Chapel Hill, NC: University of North Carolina at Chapel Hill.

Bryman, A. (2012) Social research methods. Oxford, UK: Oxford University Press.

Chaudhry, V., & Owen, D. (2005). Examining inclusion: Disability and community driven

development. Social Development Notes, 100, 1–4.

Dam, R., & Siang, T. (2019). Personas—A simple introduction [Webpage]. Retrieved from

https://www.interaction-design.org/literature/article/personas-why-and-how-you-should-

use-them

Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the

influence of implementation on program outcomes and the factors affecting

implementation. American Journal of Community Psychology, 41, 327–350.

Fenton, K. A., Johnson, A. M., McManus, S., & Erens, B. (2001) Measuring sexual behavior:

methodological challenges in survey research. Sexually Transmitted Infections, 77, 84–92.

Freeland, N. (2017). Mid-term review of the government of the Republic of Zambia—United

Nations Joint Programme on Social Protection. Lusaka, ZM: UNICEF Zambia.

Freeland, N. (2018). Report on the process evaluation of the Social Cash Transfer (SCT)

Programme. Lusaka, ZM: UNICEF Zambia.

Goodman, R. M., Speers, M. A., McLeroy, K., Fawcett, S., Kegler, M., Parker, E.,

…Wallerstein,N. (1998). Identifying and Defining the Dimensions of Community

Capacity to Provide a Basis for Measurement. Health Education and Behavior, 25(3), 258–

278

https://www.interaction-design.org/literature/article/personas-why-and-how-you-should-use-them

https://www.interaction-design.org/literature/article/personas-why-and-how-you-should-use-them


51

Innovations for Poverty Action. (2016). Catch Up pilot using Teaching at the Right Level

methodology: Process monitoring report. Retrieved from

https://www.unicef.org/zambia/media/591/file

Kalyanpur, M. (1996). The influence of western special education on community based services

in India. Disability and Society, 11, 249–270.

Kisanji, J. (1998). The march towards inclusive education in non-Western countries: Retracing the

steps. International Journal of Inclusive Education, 2, 55–72.

LaFond, A., & Brown, L. (2003). A guide to monitoring and evaluation of capacity-building

interventions in the health sector in developing countries (Measure Evaluation Manual

Series, No. 7). Chapel Hill, NC: Carolina Population Centre, University of North Carolina

at Chapel Hill.

Miller, E., & Daly, E. (2013) Understanding and measuring outcomes: The role of qualitative

data. Glasgow, UK: Institute for Research and Innovation in Social Services.

Milstein, B., & Cotton, D. (2000). Defining concepts for the presidential strand on building

evaluation capacity. Working paper circulated in advance of the November 2000 meeting

of the American Evaluation Association.

Mittler, P. (2005). The global context of inclusive education. In D. Mitchell (Ed.), Contextualizing

inclusive education (pp. 22–36). London, UK: Routledge.

National Planning Commission. (2013). National development plan vision 2030. Retrieved from

http://policyresearch.limpopo.gov.za/bitstream/handle/123456789/941/NDP%20Vision%

202030.pdf?s

Organisation for Economic Co-operation and Development. (1991). Principles for evaluation of

development assistance: Development Assistance Committee. Paris, FR: Author.

Owen, J. M. (2003). Evaluation culture: A definition and analysis of its development within

organizations. Evaluation Journal of Australasia, 3, 43–47.

Pearson, J. (2010, June). A learning package for capacity development: “Learning to change

capacity development practice” [Working paper]. Retrieved from

http://lencd.org/files/event/2011/Partners_Meeting_2011,_Kigali/lencdscopingstudylearni

ngpackage100908s-1.pdf

Peters, S. (2007). A historical analysis of international inclusive education policy and individuals

with disabilities. Journal of Disability Policy Studies, 18(2), 98–108.

Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building.

American Journal of Evaluation, 29, 443–459.

https://www.unicef.org/zambia/media/591/file

http://policyresearch.limpopo.gov.za/bitstream/handle/123456789/941/NDP%20Vision%202030.pdf?s

http://policyresearch.limpopo.gov.za/bitstream/handle/123456789/941/NDP%20Vision%202030.pdf?s

http://lencd.org/files/event/2011/Partners_Meeting_2011,_Kigali/lencdscopingstudylearningpackage100908s-1.pdf

http://lencd.org/files/event/2011/Partners_Meeting_2011,_Kigali/lencdscopingstudylearningpackage100908s-1.pdf


52

Rapkin, B. D., & Trickett, E. J. (2005). Comprehensive dynamic trial designs for behavioral

prevention research with communities: Overcoming inadequacies of the randomized

controlled trial paradigm. In E. Trickett & W. Pequenaut (Eds.), Increasing the

community impact of HIV prevention interventions (pp. 249–277). New York, NY:

Oxford University Press.

SOS Children’s Villages International. (2013). A snapshot of alternative care arrangements in

Zambia. Innsbruck, AT: Author. Retrieved from

https://www.sos-childrensvillages.org/getmedia/7ba549de-ad9e-4bba-a7c7-

ea700476c8fe/ZAMBIA-FINAL-to-upload.pdf

Teaching at the Right Level. (2019). TaRL case study: Zambia. Retrieved from

https://www.teachingattherightlevel.org/tarl-in-action/zambia-case-study/

UNICEF. (2010). The approach of UNICEF to capacity development. Retrieved from

https://www.unicef.org/about/execboard/files/2010-

CRP20_Capacity_Development_oral_report.pdf

UNICEF. (2017). Integrating case management for vulnerable children: A process guide for

assessing and developing an integrated case management system in Eastern and

Southern Africa. Retrieved from http://maestral.org/wp-

content/uploads/2017/08/Integrating-Case-Managment-for-VC.pdf

UNICEF (2018). Consultancy to conduct an evaluation on capacity development initiatives

across the programme outcomes: Terms of reference [ToR]. Lusaka, ZM: Author.

UNICEF Zambia. (2017a) Impact evaluation of the sanitation and hygiene program in

Zambia. Lusaka, ZM: Author.

UNICEF Zambia. (2017b) UNICEF annual report 2017: Zambia. Lusaka, ZM: Author

UNICEF Zambia. (2018a). Social Cash Transfer Programme: Training guide. Selection of

beneficiary households. Lusaka, ZM: Author.

UNICEF Zambia. (2018b). Request for proposals: Terms of reference. Lusaka, ZM: Author.

UNICEF Zambia. (2019). Child protection. Lusaka, ZM: Author. Retrieved from

https://www.unicef.org/zambia/child-protection

Zaltsman, A. (2006, May). Evaluation capacity development: Experience with institutionalizing

monitoring and evaluation systems in five Latin American countries: Argentina, Chile,

Colombia, Costa Rica and Uruguay [ECD Working Paper Series, No. 16]. Washington,

DC: World Bank. Retrieved from

http://documents.worldbank.org/curated/en/399731468126575141/pdf/382510ENGLISH

01tionalizing01PUBLIC1.pdf

https://www.sos-childrensvillages.org/getmedia/7ba549de-ad9e-4bba-a7c7-ea700476c8fe/ZAMBIA-FINAL-to-upload.pdf

https://www.sos-childrensvillages.org/getmedia/7ba549de-ad9e-4bba-a7c7-ea700476c8fe/ZAMBIA-FINAL-to-upload.pdf

https://www.teachingattherightlevel.org/tarl-in-action/zambia-case-study/

https://www.unicef.org/about/execboard/files/2010-CRP20_Capacity_Development_oral_report.pdf

https://www.unicef.org/about/execboard/files/2010-CRP20_Capacity_Development_oral_report.pdf

http://maestral.org/wp-content/uploads/2017/08/Integrating-Case-Managment-for-VC.pdf

http://maestral.org/wp-content/uploads/2017/08/Integrating-Case-Managment-for-VC.pdf

https://www.unicef.org/zambia/child-protection

http://documents.worldbank.org/curated/en/399731468126575141/pdf/382510ENGLISH01tionalizing01PUBLIC1.pdf

http://documents.worldbank.org/curated/en/399731468126575141/pdf/382510ENGLISH01tionalizing01PUBLIC1.pdf

Annexes

Inception Report for the Evaluation of UNICEF Zambia’s Capacity Development Interventions

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG A-1

Annex A. TOR

TERMS OF REFERENCE

Title: Evaluation of the investment in institutional, community and individual capacity development.

Purpose: To conduct an evaluation on Capacity Development Initiatives across the programme outcomes.

Location: Lusaka with a few travels to some provinces and districts

BACKGROUND

UNICEF Zambia aims to strengthen national systems for the realization and protection of the rights of

children, with attention to the building of sustainable and efficient national and subnational capacities in

terms of technical effectiveness and the achievement of high-impact results, and that also include the

Government’s implementation of its decentralization policy.

UNICEF Zambia’s current country programme contributes to the following strategic results: improved

survival of children under 5, with an emphasis on the neonatal period; reduced stunting; improved access

to water and sanitation; improved education quality and learning outcomes; strengthened child and family

welfare systems to prevent and respond to violence, abuse, neglect and exploitation; and strategic

harnessing of social protection interventions for the most vulnerable children and adolescents and their

multiple and interconnected vulnerabilities that include pregnancy, early marriage, HIV infection and

gender-based violence.

The country office has programmed $120 million for the country programme, against a planned figure of

$225 million,3 including Other Resources-Emergency (ORE). The total includes $25.5 million in regular

resources; and $95 million in other resources-regular (ORR) against $182 million planned.

At Outcome level; UNICEF Zambia is working towards the attainment of the following outcomes;

Outcome 1: By 2021, government capacity on planning, service delivery and monitoring enhanced for

quality equitable and socially accountable health services.

Outcome 2: By 2021, children, adolescents, and pregnant women use quality, accessible, and proven HIV

prevention and treatment interventions.

Outcome 3: By 2021, children and mothers use high-impact nutrition interventions (services and practices)

for reduction of stunting.

Outcome 4: By 2021, Children including adolescents, pregnant women and new-borns, especially those

from rural and peri-urban areas, benefit from systems that deliver improved and equitable access to and use

of safe drinking water, improved sanitation and improved hygiene practices.

Outcome 5: By 2021, boys and girls of school-going age demonstrate improved learning outcomes as a

result of equitable and inclusive access to quality education.

Outcome 6: By 2021, children benefit from a system that ensures integrated, improved and equitable

prevention of and response to violence, abuse, exploitation and neglect.

3 The Country programme ceiling will be revised for the one-year extension. During the MTR, it was proposed to increase the RR

amount by one year while maintaining the OR ceiling at $182,280,000



Outcome 7: By 2021, Vulnerable children and families benefit from and participate in a strengthened

government response to poverty and vulnerability.

In striving to achieve these outcomes and strategic results, UNICEF invests significant amounts of money

in various capacity development initiatives both horizontally (across central government and civil society

structures) and vertically (national and subnational levels of government).

The Country Office ’s approach to capacity development is centred around participatory learning and

training; communication for development (C4D), including promotion of positive social norms and

behaviors; and strengthening national and subnational systems for data collection and use. For water,

sanitation and hygiene (WASH) emphasis has been placed on capacity development for construction,

operations and maintenance, with a view to enhancing sustainability; water quality monitoring; mobile-to-

web monitoring; community-led total sanitation (CLTS); school-led total sanitation and menstrual hygiene

management; and enforcement of public health laws. Health system strengthening is done through the

district health information system platform, which include improving assurance systems to institutionalize

clinical mentorship systems and reviewing bottlenecks in procurement and supply chain management.

Support is provided to reinforce community-based primary health care systems through training of

community-based volunteers (CBVs), and revitalization of neighborhood health committees. Coupled with

the above, UNICEF has supported trainers in sexual and reproductive health (SRH)/HIV peer education

and adolescent health training.

As part of UNICEF’s drive to support a robust child protection system and systematic provision of

alternative care, social welfare officers and staff from child care facilities have been oriented on the new

alternative care regulatory framework. In addition, health workers, volunteers and civil registrars received

training on civil registration as part of UNICEF’s support to the decentralization of civil registration and

vital statistics (CRVS).

In education, school- and community-level participatory learning activities focus on parental engagement

in ECD centres, learner-centred catch-up methodology, use and interpretation of school and community

profiles and school improvement plans.

In line with the 2016-2021 Costed Evaluation Plan (CEP), an outcome evaluation needs to be conducted to

ascertain the degree of effectiveness and efficiency of the Capacity Development Interventions that have been

undertaken by UNICEF in partnership with donors, government, and civil society for the period 2016-2018.

2.0 JUSTIFICATION

With the above support or contribution to structural and systems development across the programme areas,

UNICEF Zambia needs to generate evidence on the effectiveness and sustainability of its medium to long

term capacity enhancement investments at community, district, provincial and national levels. It is against

this background that UNICEF wishes to engage an independent consultant to conduct an evaluation in

unbiased manner.

3.0 OBJECTIVES

The overall purpose of this evaluation is to distil learning and provide recommendations to inform

implementation of the Country programme, particularly on how UNICEF Zambia can improve its capacity

building strategy to better contribute delivering results for Children.

The evaluation should provide an unbiased assessment of the relevance and effectiveness of UNICEF

Zambia’s capacity building strategies as implemented since 2016, determining what has worked and why.

At the same time, the evaluation should be forward-looking, considering the changes in the programming

context and provide recommendations improvement.



The evaluation will provide insights on the programme of cooperation’s strategic position with regards to

capacity development. It will identify the risks and opportunities in the context of capacity development,

and will provide UNICEF with insights into the possible strengthening of the approach to capacity

development at all levels of programmatic engagement.

Specific objectives:

• Highlighting consistent patterns and good practices across capacity building and system

strengthening initiatives/strategies, identify barriers or lessons learnt and study determinants for

success

• Assess relevance, efficiency, effectiveness, equity-focus and sustainability of capacity building

and system strengthening initiatives with regards to achieving country programme results

• Make recommendations to improve future capacity building and system strengthening work

4.0 DESCRIPTION OF THE ASSIGNMENT (SCOPE OF WORK) / SPECIFIC TASKS

To meet the objectives of the outcome evaluation, the consultancy firm will undertake the evaluation guided

by the following evaluation scope and criteria:

4.1 Evaluation Scope:

This evaluation will cover the partners at national and communities (rural and urban) of the 10 provinces

of Zambia in which the systems strengthening and capacity building initiatives have been or are being

implemented either directly or through the Implementing Partners(IPs) since 2016

4.2 Evaluation Criteria and proposed evaluation questions

a) Relevance

To what extent do the intended outcome and the relevant outputs address the national priorities

as indicated in the National Strategies that culminate into the realization of vision 2030.

To what intent are the key strategies in capacity initiatives aligned with UNICEF’s

Global/Regional priorities.

Has UNICEF been able to adapt to the changing contexts- social, political and economical to

address capacity/ systems strengthening needs in the Country?

Has investment in capacity and systems strengthening across the Programmes been relevant to

the needs of Children, Women and the other marginalized groups?

b) Effectiveness

To what extent have outcomes been achieved, are there any additional outcome(s) being

achieved beyond the intended outcome? Did the capacity building/system strengthening

interventions lead to major unexpected results—both positive and negative?

Were the assumptions underlying the programme intervention strategy correct? Which factors,

internal and external to the programme strategies, explain the extent to which results have been

achieved?

What are consistent patterns and good practices across capacity building and system

strengthening initiatives

What were the facilitating and constraining factors? What are the challenges to achieving the

outcomes?



How have the results, both at outcome and output benefited men and women? Have they

benefitted equally

Is the current set of indicators, both outcome and output indicators, effective in informing the

progress made towards the outcomes? If not, what indicators should be used?

c) Efficiency

Were the resources and inputs converted to outputs in a timely and cost-effective manner? Did

ZCO use resources in the most economical manner to achieve expected equity-focused results?

If so how?

Has there been any duplication of efforts among UNICEF’s own interventions and

interventions delivered by other organizations or entities in contributing to the outcomes?

Would it have been possible to achieve the same results at lower costs? If so how? What

alternative models exist to achieve the results at the lower costs?

How well did the programme coordinate with other, similar programme strategies within the

Country Office for synergy and in order to avoid overlaps/duplication?

Were the programme strategies implemented in a timely manner? If not why?

What were the most important cost drivers in the delivery of the programme, were costs

contained without compromising results? If so how?

d) Sustainability

How strong is the level of ownership of the results by the relevant government entities and

other stakeholders?

What is the level of capacity and commitment from the Government and other stakeholders to

ensure sustainability of the results achieved?

What could be done to strengthen sustainability?

e) Impact

Have the capacity building strategies so far contributed or is likely to contribute to medium to long-

term social, economic, technical, changes for individuals, communities, and institutions?

What positive/negative, intended or unintended outcomes have the programmes strategies for

systems strengthening contributed to?

f) Equity

Did the programme interventions reach the worse off individuals or communities like it did for the

other groups?

What key barriers-political, economic, social hindered all the affected populations, communities

and institutions

4.3 Methodology

The evaluation will engage a wide array of stakeholders and beneficiaries, including national and local

government officials, civil society organizations and community members.

An evaluation design with a detailed methodology is expected to be developed in consultation with the

UNICEF. The outcome evaluation is expected to take a “theory of change’’ (TOC) approach to determining



causal links between the programme strategies/interventions UNICEF has supported, and observed progress

in capacity building initiatives at national, provincial and local levels in Zambia.

Evidence obtained and used to assess the results of Capacity Building support should be triangulated from

a variety of sources, including verifiable data on indicator achievement, existing reports, implementing

partner interviews, focus groups, surveys and site visits.

The following steps in data collection are anticipated:

a) Desk Review

A desk review should be carried out of the key strategies and documents underpinning the Capacity

building/systems strengthening work of UNICEF in Zambia in its programme outcomes. This includes

reviewing the pertinent country programme documents, these will include;

• 2016-2021 Country Programme document

• COMPACT

• Programme Outcome RFs/Theory of Change

• Annual Management Plans and Reports

• Programme Monitoring Reports

• 2016, 2017 and 2018 Country Office Annual Reports(COAR).

• Evaluation and Research reports relevant to the assignment

• National Development Plans (Sixth National Development Plans and the Seventh National

Development Plans)

In addition, the evaluators are expected to review pertinent national strategies, management information

systems and other key capacity building products that are pertinent to UNICEF’s capacity building or

systems strengthening initiatives across the programme areas/sectors to be made available by UNICEF

Zambia Country Office.

b) Field Data Collection

Following the desk review, the evaluators will build on the documented evidence through an agreed set of

field and interview methodologies, including: a) Interviews with key partners and stakeholders b) Field

visits to project sites and partner institutions c) Survey questionnaires where appropriate and d)

Participatory observation, focus groups, and rapid appraisal techniques.

4.4 Gender and Human Rights, including child rights

The Consultant’s team will encourage women’s participation during data collection as they are both duty

bearers and rights-holders of the target population. All data collected should be disaggregated by sex, age

and location.

Gender roles and expectations will need to be considered both with respect to the ability of the community

members to participate in the evaluation and group dynamics of any FGD or interviews. The evaluation

should follow the United Nations Evaluation Group norms and standards. These will be shared with the

selected contractor.

The evaluation process will be guided by relevant instruments or policies on human rights, including child

rights and gender equality e.g. International human rights law and human rights principles, including the

Convention on the Rights of the Child, the Convention on the Elimination of All Forms of Discrimination

against Women etc.



4.5 Ethical Consideration Ethical aspects of the evaluation which include, among others, data collection from human subjects and their consent should be covered in detail in the technical proposal. UNICEF has a set of ethical principles, and checklist regarding research and evaluations which must be upheld. The evaluation team will meet human objects particularly children and should take precautions to protect the rights and wellbeing of any children.

4.6 Potential risks and challenges

In order to perform the evaluation, the Consultant and his/her team might face risks and challenges related

with the availability of information, delays in the data collection process due to unexpected events that

affect the fieldwork, lack of collaboration from key stakeholders, methodological challenges, among others.

The team must identify potential risks and challenges in advance and discuss possible strategies to

overcome these situations so that the evaluation is conducted within the planned timeline.

5.0 EXPECTED DELIVERABLES

Phases Task Deliverables

Timeframe

(Tentative)

Inception Phase Finalization of evaluation

design, methodology, and

detailed work plan, Inception

meeting

Inception Report outlining the

overall approach and

methodology, data collection

tools, a detailed implementation

plan and Power Point

Presentation of inception report

15 days

Implementation Phase Data collection- review of

existing literature, collection of

primary data from sources

identified in collaboration with

UNICEF and Government

Counterpart- Ministry of

National Development

Preliminary report 25 days

Report writing phase Documentation of key findings

and validation meeting

20 days

Final Phase Final report writing phase Final report as per agreed

format.

15 days

6.0 EVALUATION REPORT STRUCTURE

Guided by the UNICEF Evaluation Report Standards and the GEROS Quality Assessment System, the

Consultant will prepare an evaluation report that describes the evaluation and puts forward the evaluator's

findings, recommendations and lessons learned. The evaluation report will not be more than 60 pages.

Below is a sample final evaluation report structure, which has the following components:

• Table of Contents

• Acronyms

• Executive Summary

• Background and Programme Description

• Purpose of Evaluation

• Evaluation Objectives and Scope

• Evaluation Methodology

• Findings and analysis



• Conclusions

• Recommendations

• Lessons learned

• Annexes: including the terms of reference, evaluation work-plan and any other relevant documents

7.0 PROJECT MANAGEMENT

The Consultant will be contracted by, and report to UNICEF Zambia, which will assess the deliverables for

payment. Working in collaboration with the Ministry of National Development Planning through the

directorate of Development Cooperation, Monitoring and Evaluation will, UNICEF’s Social Policy and

Research and the Programme Management Excellence(PME) will manage the process with the Monitoring

and Evaluation Officer as a focal point person.

Working in close collaboration with the Chief SPR and PME, the Monitoring Officer will provide

supportive supervision of the Consultant. This supervision will take the form of periodic (every two weeks)

at UNICEF to brief PME and SPR Chiefs on progress and challenges being faced by the Consultant. A

schedule will be agreed upon with the Consultant once on board.

Quality Assurance

Throughout implementation of the evaluation, reports will go through a quality assurance process which

will involve the Research and Evaluation Committee.

8.0 PAYMENT SCHEDULE

Payment4 Conditions

20% Inception Report

30% Draft Preliminary Report

50% Final Report

9.0 QUALIFICATION/SPECIALIZED KNOWLEDGE AND EXPERIENCE

The consultancy firm should have a team of experts with the following qualifications:

One team leader

1. Advanced University Degree in Social Sciences, Development Studies, Public Administration,

Economic Development M&E or related field.

2. Work Experience: A senior and experienced individual with least 10 years of experience in

development programmes with substantial, current knowledge of UNICEF programme policies and

strategies, including theories of change for systems development programmes

3. Competencies: Sound knowledge and application of human rights and results-based programming,

including qualitative and quantitative approaches;

4. Demonstrated excellent writing skills in English. Strong coordination, communication, analytical skills

(examples of recent work in a similar assignment must be attached to the proposal)

5. Strong consultation and facilitation skills;

6. Demonstrated ability to work in a multi-cultural environment and establish harmonious and effective

working relationships, both within and outside the organization.

7. Proven experience in evaluation of Capacity building initiatives

8. Working experience in Zambia and/or East and Southern Africa will be an asset

4 In general, payments should be made against delivery of services / products. Advance payments on signature of contract are discouraged, and need to be explicitly justified.



Other team members should demonstrate the following;

1. A minimum of a master’s degree in Public Policy, Economics, Statistics, Social Sciences, or similar

relevant field;

2. At least 7 years of solid work experience in the areas of programme planning, evaluations;

3. Demonstrated working knowledge of Human Rights-Based Approach to Programming, Results-Based

Management, Equity and Gender Mainstreaming

4. Excellent communications skills in English and ability to communicate complex ideas in a

straightforward manner;

5. Excellent presentation skills and command of Microsoft Office Power Point and Word;

6. Knowledge of the region, Zambia and UNICEF policies and programming procedures will be an

advantage

ADMINISTRATIVE ISSUES

• Interviews, if necessary, will be conducted to seek clarification on the technical proposal and make

enquiries on the experts/positions proposed (in general, the evaluation of experts is conducted on the

basis of their CVs).

• The bidder should be provide an all-inclusive cost in the financial proposal. Bidder should be

reminded to factor in all cost implications for the required service / assignment

• When travel is expected as part of the assignment, it shall be clearly specified (e.g. location, duration,

number of journeys …etc.) in the TOR. Bidder shall be required to include the estimate cost of travel

in the financial proposal. It is essential to clarify in the TOR that i) travel cost shall be calculated

based on economy class travel, regardless of the length of travel and ii) costs for accommodation,

meals and incidentals shall not exceed applicable daily subsistence allowance (DSA) rates, as

promulgated by the International Civil Service Commission (ICSC).

• Unexpected travels shall also be treated as above.


AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG B-1

Annex B. Protocols

Health: Capacity development in use of the Health Management Information

System and the District Health Information System Introductory Comments and Informed Consent

Welcome to the discussion. Thank you for taking time to speak with me today. I am [NAME] from the

American Institutes for Research [AIR] and I will be conducting today’s interview. I am part of a team

which is conducting a study to evaluate the capacity development interventions implemented within the

health sector, specifically the Health Management Information System Strengthening (HMISS)

intervention. The purpose of this study is to explore the relevance, effectiveness, perceived impacts and

sustainability of the HMISS intervention.

The purpose of this Key Informant Interview is to better understand how the HMISS was implemented,

the process of designing the HMISS, and whether the HMISS led to any specific changes. In addition, we

also want to capture the engagement between stakeholders who were involved in this intervention,

explore anticipated outcomes and unanticipated interactions.

We are looking forward to knowing your opinions and experiences on this CD intervention, we encourage

you to be candid in your responses. This information will be used to inform UNICEF on what works and

whether CD interventions have led to the desired improvements in the skills and knowledge of

government officials, community-based volunteers and community members.

You should feel free to speak freely, as your name will be kept private and separate from the interview.

Only AIR and the researchers working with AIR will have access to your name and the information you

provide, and this will only be used for follow-ups and directly-related research purposes. All information

that is collected in this study will be treated confidentially. While the team will make results available, no

individuals will be identified in any way. There is minimal risk involved in this discussion. However,

participation in this discussion is completely voluntary, and you may choose to withdraw at any time.

Today’s session will take approximately 30-45 minutes. Your participation is voluntary. You can decide

to discontinue participating at any time. You also do not have to answer any questions you do not want to

answer. You may indicate at any time if you do not want to be quoted.

We would also like to audio-record today, so that our research team can review the recording later. Any

information that refers to you personally, like your name or organization will not be included in our

reporting of the interviews. Are you OK with being audio recorded today?

Do you have any questions? Do you agree to participate in today’s discussion? If you have questions

about the interview, please contact:

Hannah Ring of the American Institutes for Research (Tel. +202-403-6715), 1000 Thomas Jefferson St.

NW, Washington, DC 20007, USA

Claude Kasonka of the American Institutes for Research Zambia Office, (Tel. +260.211.372778), Elunda

II, Addis Ababa Roundabout, Regus office #115, Rhodes Park, Lusaka, Zambia



KII with Central Health Officers, MoH

Time: 30 mins

Respondent: Central Health Officers, MoH / M & E unit officers from the MoH

Goal: To better understand how the HMISS was implemented, the process of designing the HMISS, and

whether the HMISS led to any specific changes. In addition, to understand how central government

stakeholders defined existing needs, and how they partnered with UNICEF and other donors to select

priority areas, as well as organizations and individuals to be targeted.

I. Introduction and Background

1. Please tell me a bit about your position at the MoH.

a. How long have you worked in this position?

2. What was the main aim/purpose of the HMISS?

a. Who was included in the development of this intervention?

b. For whom was the HMISS developed (probe: central, provincial/district, facility-level)?

II. Design and Purpose [Relevance]

3. Please describe the needs assessment conducted by Akros on the use of HMIS.

a. What kind of gaps were identified from the assessment?

4. Can you walk me through how the CD trainings were implemented?

a. What types of trainings were included?

i. How were the trainings selected?

ii. How did you decide at what level what type of training was to be

implemented?

b. What challenges were there in the training delivery, if any?

c. What were the factors that made facilitated trainings?

5. Did the CD trainings include a M&E framework within its structure?

a. If yes, what were the M&E steps taken?

b. What was measured and what were the indicators used?

c. Who is responsible for monitoring and evaluating this CD intervention?

III. Activities [Effectiveness]

6. Why were the trainings were implemented at central, provincial/district, facility-level?

a. What kind of capacities needed to be developed?

7. What kind of trainings did you receive? (if any)

a. What was the purpose of this training?

b. Who provided the training?

c. Can you describe your experiences with the trainer?

i. Prompt: Was the trainer professional and was the trainer knowledgeable

about the training topic?

d. What were the challenges (if any) in this training approach?

e. At what level of staff was this training conducted?

f. Please describe the most important concepts you learned in the training.

g. Did you receive any training materials during your training?

i. If yes, what are they?

a. How often do you use them and in what ways?

ii. If no, would having access to training materials help you?

h. What recommendations do you have for improving the trainings?

i. For the ToT approach at the provincial/district level:

i. What were the challenges (if any) in the ToT approach?

ii. What were the facilitators in implementation of the ToT approach?



8. Did you or other MoH staff participate in workshops conducted by Akros?

i. If yes, what was the purpose of this workshop?

ii. For whom was this training conducted?

iii. What were the strengths of the workshop?

iv. What could be improved about these CD workshops?

IV. Perceived Changes [Perceived Impact]

9. Have there been any changes in staff’s capacities because of the CD training and workshops?

a. If yes, can you describe the changes?

i. How have your skills or knowledge of using the HMIS changed?

ii. Have there been increases in number of new equipment? Can you describe your

experiences in using the equipment?

iii. What has changed in how you and your staff enter and report data?

b. Were there changes at other levels (central, provincial/district and facility-level)?

i. At what levels is there a need for more CD support?

c. Do you think the trainings led to improvements in health service delivery?

i. If yes, how so?

ii. If no, what further CD is required?

V. Future Plans [Sustainability]

10. Did you consider how to ensure that changes continue in the future? Please describe.

a. What are the major barriers to continuation of these changes?

11. What kind of support will ensure that the skills and knowledge MOH staff received will continue to

help them achieve programme objectives? Probe: incentives, performance review

12. Currently, are there assessments of staff who have participated in this CD intervention?

A. If yes, who is conducting this assessment?

B. If no, how are staff evaluated on the use of their capacities to collect, manage and use

data to improve health service delivery?

13. Do you think it will be beneficial to expand the CD activities and trainings to other provinces?

A. Why or why not?

B. Do you suggest any changes to be made to the trainings and activities provided before

offering them elsewhere?

14. Is there anything else you would like to add before we conclude the interview?

Interviewer: Thank you very much for your participation in this study and for taking the time to speak

with us today. We greatly appreciate your cooperation.



KII with UNICEF officers, UNICEF Zambia

Time: 30 mins

Respondent: UNICEF Officers for the HMISS CD Intervention


whether the HMISS led to any specific changes. In addition, to understand how UNICEF defined existing

needs, and how they partnered with the MoH, other donors and its technical partner to select priority

areas, as well as organizations and individuals to be targeted.

Instructions Prior to Beginning Interview: Please introduce yourself to the respondents by stating your

name and your role as a researcher, then explain the purpose of the research and obtain verbal consent

from the respondent (see script above).


1. Please tell me a bit about your position at UNICEF Zambia.

a. For how long have you been working at UNICEF Zambia?

2. What was the main aim/purpose of the CD training on the HMISS?

A. Who was included in the development of the CD intervention?

B. For whom was this HMISS developed (probe: central, provincial/district, facility-level)?


3. Please tell us about the needs assessment that guided the CD intervention on HMIS.

a. What gaps were identified from the assessment? Probe: resources, skills, knowledge

4. Can you walk me through how the CD trainings were implemented?

A. What types of trainings were included?

ii. How were the trainings selected?

iii. How did you decide at what level what type of training was to be implemented?

iv. What challenges were there in the training delivery, if any?

v. What were the factors that made facilitated trainings?

vi. What steps were taken to ensure gender balance among training recipients?

vii. Prompt: Did UNICEF ensure that recipients included at least 50% women?

viii. Prompt: Did UNICEF have conversations with the implementer on the timings of when

the trainings were conducted? (to ensure trainings were conducted when women who

may have domestic and child caring responsibilities could actively participate)

5. Did the CD trainings include a M&E framework?

a. If yes, what were the M&E steps taken?

b. What was measured and what were the indicators used?

c. Who is responsible for monitoring and evaluating the HMISS trainings?

III. Activities [Relevance/Effectiveness]

6. Why were the trainings were implemented at central, provincial/district, facility-level ?

a. What kind of capacities needed to be developed?

7. For the ToT approach at the provincial/district level:

A. What were the challenges (if any) in the ToT approach?

B. What were the facilitators in implementation of the ToT approach?

1. Did you (UNICEF) provide any training materials? Please describe.

2. How do you think recipients responded to the trainings?

3. What were the strengths of the workshop?

4. What could be improved about these CD workshops?

VI. Perceived Changes [Perceived Impact]

8. Have there been any changes in staff’s capacities because of the CD training and workshops?

A. If yes, can you describe the changes?

i. How have skills or knowledge of using the HMIS changed?



ii. Have there been increases in number of new equipment?

iii. What has changed in how staff enter and report data?

B. Were there changes at other levels (central, provincial/district and facility-level)?

i. At what levels is there a need for more CD support?

C. Do you think the trainings led to improvements in health service delivery?

i. If yes, how so?

ii. If no, what further CD is required?

VII. Future Plans [Sustainability]

9. Did you consider how to ensure that changes continue in the future? Please describe.

A. What are the major barriers to continuation of these changes?

10. What kind of support will ensure that the skills and knowledge MOH staff received will continue

to help them achieve programme objectives? Probe: incentives, performance review

11. Currently, are there assessments of staff who have participated in this CD intervention?

A. If yes, who is conducting this assessment?

B. If no, how are staff evaluated on the use of their capacities to collect, manage and use


12. Do you think it will be beneficial to expand the CD activities and trainings to other provinces?

A. Why or why not?

B. Do you suggest any changes to the CD activities before offering them elsewhere?






KII: Implementer of HMISS CD Intervention

Time: 30 mins

Respondent: Implementer, AKROS


whether the HMISS led to any specific changes. In addition, to understand how central government

stakeholders and UNICEF Zambia defined existing needs, and how the implementers partnered with

UNICEF and other donors to select priority areas, as well as organizations and individuals to be targeted.




Interviewer: We would like to begin our conversation today by asking you some questions about your

role and knowledge on the HMISS intervention.


1. Please tell me a bit about your role at Akros.

a. How long have you been working at Akros?

2. What was the main aim/purpose of the HMISS intervention?

a. Who were the stakeholders included in the development of this intervention?

b. For whom was this HMISS developed?

c. Was this for staff at all levels (central, provincial/district and facility-level)?

d. What kind of capacities needed to be developed?


3. Please describe your experience collaborating with UNICEF Zambia for the purpose of designing

and implementing the CD trainings.

4. Please describe your experience collaborating with UNICEF Zambia for the purpose of

designing and implementing the CD trainings.

5. What other stakeholders were included in the design and implementation of the CD trainings?

6. The literature shows that a needs assessment was conducted by Akros on the use of HMIS.

a. What kind of gaps were found?


7. Let’s talk a bit about the trainings that were implemented by Akros:

8. Trainings at the national-level:

a. What type of training was implemented at this level?

b. Who were the recipients of the training?

c. How was the content of the trainings selected?

d. What materials were used in this training?

e. Were there any challenges encountered?

- If yes, what were they?

- How were the challenges overcome?

9. Trainings at the provincial/district level:

a. For the ToT approach at the provincial/district level:

i. What were the challenges (if any) in the ToT approach?

ii. What factors facilitated the delivery of the ToT?

iii. Is there anything you would do differently now in the design and

delivery of the ToT approach?

b. Who were the recipients of the ToT?

c. How was the content of the ToT training selected?

d. What kind of materials were used in this ToT?



e. Was there other type of trainings implemented at this level?

10. Trainings at the facility/ community level:

a. What type of training was implemented at this level?



d. What kind of materials were used in this training?

e. Were there any challenges encountered? Please describe.

i. How were the challenges overcome?

11. General:

a. Please describe how you worked with the MoH when delivering the trainings.

b. Please describe how you worked with UNICEF when delivering the trainings.

12. What was the gender balance among the recipients?

a. What steps were taken to ensure more women and disabled staff were also participating

in the trainings?

b. How did you account for any gender bias in the delivery of the trainings or in the content

of the trainings?

13. Did the CD trainings include a M&E framework? If yes:

a. What was measured and what were the indicators used?

b. Who is responsible for monitoring and evaluating the trainings?


14. How do you think the trainings changed MoH staff’s capacities? E.g.:

a. Have the trainings helped staff in making better decisions for the service delivery?

b. Can you describe the staff’s response and usage of the new equipment received?

c. Can you describe any changes in how MoH staff are doing the reporting and entry?

15. Do you see these changes at all levels (central, provincial/district and facility-level)?

A. If no, at what levels do you see the most changes?

B. At what levels is there a need for more CD support?

16. Would you say that there been improvements in health service delivery due to the trainings?

a. If yes, how so?

b. If no, what further support is required to use the HMIS effectively?


17. What factors were considered to ensure that the changes from above continues in the future?

A. What are the barriers to continuation, if any?

18. Currently, are there assessments of staff who have participated in trainings on the HMISS?

a. If yes, who is conducting this assessment?

b. If no, how are staff evaluated on the use of their capacities to collect, manage and use


19. What kind of support will ensure that the skills and knowledge MOH staff received will continue

to help them achieve programme objectives? Probe: incentives, performance review

20. Do you suggest any changes to be made to the trainings and activities provided before offering

them elsewhere?




KII with Province-level Senior Health Information Officers (SHIOs) and District-Level District

Health Information Officers (DHIOs)

Time: 45- 60 mins

Respondent: Province-level Senior Health Information Officers (SHIOs) and District-Level District

Health Information Officers (DHIOs) from the MoH, GRZ



Goal: To better understand how the HMISS was implemented at the provincial/district levels, the process

of designing the HMISS CD intervention activities at these levels, and whether the HMISS led to any

specific changes. In addition, to understand how trainings may have led to improvements in the capacities

of relevant staff to manage information systems, enter and clean data, and use improved data management

systems.







1. Please tell us a bit about your role at the MOH.

2. How long have you been working as a SHIO/DHIO at the MoH?


3. Why do you think the trainings were implemented at this level? (province for SHIO, district

for DHIO)?

a. What kind of capacities needed to be developed? Prompt: Were there gaps in

resources, in skills and knowledge?

4. Did the ToT training include a M&E framework?

a. If yes, what was measured and what were the indicators used?

b. Who is responsible for M&E?

III. Activities [Relevance/ Effectiveness]

Participation of SHIO/DHIO in CD training activity at provincial/district levels:

5. Can you tell me more about the training of trainers [ToT] you participated in?

A. What was the aim of the ToT CD activity?

B. Who provided the training?

C. Can you describe your experiences with the trainer?

i. Prompt: Was the trainer professional and was the trainer knowledgeable


D. Please describe the most important concepts you learned in the ToT training.

i. Prompt: how to mentor/supervise?

ii. Prompt: how to train facility-level staff/CHWs on DHIS2 data entry, ways to

scrutinize data, on ways to troubleshoot the system, to conduct real-time

monitoring, quality data-checks, data analysis and use

E. Did you receive any training materials during your training?


A. How often do you use them and in what ways?

i. If no, would having access to training materials help you?

6. Please describe strengths and challenges in the delivery of the ToT activity.

7. What were the challenges in participating in this ToT activity?

8. What factors did you like about the ToT activity?

9. Do you have any recommendations for improving the ToT CD activity?

Participation of SHIO/DHIO in mentorship/supervision CD activities implemented at the facility-level:

10. Did you participate in mentorship or supervision activities as a SHIO/DHIO?

A. If yes, can you take me through the steps of how you chose a mentee?

11. What kind of support and guidance did you provide the facility-level staff/ your mentees?



A. Prompt: train on DHIS2 data entry, ways to scrutinize data, ways to troubleshoot,

conduct real-time monitoring, quality data-checks, data analysis and data use

12. Are you currently still a mentor / supervisor?

A. If yes, has mentorship/supervision has been helpful to the facility-level staff?

B. If no, do you think having an ongoing mentor or supervisor will be helpful for

facility-level staff? Why?

i. Do you think having the same gender/sex mentee would be helpful for

you and your mentee? Why?

13. What challenges (if any) did you encounter when mentoring or supervising?

14. What factors made it easier for you to mentor/supervise?

15. Do you have any recommendations for improving these mentorship/supervision visits?


16. What changes in (facility-level or provincial/district-level) staff’s capacities (including yours)

resulted from the CD trainings, if any? Probes:

i. Skills or knowledge of using the HMIS

ii. Increases in number of new equipment received?

iii. Experiences in using the equipment?

iv. Changes in how you and your staff are doing the reporting and entry? Prompt: data

entry, data quality checks, data aggregation, data on RTM, data analysis and using

data for decision making?

17. Do you see these changes at all levels (central, provincial/district and facility-level)?

A. If no, at what levels do you see the most changes?

B. At what levels is there a need for more CD support?

18. Would you say that there have been improvements in health service delivery from these

trainings?

A. If yes, how so?

B. If no, what further CD activity is required?


19. Are you able to apply the practices you have learned in your trainings in your daily tasks?

A. If yes, which ones? What factors allow you to do this?

B. If no, what are the barriers that prevent you from doing this?

20. Do you think you can continue to train the trainees (facility-level staff and CHWs) on DHIS2

data entry and help enhance their capacity to use data for decision making?

A. If no, what kind of support do you need?

B. What kind of support do you think trainees (facility-level staff and CHWs) need to

effectively learn from your trainings?

21. What kind of support will ensure that the skills and knowledge facility staff received will

continue to help them achieve programme objectives? Probe: incentives, performance review

22. How are staff evaluated on the use of their capacities to collect, manage and use data to

improve health service delivery?

23. Would it be beneficial to expand CD trainings to other provinces in Zambia? Why?

A. Do you suggest any changes to be made to the trainings and activities provided

before offering them elsewhere?






KII with Facility In-Charge person | Environmental Health Technicians | Community Health

Workers

Time: 45- 60 mins

Respondent: In-Charge person of Health Facility| Environmental Health Technician | Community

Health Workers

Goal: To better understand how the HMISS was implemented at the facility and community levels, the

process of designing the HMISS CD intervention activities at these levels, and whether the HMISS led to

any specific changes. In addition, to understand whether training participants (e.g., Community Health

Workers [CHWs] and facility-level staff members) have a shared understanding of the goals of the

trainings, and whether those trainings improved their capacities to perform their tasks and deliver services

to the end beneficiaries.







1. Please tell me a bit about your position.

2. For how long have you been working as a MoH Facility-level staff/ Environmental Health

Technicians (EHT)/Community Health Workers (CHW)?


3. What was the main aim/purpose of the HMISS CD intervention?

4. What kind of capacities needed to be developed at the facility-level?

A. Prompt: Were there gaps in resources, in skills and knowledge?


5. Can you tell me more about the mentorship/supervisions CD activity you participated in?

a. Who mentored or supervised you during training?

i. Can you describe your experiences with the mentor/supervisor?

b. Was the mentor/supervisor professional and were they knowledgeable about the

training topic?

c. Please describe the most important concepts you learned during the

mentorship/supervision.

i. Prompt: training on DHIS2 data entry, ways to scrutinize data, ways to

troubleshoot the system, conduct real-time monitoring, quality data-

checks, data analysis and how to utilize the data?

d. Did you receive any training materials during your training?


ii. How often do you use them and in what ways?

iii. If no, would having access to training materials help you?

e. Do you currently still have a mentor, or supervisor?

i. If no, do you think having an ongoing mentor or supervisor will be helpful

for your CD?

f. Would you say that the mentorship/supervision was helpful? Why?

o Is there another type of CD activity that would have been more helpful?

g. Do you have any recommendations for improving mentorship/supervision visits?

6. Were there any factors that made it difficult for you to participate in the

mentorship/supervision CD activities? Please describe.

A. Were there any gender-related challenges that prevented you from participating?



7. What factors made it easier for you to participate in the mentorship/supervision CD

activities?

8. Do you have any recommendations for improving the mentorship/supervision CD activities?


9. Do you think the CD activities (mentorship/supervision) have helped improve your overall

work performance? How so?

i. How do you think your skills or knowledge of using the HMIS have changed?

ii. Prompt: capacity in data entry, data quality checks, data aggregation, data, on

RTM, data analysis and using data for decision making

iii. Increases in number of new equipment received?

i. Can you describe your experiences in using the equipment?

a. Can you describe any changes in how you and other facility-level

staff are doing the reporting and entry?

ii. Prompt: capacity in data entry, data quality checks, data aggregation, data,

on RTM, data analysis and using data for decision making? improved in

data entry, data quality checks, data aggregation, data, on RTM, data

analysis and using data for decision making?

a. If no, what kind of support do you (and your colleagues at facility-

level) need to further develop capacities in using HMIS?

10. Would you say that there been improvements in health service delivery due to these trainings

on the use of the HMIS?

A. If yes, how so?

B. If no, what further CD activity is required?


11. Are you able to apply the skills and knowledge that you developed in your CD activities

(mentorship/supervision) into your daily tasks?

A. If yes, can you describe some of the factors that allow you to do this?

i. Are there any particular training concepts that you are using?

B. If no, what are the barriers that prevent you from doing this?

12. How are staff evaluated on the use of their capacities to collect, manage and use data to

improve health service delivery, if at all?

13. Would it be beneficial for CD trainings to be expanded to other provinces in Zambia? Why?

A. Do you suggest any changes to the activities before offering them elsewhere?






KII with Community Beneficiaries

Time: 45- 60 mins

Respondent: Community-level Beneficiaries

Goal: To better understand how the HMISS was implemented at the community levels, and whether the

HMISS led to any specific changes. In addition, to understand whether the trainings and the HMISS had

an impact on the end-users who are the members of the community.





knowledge on the HMISS intervention.


1. Please introduce yourself and tell us a bit about yourself, including your occupation.

a. Do you have any children? If yes, how many?

II. Purpose

2. Can you talk me through your visits to the health facilities?

a. Why do you visit the health facility?

o Prompt: When your child is sick? When you are pregnant? Do you get your

children for health check-ups regularly?

b. Can you provide an estimate on how many times you visit the health facility?

i. Prompt: How many times in a month/year do you visit the health facility?

III. Activities and Perceived Changes [Relevance/Effectiveness]

3. Can you describe your experiences using the health facilities?

a. What type of services do you receive?

b. Have you been satisfied with the service you receive?

i. If yes, what factors helped make your visit a positive experience?

ii. If no, can you describe why not?

c. Can you tell me more about your interactions with staff?

a. Who do you usually interact with?

b. Are you satisfied with the number of women staff members in the facility?

a. Would you feel more comfortable being attended to by a person of your

sex/gender?

c. How do the staff treat you?

d. What do you think the staff do best at the health facility?

e. What do you think staff do not do well?

a. What do you think is preventing staff from providing good care and

services? Prompt: Are staff not sufficiently trained to provide care? Do

they not have sufficient resources to provide care?

4. What recommendations do you have for staff to improve their delivery of care to you?

5. Have you interacted with community health workers before?

a. If yes, where? Prompt: was this at your home?

b. What did they speak to you about?

6. Did UNICEF Zambia, the health facility staff or anyone else speak with you earlier to understand

what community members want from health facilities?

7. Is there anything else you would like to add?





Social protection: Capacity development in beneficiary targeting and selection

for scale-up of the SCT programme

Introductory Comments and Informed Consent

Welcome to the discussion. Thank you for taking time to speak with me today. I am [NAME]

from the American Institutes for Research [AIR] and I will be conducting today’s interview. I am

part of a team which is conducting a study to evaluate the capacity development interventions

implemented within the social protection sector, specifically the scaling up of the SCT through

increasing the capacity in selection of beneficiaries. The purpose of this study is to explore the

relevance, effectiveness, perceived impacts and sustainability of the SCT scale-up intervention.

The purpose of this Key Informant Interview is to better understand how the SCT scale-up

intervention was implemented, the process of designing the SCT scale-up intervention and

whether the SCT scale-up intervention led to any specific changes. In addition, we also want to

capture the engagement between stakeholders who were involved in this intervention, explore

anticipated outcomes and unanticipated interactions.

We are looking forward to knowing your opinions and experiences on this CD intervention, we

encourage you to be candid in your responses. This information will be used to inform UNICEF

on what works and whether CD interventions have led to the desired improvements in the skills

and knowledge of government officials, community-based volunteers and community members.

You should feel free to speak freely, as your name will be kept private and separate from the

interview. Only AIR and the researchers working with AIR will have access to your name and

the information you provide, and this will only be used for follow-ups and directly-related

research purposes. All information that is collected in this study will be treated confidentially.

While the team will make results available, no individuals will be identified in any way. There is

minimal risk involved in this discussion. However, participation in this discussion is completely

voluntary, and you may choose to withdraw at any time.

Today’s session will take approximately 30-45 minutes. Your participation is voluntary. You can

decide to discontinue participating at any time. You also do not have to answer any questions

you do not want to answer. You may indicate at any time if you do not want to be quoted.

We would also like to audio-record today, so that our research team can review the recording

later. Any information that refers to you personally, like your name or organization will not be

included in our reporting of the interviews. Are you OK with being audio recorded today?

Do you have any questions? Do you agree to participate in today’s discussion? If you have

questions about the interview, please contact:

Hannah Ring of the American Institutes for Research (Tel. +202-403-6715), 1000 Thomas

Jefferson St. NW, Washington, DC 20007, USA

Claude Kasonka of the American Institutes for Research Zambia Office, (Tel. +260.211.372778), Elunda

II, Addis Ababa Roundabout, Regus office #115, Rhodes Park, Lusaka, Zambia



KII with Ministry Officials at Central/Institutional Level:


1. Could you please tell us about your position at MCDSS?


2. What was the main aim/purpose of capacity development activities in 2017, when trainings were

delivered for employees and staff on screening and enrolling individuals into SCT?

3. Can you please describe TRANSFORM trainings provided for officials from the MCDSS SCT Unit,

as well as officers from the MCDSS M&E Unit?

a. What were the goals of the TRANSFROM training package?

b. Who was the trainings developed for?

c. Was TRANSFORM training provided to staff at all levels (central/provincial/district and

community-level)?


4. Was there an assessment of existing needs when deciding who (which particular

individuals/organizations) to target for the TRANSFORM trainings?

A. If so, who conducted the needs assessment?

B. What did the needs assessment entail?

5. What types of gaps in knowledge/skills existed prior to the trainings in 2017, among the different

groups of staff involved in targeting and selection for the SCT programme?

6. What (if any) challenges do employees of [UNICEF/MDCSS] face as they do targeting and selection

for the SCT?


7. Why were the TRANSFORM trainings implemented at national, provincial/district and community

levels?

A. What kinds of capacities needed to be developed at each of these levels?

8. How were the provinces and districts selected to receive TRANSFORM trainings?

9. What kinds of trainings did you and your colleagues at MCDSS receive in 2017 related to screening

and enrolling SCT beneficiaries?

A. Who provided the training?

B. Can you describe your experience in the training?

C. What were the challenges in participating in the trainings?

D. Can you share any important concepts you learned in the training?

E. Did you receive any training materials during your training?

i. If yes, what types of materials were they?

ii. Did you use them after trainings?

iii. If no, would having access to training materials been helpful for you?

F. What types of recommendations do you have for improving the trainings for officials at your

level?


10. How (if at all) did MCDSS staff’s capacities related to targeting, case management, selection and

enrolment of beneficiaries in SCT (including yours) change following the TRANSFORM training in

2017?



a. What types of changes did you observe in staff’s ability to screen and enroll SCT

beneficiaries?

b. If no changes were observed, why do you think there have not been any improvements in

staff’s ability to screen and enroll SCT beneficiaries?

c. What particular indicators have you used to measure progress or the development of capacities

in targeting/selection/enrollment?

11. How would you describe changes in other level staff’s (provincial/district and facility-level)

capacities after the CD training in 2017?

a. What types of changes happened at these levels?

b. Is there a need for more support for capacity development? If so, at what levels?


12. Did the government share the costs of the trainings, including the TRANSFORM package of trainings

rolled out in 2017?

13. Did the government decide to invest any resources after the trainings including TRANSFORM

trainings implemented in 2017?

A. If so, what kinds of resources and investments?

B. How were the trainings expanded further?

14. Are there similar activities being supported by other donors?

15. Are there plans to assess the capacities of staff at the ministry levels, but also provincial, district and

local levels?

16. Thinking back, how would you have asked implementers to design and deliver the trainings

differently?

A. What in particular could be improved? Why?

17. What could UNICEF do better to support capacity building of SCT staff in targeting/selection?

18. Is there anything else you would like to add before we conclude?

Thank you so much for your time!



KII with UNICEF Officers at Institutional Level


1. Could you please describe your role at UNICEF?

A. How long have you worked in this position?

2. What was the main aim/purpose of capacity development activities in 2017, when trainings were

delivered for different staff on screening and enrolling individuals into SCT?

3. Can you please describe TRANSFORM trainings as well as the ToT approach that UNICEF

supported?

A. What were the goals of the trainings?

B. Who was the trainings developed for?

C. Was this for staff at all levels (central/provincial/district and facility-level)?



individuals/organizations) to target for the TRANSFROM and ToT trainings?

a. If so, who commissioned the needs assessment?

b. Who conducted the needs assessment?

c. What did the needs assessment entail?

d. What kinds of gaps in skills and knowledge of different staff were identified based on the

assessment?

e. If not, do you think the way the TRANSFORM and the TOT trainings would have benefited

from an assessment?


groups of staff involved in carrying out the SCT programme?

6. What (if any) challenges do employees of UNICEF face currently in implementing the SCT?


7. How were the provinces and districts selected as targets for the TRANSFORM training package

and ToT?

8. For the TRANSFORM approach:

A. What types of challenges did the implementers face when providing trainings at different

levels?

B. What factors facilitated implementers when they provided trainings at different levels?

9. For the ToT approach:

A. What types of challenges did OPM face when providing trainings at different levels?

B. What factors facilitated OPM when they provided trainings at different levels?

10. How did UNICEF facilitate different training activities?

A. Did UNICEF provide any training materials?

B. If so, what types of materials?

11. How did training participants respond to the trainings?

12. What steps were taken to ensure that there was gender balance among the recipients of the

trainings?

A. Prompt: What steps were taken to ensure that recipients included at least 50% of women?

B. Did UNICEF have conversations with the implementer on the timings of when the trainings

were conducted? (To ensure that trainings were not conducted during after work hours, which

may prevent women who may have domestic and child caring responsibilities from actively

participating?)



13. Were there any challenges in implementation of the trainings?

A. If yes, what were they?

B. Prompt: Were there any gender-related challenges?

C. Prompt: Were there any challenges related to the trainings being inaccessible to staff with

disabilities?


14. Did the TRANSFORM trainings and ToT approach have their own monitoring and evaluation?

A. If so, could you please tell us more about this M&E?

B. How often was it implemented?

C. Are the results of the M&E available?

15. In your observations, what types of changes happened in targeting, case management, selecting and

enrolling beneficiaries in SCT after the trainings in 2017?

A. If so, which particular aspects of work changed?

B. What types of changes were those? Please provide some illustrations.

C. If not, why do you think there have not been any improvements?

D. What particular indicators have you used to measure progress or the development of

capacities?


16. Did the government share the costs of the trainings with UNICEF, including the TRANSFORM

package of trainings and ToT rolled out in 2017?

17. Did the government decide to invest any resources after the trainings including TRANSFORM

trainings implemented in 2017?

A. If so, what kinds of resources and investments?

B. How were the trainings expanded further?


19. Are there plans to assess the capacities of staff at the ministry levels, but also provincial, district

and local levels?


differently?

A. What in particular could be improved?

B. Why do you think these changes would have helped?

21. What could UNICEF do better to support training and capacity building of SCT staff?





KII with Master trainer, Provincial government officials

I. INTRODUCTION/BACKGROUND

1. What institution do you work for and what is your role in this institution?

a. How long have you been working on the SCT programme and what are your

responsibilities under the programme?

2. Can you please tell us a bit about the scale-up of the SCT programme and the trainings on

targeting and selection of beneficiaries for the SCT programme?

II. PURPOSE AND DESIGN

Thank you. Now I want to ask some more questions about the trainings you received during the scale

up of Social Cash Transfer programme.

3. What training did you receive on targeting/selecting beneficiaries for the SCT?

a. Who trained you?

b. Can you describe the topics covered in the training?

c. Did you feel like you gained new knowledge and skills through those trainings? If so

what were they?

d. Did your responsibilities change after you went through the training? If yes, how so? (If

he/she mentions training of the DSWO, DWAC chairperson and enumerators, probe: if

they felt adequately qualified to train the DSWO, DWAC chairperson and enumerators).

e. In your view, were the training content and methods appropriate given your needs and the

needs of DSWOs, DWAC chairpersons and enumerators? (Probe for: did they feel they

adequately trained the DSWO, DWAC chairperson and enumerators).

f. Did you receive any training materials during your training? If yes, what are they? (probe

for: ICT equipment and other materials)

g. How often do you use them and in what ways?

h. If no, would having access to training materials help you?

i. Is there anything you did you not like about the training you received or delivered on

targeting/enrolling SCT beneficiaries?

j. Were there any challenges delivering the training to DSWO, DWAC chairperson and

enumerators? If yes, what were they?

i. Prompt: Were there any gender-related challenges?

ii. Prompt: Were there any challenges related to the trainings being inaccessible

to staff with disabilities?

k. Do you have any recommendations for improving the training you received?

4. Do you think the training you received on targeting/enrolling SCT beneficiaries was responsive to

your particular needs? If so, why?

III. ACTIVITIES

Next, I want to ask you some questions about the training activities under the scale up of Social Cash

Transfer programme

5. Are you aware of goals of the trainings on targeting/enrolling SCT beneficiaries?

6. Do you think the training you received prepared you to train others on selecting/enrolling

beneficiaries for the SCT? How so?

7. How would you describe the results of the training you conducted with DSWO, DWAC

chairperson and enumerators? Do you think you were able to transfer knowledge/skills effectively

through the training? Why or why not?

Was there anything that made it easier or harder to deliver the training to DSWOs, DWAC

chairpersons, and enumerators? Please describe.



IV. PERCEIVED CHANGES

Next, I would like to ask you some questions about the changes that the trainings had on social cash

transfer programme

8. Have you observed changes in yourself or others in terms of knowledge/skills and capacities to use

technology/ software (MIS and Mobile technology) since the training?

9. What impact do you think the CD training you provided to the DSWO and enumerators has had

on their knowledge/skills and capacities to use technology software (MIS and Mobile technology)?

10. Do you think the training you provided to DSWOs and enumerators improved the targeting,

selection, and enrollment of SCT beneficiaries? If yes, how so?

V. FUTURE PLANS/ACTIONS

Finally, I want to ask questions related to the utilization of the MIS and Mobile technology and the

programme’s future plans

11. Have you continued to use the skills/technologies you learned through the TRANSFORM

trainings? How so?

a. In what ways do you use the MIS/Web portal?

b. Do you think the staff you trained have continued to apply what they learned during

training when targeting and selecting SCT beneficiaries? Are they comfortable using MIS

and mobile technology for targeting/selection?

c. Did you make sure trainees could ask follow-up questions?

12. How do you use MIS in decision making?

13. How does the DSWO you trained use the MIS to make decisions?

14. What do you appreciate about the MIS and Mobile Technology in comparison to the previous

method of targeting and selecting SCT beneficiaries?

15. What challenges have you faced using the MIS?

16. What challenges do the DSWO and enumerators face using the MIS and Mobile Technology?

17. What factors have made it easier or harder to roll out the MIS and Mobile technology?

18. Is there anything/anyone that helped you to continue using the skills/ technologies you received

from the training?

19. Is there anything/anyone that helped the DSWOs and enumerators you trained to continue using

the skills/ technologies (MIS and Mobile Technology) learned during training?

20. Has MCDSS allocated resources to continue providing trainings to the TOTs/ Master trainers?

21. Do you have any recommendations for how TRANSFORM trainings could be improved?

22. Is there anything else, you like to mention before we end the interview?

Thank you for your time



KII Guide ACC (Chairperson/ Secretary)

I. INTRODUCTION/ BACKGROUND

1. How long have you been a member of ACC and what are your responsibilities under the SCT

programme?

2. What was the purpose of the training on beneficiary targeting and selection that you received as

part of the scale up of the SCT programme?


Thank you. Now I want to ask some more questions about the trainings you received on beneficiary

targeting/selection

3. What trainings did you receive on targeting/selecting SCT beneficiaries during the scale-up?

a. Who trained you?

b. Was the trainer knowledgeable about the training topic?

c. What were the objectives of the training?

d. Can you describe the topics covered in the training?

i) Please describe the most important concepts you learned in the training.

e. Did you feel like you gained new knowledge and skills through trainings? If so, what?

f. Did your responsibilities change after the training? How so?

i) (If he/she mentions monitoring all the CWACs at ward level Ask: if they felt

adequately qualified to monitor all the CWACs at ward level and handle

grievances from all CWACs before the DWAC chairperson).

g. Did you face any challenges participating in this training?

h. Did you receive any training materials during your training? If yes, what are they? (probe

for: Manuals).

i. How often do you use them and in what ways?

j. If no, would having access to training materials help you?

k. What challenges, if any, did you encounter in the implementation of this training?

l. Do you have any recommendations to improve the trainings?

4. Do you think the targeting/selection trainings under the scale up of SCT programme are

appropriate for your community? How so?

III. ACTIVITIES

Next, I want to ask you some questions about the training activities in the scale up of SCT programme

5. Do you think the training you received on targeting/selecting SCT beneficiaries achieved its

intended goals?

6. Did the training you received improve your ability to carry out your work as an ACC member?

7. Did the training you received equip you with skills you need to work with DSWO, DWAC

members, CWAC members and enumerators on targeting/selecting SCT beneficiaries?

8. Did anything/anyone make it easier or harder to participate in the trainings for ACC members ?

Please describe.


Next, I would like to ask you some questions about the changes that the trainings have had on social

cash transfer programme.



9. Have you noticed any change in your knowledge/skills and capacities (as they related to your role

on the SCT) since you received the training?

10. What impact do you think the training has had on your working relationship with the CWACs?

11. Did the training the CWACs received improve the selection and targeting of the beneficiaries?

How so?

12. In your view, did the trainings lead to improved delivery of services to beneficiaries? If yes,

explain.


Finally, I want to ask questions related to programme’s future plans and the utilization of forms 1

and 2.

13. Have you continued to use what you learned during the training? How so?

14. Do you think the CWACs have continued to use Form 1 and Form 2 in the targeting and the

selection of SCT beneficiaries?

15. Do you use Form 1 and Form 2 in decision making? How so?

16. Do members of the CWACs you work with use the forms in decision making? How so?

17. How are Form 1 and Form 2 different from the previous method of targeting and selecting SCT

beneficiaries? Do the forms make it easier or harder?

18. What challenges have you faced using the forms 1 and 2.

19. What challenges do the CWACs face using these forms?

20. Who in particular has helped you to ensure that you can keep using the knowledge and skills

acquired through the trainings?

21. Who in particular has helped the CWAC members to ensure that they can keep using the

knowledge and skills acquired through the trainings?

22. Were you provided with materials and resources that you could use after the training?

23. How do you address any follow-up questions that may have emerged after you attended trainings?

24. Has MCDSS allocated resources to continue providing capacity development trainings to the

ACCs?

25. Do you have any recommendations for how to improve trainings for ACC members and CWACs

on beneficiary targeting and selection?





KII with DSWO


1. What institution do you work for and what is your role in this institution?

a. How long have you been working on the SCT programme and what are your

responsibilities under the programme?

2. What was the purpose of the training on beneficiary targeting and selection during the scale up of

the SCT programme?


Thank you. Now I want to ask some more questions about the trainings on beneficiary targeting and

selection

3. What training did you receive on beneficiary targeting/selection during the scale-up of the SCT

programme?

a. Who trained you?

b. What were the objectives of the training?

c. Can you describe the topics covered in the training? (Probe for: reactivation and

formation of CWACs, definition of catchment areas, entering of data from CWACs and

generation of lists of households for enumeration, listing eligible beneficiary households,

the use of the Web portal/ Management information system (MIS) to enter and monitor

data from enumerators, verification and validation).

d. Did you feel like you gained new knowledge and skills through the trainings? If so what

were they?

e. Did your responsibilities change after you went through the training? (If he/she mentions

reactivation and formation of CWACs, definition of the catchment area, entering of data

from CWACs and generation of lists of households for enumeration, listing eligible

beneficiary households, the use of the Web portal/ Management information system (MIS)

to enter and monitor data from enumerators and validation, Ask: if he/ she felt

adequately trained to carry out these responsibilities).

f. Did you receive any training materials during your training? If yes, what are they? (probe


g. How often do you use them and in what ways?

h. If no, would having access to training materials help you?

i. What did you not like about trainings?

j. Did you face any obstacles to participating in this training? If yes, what were they?

k. Do you have any recommendations for improving the trainings?

4. Do you think the training you received is appropriate for your community? If so, why?

III. ACTIVITIES

5. Next, I want to ask you some questions about the training activities Do you think the training you

received on beneficiary targeting/selection/enrollment achieved its intended goals?

6. Did the training you received improve your ability to carry out your work as a DSWO under the

SCT?



7. Did the training you received equip you with skills you need to work with DWAC members,

CWACs, ACCs and enumerators?

8. Did anything make the trainings easier or harder to participate in?


Next, I would like to ask you some questions about the changes that the trainings have had under the

social cash transfer programme

9. What impact do you think the training you received has had on your knowledge/skills and

capacities to use technology/ software (MIS /Web portal)?

10. What impact do you think the training of enumerators, CWACs and ACCs had on their

knowledge/skills and capacities to use technology software such as the mobile technology and

forms one and two?

11. Did the training you received change your work practices or those of DWAC members,

enumerators, CWACs, and ACCs?

12. Did the training of the CWACs, ACCs and enumerators improve the selection and targeting of the

beneficiaries? How so?

13. In your view, did the trainings lead to improved delivery of services to beneficiaries? If yes, please

explain.




14. Have you continued to use the skills/technologies (MIS) you learned during the training? How so?

15. Do you use the MIS in decision making? How and for what decisions?

16. How is the MIS different from the previous method of targeting and selecting beneficiaries for the

SCT programme?

17. What challenges have you faced using the MIS? (Probe for: challenges that enumerators faced in

uploading information on the MIS web portal and use of the Mobile technology (M-tech)

18. What made it easier or harder to roll out the MIS and Mobile technology?

19. What will help you continue using the skills/ technologies you learned during the training?

20. Who in particular has helped to ensure that you can keep using the knowledge and skills acquired

through the trainings?

21. Who in particular has helped to ensure that the enumerators keep using the knowledge and skills

acquired through the trainings?

22. Has MCDSS allocated resources to continue providing capacity development trainings to

DSWOs?

23. Do you have any recommendations for how to better deliver trainings on beneficiary

targeting/selection?


Thank you!



KII with Members of the Community Welfare Assistance Committee (CWAC)/ Secretary and

Chairperson of the CWAC


1. How long have you been a member of CWAC and what are your responsibilities under the SCT

programme?

2. What training did you receive on beneficiary targeting and selection during the scale up of the

SCT programme? Please describe.


Thank you. Now I want to ask some more questions about the training you received on beneficiary

targeting and selection during the scale-up of the Social Cash Transfer programme

3. What were the objectives of the training?

a. Who trained you?


c. Can you describe the topics covered in the training?

d. Please describe the most important concepts you learned in the training.

e. Did you feel like you gained new knowledge and skills through those trainings? If so what

were they?

f. Were there any challenges that prevented you from participating? For example, were

trainings conducted during after work hours, which may have prevented you from actively

participating due to domestic and child caring responsibilities? (Probe for any challenges

specifically related to gender)

g. Did your responsibilities change after the training? (If he/she mentions identification and

listing the eligible beneficiary households in form 2; participate in the verification and

validation process and accompanying the enumerators and helping them locate the listed

beneficiaries, probe: if they felt adequately qualified to carry out these activities).

h. Did you receive any training materials during your training? If yes, what are they? (probe

for: Manuals).

i. How often do you use them and in what ways?

j. If no, would having access to training materials help you?

k. Were there any challenges in the implementation of this training (i.e., logistical

challenges)? If yes, what were they?

4. Do you have any recommendations for improving the trainings for CWACs on beneficiary

targeting/selection/enrollment?

III. ACTIVITIES

Next, I want to ask you some questions about the training activities you participated in

5. What were the goals of the trainings on beneficiary targeting/selection/enrollment for CWAC

members?

6. Do you think the training you received achieved its intended goals?

7. Did the training you received improve your ability to carry your work as CWAC member?

8. Did the training you received equip you with the skills you need to work with the DSWO, DWAC

members, ACC members and enumerators?

9. Did anything/anyone make the training easier or more difficult for CWAV members to participate

in?






10. What changes have you observed in your own knowledge/skills and ability to use forms 1 and 2

since you received the training?

11. What impact do you think the training for CWAC members had on your working relationship with

the DWAC, DSWO, ACC and enumerators?

12. Did the training you received change how you target/select/enroll beneficiaries? How so?

13. Did the training that you received improve the selection and targeting of the beneficiaries? How

so?

14. In your view, did the various trainings you received lead to improved delivery of services to

beneficiaries? If yes, please explain.


Finally, I want to ask questions related to future training plans and the utilization of forms 1 and 2.

15. Have you continued to apply the skills you learned from the training? How so? Are you using

Form 1 and Form 2 in the targeting and the selection of SCT beneficiaries?

16. Do you use Form 1 and Form 2 in decision making?

a. What kinds of decisions are those?

17. How does Form 1 and Form 2 compare to the previous method of targeting and selecting

beneficiaries of the SCT programme? Do you think it’s more/less efficient?

18. Do you face any challenges using the forms 1 and 2? What are they?

19. Did anything/anyone make it easier or more difficult to roll out forms 1 and 2? Please describe.

20. Has anyone helped you to keep using the knowledge and skills acquired through the trainings?

Who?


22. Have you had any follow-up questions since the training? Did anyone answer these for you?

Who?

23. Has MCDSS allocated resources to continue providing trainings to the CWAC members? Please

describe.

24. Do you have any recommendation for future CWAC trainings on beneficiary






FGD with Members of the District Welfare Assistance Committee (DWAC)


1. How long have you been a member of DWAC and what are your responsibilities under the SCT

programme?

2. What, in your view was the purpose of the training on beneficiary targeting and selection for the

scale up of the SCT programme?


Thank you. Now I want to ask some more questions about the trainings under the scale up of Social

Cash Transfer programme.

3. What training did you receive on beneficiary targeting/selection/enrollment?

a. Who trained you?

b. What were the objectives of the training.

c. Can you describe the topics covered in the training?

d. How was the training conducted?

4. Did you feel like you gained new knowledge and skills through those trainings? If so what were

they?

5. Did your responsibilities change after the training? (If he/she mentions sensitization of the

communities on the SCT programme, handling of grievances from CWAC and ACC selection and

training of members of CWAC. Participation in the verification process, Ask: if they felt

adequately qualified to train members of the CWAC and ACC).

6. In your own view, were the training methods appropriate given the needs of the CWAC and ACC

members you trained? (Ask: what topics did the cover in the training, how was the training

conducted, did you feel that you were adequately trained the CWAC and ACC members. Probe

for: did the members of CWAC understand how to use Form 1: list of CWAC members and Form

2: list of potential beneficiaries)

a. Did you receive any training materials during your training? If yes, what are they? (probe


b. How often do you use these materials and in what ways?

c. If no, would having access to training materials help you?

d. What did you like most about trainings? And least?

e. Were there any challenges implementing this training? If yes, what were they? Probe for

logistical issues, timing, gender-specific obstacles to ACC/CWAC members participating

in the training.

f. Do you have any recommendations to improve the trainings for CWAC and ACC

members?

7. Do you think the training you received is appropriate for the context of you district in Zambia? Is

it appropriate for the needs/abilities of CWAC/ACC members? How so?

III. ACTIVITIES

Next, I want to ask you some questions about the training activities: What were the expected outcomes

of the training on beneficiary targeting/selection/enrollment for DWAC members? For ACC/CWAC

members?

8. Do you think the training you received achieved its intended goals? Why or why not?

9. Did the training you received improve your ability to carry out your work as a member of the

DWAC?

10. Did the training you received equip you with skills needed to conduct trainings for CWACs and

ACCs? How so?

11. What were the outcomes of the training you conducted with CWACs and ACCs?



12. Did anything or anyone make it more difficult to facilitate the training for CWACs and ACCs?

Did anything make it easier to do so?



cash transfer programme

13. What impact do you think the training you received has had on your knowledge/skills? What about

you comfort/ability to use form 1 and 2 and other technologies?

14. What impact do you think the training you provided to the CWACs and ACCs has had on their

knowledge/skills and ability to use form 1and 2 and other technologies?

15. Did the training you received change your work practices? What about the practices of CWACs

and ACCs?

16. Did the training you provided to CWACs and ACCs improved the selection and targeting of SCT

beneficiaries?

17. In your view, did the various trainings led to improved delivery of services to beneficiaries? If yes,

please explain.




18. Have you continued to use the skills you learned during the training? How so?

a.

19. Do you use Form 1 and Form 2 in decision making? How so?

a. How does the MIS help you make decisions?

b. Do you think the CWACs and ACCs you trained have continued to use the knowledge

learned in the training (for example, Form 1: list of CWAC members and Form 2: list of

potential beneficiaries) in the targeting and the selection of SCT beneficiaries?

20. How does using Form 1 and Form 2 compare to the previous method of targeting and selecting

beneficiaries?

21. What challenges do the CWACs and ACCs face using Form 1 and Form 2 in targeting and

selecting beneficiaries?

22. Did anything make it easier to roll out Form 1 and Form 2? Did anything make it more difficult?

Please describe.

23. In your view, what factors will help you to continue using the skills you learned from the training?

24. In your view, what factors will help the CWACs and ACCs you trained continue using Form 1

and Form to target and select beneficiaries? What about continuing to apply the other skills

CWACs and ACCs learned during training?

25. Has anyone helped to ensure that you can keep using the knowledge and skills acquired through

the trainings?

26. Has MCDSS allocated resources to continue providing trainings or other capacity development

(mentoring, coaching, etc.) to the members of the DWAC?

27. Do you have any recommendations to improve future trainings on beneficiary






KII with SCT Scale Up CD intervention Implementers


1. Could you please describe a bit about your role at [organization to be inserted upon confirmation

from UNICEF]?

A. How long have you served in this role?

2. How closely do you usually work with the Ministry of Community Development and Social

Services (MCDSS) in general?

A. Which particular department/division do you work with closely?


3. Could you tell us about the capacity development trainings you provided to MCDSS and other

government officials on beneficiary targeting/selection/enrollment?

a. What were the goals of the TRANSFORM training package?

b. Who was the trainings developed for?

c. Was this for staff at all levels (central/provincial/district and facility-level)?

d. What kinds of capacities did you aim to improve through the trainings?

4. What were the three major factors that shaped how you decided to design and implement the

trainings?

5. When designing the TRANSFORM training package, how did your organization decide what the

existing gaps in knowledge and skills were?

a. Was there an assessment of existing needs and knowledge/skills?

b. If so, what did this assessment entail?

c. Who implemented the assessment?

d. What kinds of gaps were found?

e. If not, how do you think the TRANSFORM training would have benefited from an

assessment?

6. Who in the UNICEF Zambia staff did you work with closely for the purpose of designing and

implementing the TRANSFROM training package?

a. What was your experience like working with them?

7. Who in the MCDSS staff did you work with for the purpose of designing and implementing the

TRANSFROM training package?

A. What was your experience like working with them?

8. What was the gender balance among the recipients?

A. What steps were taken to ensure more women and disabled staff were also participating

in the trainings?

i. Prompt: Did UNICEF or the Ministry require that the recipients included at least

50% of women?

ii. Prompt: Did UNICEF have conversations with you to ensure that timings of

trainings were inclusive? (To ensure that trainings were not conducted during

after work hours, which may prevent some people, especially women who may

have domestic and child caring responsibilities from actively participating?)

B. How did you account for any gender bias in the delivery of the trainings and in the

content of the trainings?

9. How would you describe the cooperation between UNICEF Zambia and MCDSS with your

organization, when discussing the purpose of trainings, their scope and scale?

III. Activities [Relevance]

10. Could you tell us how TRANSFORM trainings package differs from other types of trainings?

11. Now we would like to hear a little bit about trainings at the national level:



a. What was the primary goal of trainings for the employees of MCDSS? Did this vary by

level/type of employee?




e. What types of challenges did you encounter working with the Ministry-level officials?


a. What was the primary goal of trainings for the district officials, district social welfare

officers, and members of the District Welfare Assistance Committee (DWAC) and the

Area Coordinating Committee (ACC)?

b. How was the content of the trainings selected?

c. What kind of materials were used in this training?

d. What types of challenges did you encounter working with the government officials at this

level?

Trainings at the community/facility level:

a. What was the primary goal of trainings for members of the CWAC?



d. What types of challenges did you encounter working with the CWAC?

13. Can you describe three major challenges encountered during delivery of the trainings?

a. What types of challenges were they?

b. How did UNICEF staff facilitate/hinder the delivery of trainings?

c. How did MCDSS staff facilitate/hinder the delivery of trainings?

14. How did you ensure that the trainings were understandable for participants?

15. How did you ensure that the trainings were conveniently scheduled for participants?

IV. Perceived changes [Perceived impact]

16. How do you think the trainings have changed the capacities of trainees?

a. At the Ministry level, could you describe the staff’s response to trainings, and what they

learned?

b. How did the MCDSS staff change their work after the trainings?

c. At the province/district level, could you describe how the district officials, district social

welfare officers, and members of the District Welfare Assistance Committee (DWAC)

and the Area Coordinating Committee (ACC) respond to trainings and what they learned?

d. How did they change their work after the trainings?

e. At the community level, can you describe how the CWAC members responded to

trainings and what they learned?

f. How did they change their work after the trainings?

17. How did your organization gauge whether individuals/organizations were gaining new knowledge

and skills?

18. Did the TRANSFORM trainings package include a monitoring and evaluation (M&E) component?

a. If so, what did the M&E efforts include?

b. What was being measured?

c. What types of indicators were used?

d. Who was responsible for monitoring and evaluating the trainings?


19. How did your organization ensure that TRANSFORM trainees could continue using their new

knowledge and skills in their work?

a. What kind of support do you think trainees at different levels need to continue using their

knowledge and skills?



b. What are the major barriers to ensuring that staff at the national, province/district and

community levels continue applying their newly acquired knowledge and skills in

practice?

20. Has UNICEF allocated more resources towards rolling more TRANSFORM trainings in the

future?

21. Thinking back, how would you have planned and designed the trainings now, given your

experience and knowledge?

a. What in particular would you improve?

b. Why do you think these changes would have helped?




FGD: Beneficiaries of the SCT programme


First, I would like to ask you some general questions about the Social Cash Transfer programme

1. Please introduce yourself and tell us how long you have lived in this community.

2. How long have you been receiving cash transfers?

3. Is there anything you don’t like about how the cash transfer programme is delivered to your

community?

4. How do you receive your cash transfers?

a. Who are the people (e.g., CWACs, enumerators or others) that you worked with to register

and begin receiving cash transfers?


Thank you. Now I want to ask some more questions on the community sensitization meetings for the

SCT

5. Have you attended any community sensitization meetings for the SCT programme?

a. When was the last time you were invited to a meeting?

b. When was the last time you attended a meeting?

c. Who conducted the sensitization meetings?

d. What were the objectives of the community sensitization meetings? (probe for: if listing

was conducted, selection of the CWAC members, if people were screened by the doctor

and provided with a disability certificate)

e. Can you describe the topics covered during the community sensitization meetings? (Probe

for: objectives of the programme and eligibility criteria for beneficiaries)

f. How were the community sensitization meetings conducted?

g. Was the approach to the meeting appropriate given the needs of your community?

6. What were the reactions of the community members after the sensitization meetings?

7. Did you receive any documents or materials during your community meetings? If yes, what are

they?

8. Was it difficult to attend the community meetings? Or to understand what was being said? If yes,

please describe.

9. Do you have any recommendations to improve the community meetings?

III. ACTIVITIES

10. Did the community sensitization meetings you attended benefit you in any way? How so?

11. Are you satisfied with the way the sensitization was conducted? If no, why not?

a. How did the staff treat you?

b. What do you think the staff do best?

c. What do you think staff do not do well?

d. Would you feel more comfortable with being attended to by a person of your sex/gender?

If yes, can you explain why?

12. Did the community sensitization meetings you attended help you cooperate with CWAC members

and Enumerators?


Thank you, now I would like to ask you some questions about any changes you have observed since the

community sensitization meetings

13. Did you observe any changes following the community sensitization meetings? If so, what were the

changes?

14. Have your interactions with CWAC members and enumerators changed at all since the community

sensitization meetings? How so?

V. FUTURE PLANS/ ACTIONS



Finally, I want to ask questions related to the programme’s future plans.

15. Since you became a beneficiary of the SCT, have the staff from MCDSS and the CWACs and

enumerators continued to provide what was promised to you during the community sensitization

meetings and validation processes? (probe for: visit to their homes, payment every 2 months)

16. Have you noticed any changes in how cash transfers were delivered in the past 2 years?

17. Have you noticed any changes in how enumerators, CWACs, or others work with community

members over the past 2 years?

18. Are you aware of Form 1 and Form 2/ mobile technology used during targeting and selection of

SCT beneficiaries? What do you think about these forms/the mobile technology? Are they helpful?

Do they make the process smoother?

19. Do CWAC members face any challenges using the forms 1 and 2? If so, what are these challenges?

20. Do enumerators face any challenges using the Mobile Technology? If so, what are these

challenges?

21. Do you know if there are plans to continue conducting the community meetings and making

payments to beneficiaries?

22. Do you have any recommendations for how to improve the community sensitization meetings? If

yes, what are they?

23. Do you have any recommendations for how to improve your interactions with CWAC members and

enumerators? If yes, what are they?

24. Is there anything else you would like to mention before we end the interview?




FGD: Enumerator


First, I would like to ask you some general questions about the Social Cash Transfer

programme

1. How long have you been working with the SCT programme and what are your responsibilities

under the programme?

2. What was the purpose of the training you received on targeting/selecting/enrolling SCT

beneficiaries?


Thank you. Now I want to ask some more questions about the trainings under the scale up of Social

Cash Transfer programme.

3. What training did you receive on targeting/selecting/enrolling SCT beneficiaries?

a. Who trained you?


c. What were the objectives of the training?

d. Can you describe some of the key topics covered in the training? (Probe for: the use of

Mobile Technology (M-tech)/ form 3 to enumerate all the households on the list from the

CWACs)

e. Please describe the most important concepts you learned in the training.

f. Did you feel like you gained new knowledge and skills through those trainings? If so

what were they?

g. Did your responsibilities change after the training? (If he/she mentions enumeration of all

the households on the list probe: if he/ she felt adequately trained to carry out these

responsibilities).

h. How was the training conducted?

i. Were there any challenges to participating in this training?

j. Did you receive any training materials during your training? If yes, what are they? (probe

for: mobile or other ICT equipment)

k. How often do you use the materials and in what ways?

l. If no, would having access to training materials help you?

m. Were there any challenges in the implementation of this training? If yes, what were they?

n. Do you have any recommendations for how to improve the enumerator trainings on

beneficiary targeting/selection/enrollment?

4. Do you think the training for enumerators was relevant, given your role on the SCT? Was the

training relevant given available resources and the needs of your community?

III. ACTIVITIES

Next, I want to ask you some questions about the activities under the scale up of Social Cash Transfer

programme

5. What was the intended outcome of the trainings for enumerators on beneficiary


6. Do you think the training you received achieved its intended goals?

7. Did the training you received improve your ability to carry out your work as enumerators under

the SCT programme? If yes, how so?

8. Did the training you received equip you with skills you need to work effectively with DSWOs,

DWAC members, CWACs and ACCs?

9. Did anything make the training you attended more or less effective? Please describe.

10. Did anything make it hard for you to attend the training or understand the material you were

being trained on? If yes, please describe.






11. Have you noticed any changes in your knowledge/skills and ability to use technology/ software

(Mobile technology/MIS) since the training?

12. Since the training, have you noticed any changes in the way you go about your work as an

enumerator? Please describe.

13. Do you think the training you received has improved the selection and targeting of SCT

beneficiaries? Please describe.

14. Did the trainings lead to improved delivery of services to beneficiaries? If yes, please explain.


Next, I want to ask questions related to programme’s future plans and the utilization of the Mobile

Technology and form 3.

15. Have you continued to use the skills/technologies (Mobile technology) you learned from the

training?

a. How do you use the knowledge and technologies (Mobile technology) you learned

about during the training?

16. Do you use Mobile technology and Form 3 in decision making? If yes, for which decisions?

A. How does MIS data inform your decisions?

17. How would you compare the Mobile technology to the previous method of targeting and

selecting beneficiaries for the SCT programme? Is it better/worse? More time-consuming or less

so?

18. Do you or other enumerators face any challenges using the Mobile technology, MIS and the form

3? (Probe for challenges that enumerators faced in uploading information on the MIS web portal

using the Mobile technology (M-tech)

19. Did anything or anyone make it easier to roll out the Mobile technology? Please describe. (probe

for: existing structures)

20. Has anyone helped enumerators to keep using the knowledge and skills acquired through the

trainings? Who? How have they done this?


22. Have you had any follow-up questions since the training? Were you able to get answers to these

questions? Please describe.

23. Do you know whether there are plans to continue providing trainings to enumerators? Do you

think continued trainings are needed?

41. Do you have any recommendations for how to improve training for enumerators under the SCT

programme?

42. Is there anything else you would like to mention before we end the interview?




WASH: Capacity development in use of the Mobile to Web tool to improve

real-time monitoring, and in community sensitization for scale-up of the

Community-Led Total Sanitation intervention





WASH sector, specifically the scale-up of the use of the Mobile to Web tool to improve real-time

monitoring, and community sensitization for scale-up of the Community-Led Total Sanitation intervention

as part of the Zambia Sanitation and Hygiene Programme (ZSHP). The purpose of this evaluation study

is to explore the relevance, effectiveness, perceived impacts and sustainability of the HMISS intervention.

The purpose of this Key Informant Interview is to better understand how the scale up of the CLTS and the

scale-up of the M2W tool usage was implemented, the process of designing these CD interventions, and

whether the CD interventions led to any specific changes. In addition, we also want to capture the

engagement between stakeholders who were involved in this intervention, explore anticipated outcomes

and unanticipated interactions.




















NW, Washington, DC 20007, USA or Claude Kasonka of the American Institutes for Research Zambia

Office, (Tel. +260.211.372778), Elunda II, Addis Ababa Roundabout, Regus office #115, Rhodes Park,

Lusaka, Zambia

KII with Ministry Officials at Institutional Level




1. Could you please tell us a bit about your position at [MoLG, MoH, MoCTA]?


2. What was the main aim/purpose of capacity development trainings in 2017 related to the usage of

Mobile to Web tool and real-time monitoring?

3. Can you please describe the trainings on Mobile to Web for government officials?

A. What were the goals of the training package?

B. Who was the trainings developed for? (central/provincial/district and facility level)


4. Was there a needs assessment when deciding who (which individuals/organizations) to target for

the training on using Mobile to Web tool and the real-time monitoring of the Community-Led

Total Sanitation (CLTS)?

a. If so, who conducted the needs assessment?

b. What did the needs assessment entail?

c. If not, do you think the capacity development trainings would have benefited from an

assessment?

5. What gaps in knowledge/skills existed prior to the trainings in 2017 among the different groups of

staff involved in working on Mobile to Web tool and the real-time monitoring of the Community-

Led Total Sanitation (CLTS)?

6. What (if any) challenges do employees (MoLG, MoH, MoCTA) face as they worked on the

Mobile to Web tool, before the trainings?


7. What capacities needed to be developed at national, provincial/district and community levels??

a. How were the provinces and districts selected?

b. Please describe the approach to selecting participants. Probe: rights-based, gender

8. What trainings did you and your colleagues at [MoLG, MoH, MoCTA] receive in 2017?

9. Who provided the training?

10. Can you describe your experience in the training?

a. What were the challenges in participating in the trainings?

b. What concepts did you learn in the training?

11. In what ways do you think the training content was gender-responsive? From the literature, we

know that women are usually the caretakers of the households and WASH practices can be led and

supported by women in their households.

a. Prompt: did the content of the training have components that targeted women for

community sensitization?

12. Did you receive any training materials during your training?

a. If yes, what types of materials were they?

b. Did you use them after trainings?

c. If no, would having access to training materials been helpful for you?

13. What would you recommend for improving the trainings for officials at your level?


14. In your observations, what types of changes happened in how M2W tool is operated and how real-

time of the CLTS happens after the trainings in 2017?

a. If so, which particular aspects of work changed? Please describe.

b. If not, why do you think there have not been any improvements?

c. What indicators have you used to measure progress in capacity development ?

15. In your observations, have there been changes in how the WASH sector and ZHSP since the

intervention?

a. If so, which particular aspects?



b. If not, why do you think there have not been any improvements?

VI. Future Plans [Sustainability]

16. Did the government share the costs of the trainings rolled out in 2017 in any way?

17. Did the government decide to invest any resources after the trainings implemented in 2017, into

knowledge/skill improvement of the staff who works with the M2W tool and the real-time

monitoring of the CLTS?

a. If so, what kinds of resources and investments?

b. How were the trainings expanded further?



and local levels?


differently?

a. What in particular could be improved?


21. What could UNICEF do better to support training and capacity building of SCT staff?




KII with UNICEF Officers at Institutional level:


1. Could you please tell us a bit about your position at UNICEF?

A. How long have you worked in this position?

2. What was the main aim/purpose of capacity development trainings in 2017 related to the usage of

Mobile to Web tool and real-time monitoring?

3. Can you please describe the trainings on Mobile to Web for government officials?

A. What were the goals of the training package?

B. Who was the trainings developed for (central/provincial/district/facility level)?

C. What kinds of capacities needed to be developed?



individuals/organizations) to target for the training on using Mobile to Web tool and the real-time

monitoring of the Community-Led Total Sanitation (CLTS)?

a. If so, who conducted the needs assessment?

b. What did the needs assessment entail?

c. If not, do you think the capacity development trainings would have benefited from an

assessment?


groups of staff involved in working on Mobile to Web tool and the real-time monitoring of the

Community-Led Total Sanitation (CLTS)?

6. What (if any) challenges did employees (MoLG, MoH, MoCTA) face as they worked on the

Mobile to Web tool, before the trainings?


7. How were the provinces and districts selected for the CD trainings?

8. During trainings on using the M2W and real-time monitoring of the CLTS:

a. What types of challenges did the implementers face (national, province/district,

community/facility)?

b. How were those challenges overcome?

9. How did UNICEF facilitate different training activities?

a. Did UNICEF provide any training materials?

b. If so, what types of materials?

10. How did training participants respond to the trainings?

11. How did UNICEF help account for any gender or inequality bias in the delivery of the trainings or

in the content of the trainings?

a. What steps were taken to ensure more women and disabled staff were also participating in

the trainings?

b. Were there any challenges related to the trainings being inaccessible to staff with

disabilities?

c. What was the gender balance among the recipients?

d. Did UNICEF have conversations with implementers to ensure that timings of trainings

were inclusive? (To ensure that trainings were not conducted after work hours, which may

prevent some people, especially women who may have domestic and child caring

responsibilities from actively participating?)




12. In your observations, what types of changes happened in how M2W tool is operated and how real-

time of the CLTS happens after the trainings in 2017?

a. If not, why do you think there have not been any improvements?

b. What indicators have you used to measure progress in capacity development?

13. Have there been changes in how the WASH sector and ZHSP improved since the intervention?

A. If so, which particular aspects?

B. If not, why do you think there have not been any improvements?


14. Did the government share the costs of the trainings with UNICEF when they were rolled out in

2017?

15. Did the government decide to invest any resources after the trainings implemented in 2017, into

knowledge/skill improvement of the staff who works with the M2W tool and the real-time

monitoring of the CLTS?

a. If so, what kinds of resources and investments?

b. How were the trainings expanded further?



and local levels?

18. How would you have asked implementers to design and deliver the trainings differently?

a. What in particular could be improved?


19. What could UNICEF do better to support training and capacity building of staff within WASH?





Respondent: FGD with Community Champions

Interviewer: Thank you for agreeing to participate in our study. We would like to begin our

conversation today with introductions and then I will ask you a series of questions about your work

or training(s) you received around the Mobile to web and CLTS.


1. Please tell us briefly about yourself and your role as a chief in this village.

a. How long have you served in this role?

b. Who do you mostly interact with on a day-to-day basis?

2. Are you familiar with the Mobile to web/ Real-time monitoring? Please describe the programme in

your own words.

3. What is your understanding of the purpose of community-led total sanitation (CLTS)

4. Please describe your involvement on the training(s)/ CD interventions that have taken place with your

community members


5. Please describe any CD/ trainings you received in M2W/RTM and the CLTS intervention?

a. When was the last time you received that training?

b. How long was the training overall? (in days/weeks)

c. How often do you receive such kinds of training(s)?

6. Please describe the CLTS training(s) you received?

a. How were you were trained to trigger villages on CLTS?

b. Who trained you?

c. What topics did you covered during the training(s)?


d. What materials were provided to you during your training(s)/ CD interventions?

e. In addition to earlier trainings, have you received additional training or resources?

f. Did you understand the concepts from the trainings? What, if anything, was difficult?

g. In your view did these training(s) achieve their intended purpose? Please describe.

h. Were the training methods appropriate to your needs as trainees? Please describe.

7. Could you describe the M2W/RTM training(s) you received?

a. Can you describe how you were trained on the M2W/RTM?

b. Who trained you?

c. What topics did you cover during the training(s)?

i) Please describe the most important concepts you learned in the training

d. What materials was provided to you during your training(s)/ CD interventions?

e. Did you understand the concepts from the trainings? What, if anything, was difficult?

f. How did you find the training(s) seeing that they involved technology?

g. What were the positive aspects of the training? What about negative aspects?

h. In your view did these training(s) achieve their intended purpose?

i. What do you think about the training method, were they appropriate to your needs as

trainees?

j. In addition to earlier trainings, have you received additional training or resources?

8. Are you aware of any needs assessment that was done prior to the trainings/ CD interventions you

received in M2W/RTM??

9. Do you know how you were selected to participate in the training(s)?

10. What changes would you make to the training(s)/CD in the following aspects?

a. The CLTS triggering process

b. The use of technology)



11. In your opinion, was the training appropriate for men and women? Please describe.

a. What was the gender balance among the recipients?

12. In what ways, if any, have the training(s)/ CD interventions strengthened your capacity to manage

data/information on sanitation and hygiene?

a. How have these training(s)/ CD interventions affected your community’s capacity to monitor

sanitation and hygiene?


13. What are some of the factors that facilitated training(s)/CD interventions? Probe on central training

location, availability of computers, phones and Internet connectivity, staff turnover

14. What are some of the factors that inhibited training(s)/CD interventions? Probe on central training


15. In your opinion, what practices or approaches if any, within M2W/RTM and CLTS CD

interventions proved to work consistently?


16. In your opinion, how have the trainings/ CD interventions helped develop your knowledge/skills

and capabilities to use technology/software and implement CLTS? How?

a. How have your tasks changed after the trainings?

b. How have you changed your strategies after the training?

17. Do you think the CD interventions training(s) improved your data capturing as a chiefdom?

18. Do you think the training(s) led to changes in behaviour or practices (such as timely submission of

reports, accurate data submission, implementing CLTS)?

19. Do you think the training(s) resulted in your chiefdoms improved delivery of specific services to

end users?

V. Future plans/ actions [Sustainability]

20. How do you use the skills and knowledge that you acquired through CD trainings?

21. How do you plan to continue using skills and knowledge obtained from the CD trainings? Please

provide examples?

22. How do you encourage others that they work with, to use those skills?

23. What are some of the issue that might undermine the continued use of the CD interventions?

24. Describe any plans in place to ensure the skills, knowledge, and technology built through the CD

intervention(s) are retained and passed on?

25. What suggestions do you have for improving the training(s)/CD intervention in the future?

Thank you for your time!

Respondent: KII with Chief







1. Please tell us briefly about yourself and your role as a chief in this village.

a. Who do you mostly interact with on a day-to-day basis?

2. Are you familiar with the Mobile to web/Real-time monitoring? Please describe the programme in your

own words.

3. What is your understanding of the purpose of community-led total sanitation (CLTS)?

4. Please describe your involvement in the CD intervention for community champions.


5. Please describe any CD/ trainings you received in M2W/RTM and the CLTS intervention.

a. How often do you receive such training(s)?

b. When was the last time you received that training?

6. Please describe the CLTS training(s) you have received

a. Can you describe how you were trained to trigger villages on CLTS?

i) Who trained you?

b. What topics did you cover during the training(s)?


c. What materials were provided to you for the intervention?

d. In addition to earlier trainings, have you received additional training or resources?

e. Did you have any challenges with the concepts from the trainings? Please describe.

f. Were the training methods appropriate to your needs? Please describe.

7. Could you describe the M2W/RTM training(s) you received?

a. Can you describe how you were trained on the M2W/RTM?

i) Who trained you?

b. Can you describe the topics that you cover during the training(s)?


c. What materials were provided to you during your training(s)/ CD interventions?

d. In addition to earlier trainings, have you received additional training or resources?

e. What were other training participants reactions after the trainings, do you think other people

understood the concepts from the trainings? How did you find the training(s) seeing that they

involved technology?

f. What were the positive aspects of the training? What about negative aspects? Why?

g. In your view did these training(s) achieve their intended purpose? Please describe.

h. Were the training methods appropriate to your needs as trainees? Please describe.

8. Are you aware of any needs assessment that was done prior to the trainings/ CD interventions you

received in M2W/RTM??

9. Describe how training(s)/CD interventions target individuals? Do you know how you were selected to

participate in the training(s)?

10. How appropriate were the CD training(s) considering the following aspects?

a. The CLTS triggering process

b. The use of technology)

11. In what ways, if any, have the training(s)/ CD interventions strengthened the capacity of community to

manage data/information on sanitation and hygiene?

a. How have these training(s)/ CD interventions affected your community’s capacity to monitor

sanitation and hygiene?


12. Do you think the intended outcomes of the training(s)/CD interventions have been achieved through

training(s)/CD interventions?



13. What are some of the factors that facilitated training(s)/CD interventions? Probe on central training


14. What are some of the factors that inhibited training(s)/CD interventions? Probe on central training


15. In your opinion, what practices or approaches if any, within M2W/ RTM and the CLTS CD

interventions proved to work consistently?


16. In your opinion, how have the trainings/ CD interventions helped develop your knowledge and

capabilities to use technology/software and implement CLTS? How do you think this training has

helped in developing the knowledge/skills and capabilities of other chiefs to use technology/software

and implement CLTS?

a. How have you changed your short-term tasks after the trainings? What about other training

participants?

b. How have you changed some of your long-term tasks and strategies after the training? What

about other training participants?

17. Has your work as a chief has changed since you went through the trainings? How?

18. In your opinion, have the CD interventions improved your data capturing as a chiefdom?

19. Do you think the training(s) led to changes in individual behaviour or practices (such as timely

submission of monthly reports, accurate data submission, implementing CLTS according to

guideline)? Are you and other individual who participated in the training(s) meeting their deliverables

with ease?

20. Do you think the CD interventions resulted in improved delivery of services to end users?

V. Future plans/ actions [Sustainability]

21. How do you use the skills and knowledge you acquired through CD intervention trainings?

22. How do you plan to continue using skills and knowledge obtained from the CD intervention? Please

provide examples.

23. How do you encourage others you work with to use those skills?

24. What are some of the issue that might undermine the continued use of the CD interventions?

25. Describe any plans in place to ensure the skills, knowledge, and technology built through the CD

intervention(s) are retained and passed on?

26. What suggestions do you have for improving the training(s)/CD intervention in the future?

Thank you for your time!



Respondent: KII with Provincial Officials from Ministries




I. Introduction

1. Could you please describe a bit about your institution and your position?

a. How long have you been with this institution?

2. Are you familiar with the Mobile to web/ Real-time monitoring and the DHIS2 system? Please describe

these systems in your own words.

3. What is your understanding of the purpose of community-led total sanitation (CLTS)?

II. Purpose and design [relevance]

4. Describe any capacity development intervention/ trainings you received on Real Time Monitoring

(RTM) using the Mobile to Web (M2W) platform. Describe the CLTS trainings you have received.

a. When last did you receive these training and how often do you receive such kinds of training(s)?

5. What was the aim of the CD training(s)? In your view did these training(s) achieve their intended

purpose?

6. Please describe the most important concepts you learned in the training on M2W/RTM and CLTS.

a. What were the positive aspects of the training? What about negative aspects? Why?

7. Are you aware of any needs assessment that was done prior to the trainings you received in

M2W/RTM and CLTS?

a. If not, do you know how the training content was selected?

b. As a facilitator, did you conduct any needs assessment prior to the trainings you facilitated in

M2W/RTM and CLTS?

8. Describe how training(s)/CD interventions target individuals and organizations?

a. What was the selection criteria of the facilitators and trainees?

i. How did the selection criteria account for gender balance?

b. Do you think all the relevant institutions and individuals were involved?

9. How appropriate were these training(s)/CD considering the cultural, political and economic context in

your province?

a. In your own view, were the CD trainings appropriate to your needs?

b. As a facilitator/trainer, were the training(s)/ CD methods appropriate to the needs of your

trainees (community members i.e. community champions, headmen)?

10. In what ways, if any, have the training(s) strengthened the capacity of provincial, district and

community structures to manage data/information on sanitation and hygiene/CLTS?

III. Activities

11. Let’s talk a bit about how the training was conducted?

a. What materials was provided to you during your training(s)/ CD interventions?

b. As a facilitator, what materials did you provided to the trainees during your training(s)/ CD

interventions? Were the materials adequate?

c. Was the training delivered by an education professional? Was he or she knowledgeable about

the training topic?

d. Was the training conducted in a timely manner? Was the training location central convenient

and considerate for at least every participant?

i. Were there any gender-related challenges related to training?

12. On average, how long were the trainings? What has been the frequency of the training(s)?



13. What are some of the factors that facilitated or inhibited training(s)/CD interventions attaining their

desired outcomes? Probe on central training location, availability of computers, phones and Internet

connectivity, staff turnover

14. In your opinion, what practices or approaches if any, within CD interventions proved to work

consistently? What practices or approaches do you think failed to work properly?

15. How would you describe the cost effectiveness of the form of the training(s)/ CD interventions used

M2W/RTM training?

IV. Changes

16. How has the training/CD interventions affected you and other participating individuals and

organisation on their knowledge/skills and capabilities to use technology/software?

a. Do you think capacity has been built for individuals and organisations to manage the

M2W/RTM and CLTS?

b. Describe any improvement on your tasks since receiving the CD trainings.

c. How did the CD training improved your monitoring of M2W/RTM and CLTS activities?

17. How do you think CD trainings changed management of sanitation and hygiene data?

18. Describe how the CD interventions has resulted in individuals, organizations and institutions

improved delivery of specific services to end users?

19. How would you describe the accuracy, completeness and timeliness of the data on sanitation and

hygiene since the CD intervention, if at all?

V. Future plan/actions

20. Describe if any, mechanisms in place to ensure continued practice or use of the knowledge, skills and

technology developed from the CD intervention on M2W/RTM and CLTS?

21. What are the factors that might undermine the continued use of the capacities developed on M2W/RTM

and CLTS at individual and institutional level?

22. Are you aware of any plans in place to continuously conduct refresher CD interventions on

M2W/RTM and CLTS?

23. Has your institution allocated resources or made efforts to continue or begin implementing similar

training(s)/CD interventions on M2W/RTM and CLTS? What about other institutions?

24. To your knowledge, were the CD interventions in their design conducted is such a manner that could

allow for replicability by government?

a. What do you think could undermine the replicability of such forms CD intervention by the

government?

25. Would you like to offer suggestions for improving similar training(s)/CD intervention in the future?



Respondent: KIIs with Environmental Health Technicians

Interviewer: We would like to begin our conversation today with introductions and then I will ask

you a series of questions about your work or training(s) you received around the Mobile to

web/Real Time Monitoring and CLTS.


1. Please tell us briefly about your role as an Environmental Health Technician.


b. Who do you mostly interact with on a day-to-day basis?

2. Could you please provide a short description of the Mobile to web/ Real-time monitoring and the

DHIS2 system?

3. Could you provide a description of the purpose of community-led total sanitation (CLTS)?

II. Purpose and design [Relevance]

4. Please describe any capacity development intervention/ trainings you received on Real Time

Monitoring (RTM) using the Mobile to Web (M2W) platform.

a. When was the last time you received the training?

b. How long was the training overall? (in days /weeks)


5. Please describe any capacity development intervention/ trainings you received on Community Led

Total Sanitation (CLTS).

a. When was the last time you received the training?

b. How long was the training overall? (in days /weeks)


6. What was the aim of the CD/ training(s)? In your view did these training(s) achieve their intended

purpose?

7. Please describe the most important concepts you learned in the training on M2W/RTM and CLTS.

a. What were the positive aspects of the training? What about negative aspects? Please describe.

8. Are you aware of any needs assessment that was done prior to the trainings you received in

M2W/RTM and CLTS?

a. If not, do you know how the training content was selected?

9. Describe how training(s)/CD interventions target individuals and organizations?

a. Do you think all the relevant individuals were in the community were involved?

b. What steps were taken to ensure more women and disabled staff were also participating in

the trainings?

10. How appropriate were these training(s)/CD considering the cultural, political and economic context

in your district?

a. Were the training(s)/ CD methods appropriate to your needs as trainees/facilitators?

b. As a facilitator/trainer, were the training(s)/ CD methods appropriate to the needs of your

trainees (community members i.e. community champions, headmen)?

c. In what ways do you think the training content was gender-responsive, if at all?

11. How have these training(s)/ CD interventions affected the province’s capacity to manage

information related to sanitation and hygiene/CLTS?


12. Please tell me a bit about how the training was conducted?

a. What materials was provided to you during your training(s)/ CD interventions?

b. As a facilitator, what materials did you provided to the trainees during your training(s)/ CD

interventions? Were the materials adequate?



c. Was the training delivered by an education professional? Was he or she knowledgeable


d. Was the training conducted in a timely manner? Was the training location central

convenient and considerate for at least every participant?

e. Were there any gender-related challenges that prevented people from participating?

f. Did the gender of the trainer affect your experience? Why or why not?

13. What factors facilitated training(s)/CD interventions? Probe on central training location,

availability of computers, phones and Internet connectivity, staff turnover

14. What factors inhibited training(s)/CD interventions? Probe on central training location,

availability of computers, phones and Internet connectivity, staff turnover

15. What practices or approaches if any, within CD interventions proved to work consistently? What

practices or approaches do you think failed to work properly?


16. Do you feel you have been adequately prepared to manage the M2W/RTM and CLTS?

a. Describe how the CD interventions has improved your monitoring of M2W/RTM and

CLTS related activities.

17. How do you think the CD trainings affected the governance and management of sanitation and

hygiene data?

18. Describe how the CD trainings resulted in individuals, organizations and institutions improved

delivery of specific services to end users.

19. How would you describe the accuracy, completeness and timeliness of the data on sanitation and

hygiene since the CD trainings?

V. Future plans/actions [Sustainability]

20. Describe any plans you have to continue practicing or using the knowledge, skills and technology

developed from the CD intervention on M2W/RTM and CLTS?

21. What factors that might undermine the continued use of the capacities developed on M2W/RTM

and CLTS at individual and institutional level?

22. Are you aware of any plans in place to continuously conduct refresher CD interventions on

M2W/RTM and CLTS?

23. Has your institution allocated resources or made efforts to continue or begin implementing

similar training(s)/CD interventions on M2W/RTM and CLTS? What about other institutions?

24. To your knowledge, were the CD interventions in their design conducted is such a manner that

could allow for replicability by government?

a. What do you think could undermine the replicability of CD trainings?

25. Would you like to offer suggestions for improving similar training(s)/CD intervention in the

future?



KII with WASH CD Intervention Implementers


1. Could you please tell us a bit about your role at Akros?



2. Could you tell us a bit about the capacity development trainings you provided to government

officials on the usage of Mobile to Web (M2W) tool for real-time monitoring of the Community-

Led Total Sanitation (CLTS)?

A. What were the goals of the trainings?

B. Who was the trainings developed for (central/provincial/district/facility-level)?

C. What kinds of capacities did you aim to improve through the trainings?

3. When designing the training package, how did Akros decide what the existing gaps in knowledge

and skills were?

A. Was there an assessment of existing needs and knowledge/skills?

B. If so, what did this assessment entail?

C. Who implemented the assessment?

D. What kinds of gaps were found?

E. If not, how do you think the training on M2W tool and the real-time monitoring of the

CLTS would have benefited from an assessment?

4. Please describe your experience in working with UNICEF Zambia staff to design and implement

the trainings.

5. Please describe your experience in working with ministry staff to design and implement the

trainings?

6. How would you describe the cooperation between UNICEF Zambia and the ministries with your

organization, when discussing the purpose of trainings, their scope and scale?

III. Activities [Relevance]

7. Now we would like to hear a little bit about trainings at the national level:

a. What was the primary goal of trainings for the employees of MoLG, MoH and MoCT? Did

this vary by level/type of employee?




e. What types of challenges did you encounter in this training?


a. What was the primary goal of trainings for District Community Medical Officers?



d. What types of challenges did you encounter in this training?

9. Trainings at the community/facility level:

a. What was the primary goal of trainings for chiefs, village headmen and headwomen,

community champions (volunteers) and members of Sanitation Action Groups?



d. What types of challenges did you encounter working with trainees at this level?

10. How did you ensure that the trainings were understandable for participants?



11. What steps were taken to ensure that there was gender balance among the recipients of the

trainings?

a. Prompt: Did UNICEF or the ministry require that recipients included be at least 50%

women?

12. In what ways do you think the training content was gender-responsive? Prompt: Was the content of

the trainings targeted at women?

13. How did you ensure that the trainings were conveniently scheduled for participants?

IV. Perceived changes [Perceived impact]

14. How do you think the trainings changed the capacities of trainees, if at all?

15. At the Ministry level, could you describe the staff’s response to trainings?

a. How did the MoLG, MoH and MoCT staff change their work after the trainings?

16. At the province/district level, could you describe how District Community Medical Office CC)

respond to trainings?

a. How did they change their work after the trainings?

17. At the community/facility level, can you describe how chiefs, village headmen and headwomen,

community champions (volunteers) and members of Sanitation Action Groups respond to trainings?

a. How did they change their work after the trainings?

18. How did Akros gauge whether trainees were gaining new knowledge and skills?

19. Did the trainings include a monitoring and evaluation (M&E) component?

a. What was being measured?

b. What types of indicators were used?

c. Who was responsible for monitoring and evaluating the trainings?


20. Were there any efforts to ensure that trainees continue using new knowledge and skills in their

work? Please describe.

a. What kind of support do you think trainees at different levels need to continue using their

knowledge and skills?

b. What barriers exist to ensuring that staff at national, province/district and community levels

continue applying their knowledge and skills in practice?

21. Has UNICEF allocated more resources towards rolling more trainings on the usage of M2W tool

and real-time monitoring of the CLTS in the future?

22. How would you have planned and designed the trainings now, given your experience and

knowledge?

a. What in particular would you improve?





Respondent: FGD with Community Beneficiaries of the CLTS



or training(s) you received around CLTS.

I. Introduction

1. Please describe in your own words what were the main aims of the community sensitization

meetings you participated in?

2. What would you say were the strengths of the CLTS sensitization meeting?

3. What would you say were the weaknesses of the CLTS sensitization meeting?

II. Purpose and design

4. Did you participate in the CLTS “community triggering” meetings? Please describe.

a. What topics were covered during the community sensitization meetings?

b. Who conducted the community sensitization meetings and how?

6. What was your reaction to the sensitization meetings?

a. What were the positive aspects of the community sensitization meetings?

b. What were the negative aspects of the community sensitization meetings?

7. Have you had other any other meetings or received any additional resources?

III. Activities

6. How many community sensitization meetings have you had on CLTS?

a. Please describe how you knew about the meeting on CLTS?

7. What materials were used in the sensitization meetings on CLTS?

a. Were these materials left you to keep? Were they adequate for everyone?

8. Please tell us about the presenters of the sensitization meetings?

a. Did you understand their explanations on the topics?

b. Was the gender/sex of the trainer or community champion the same as yours?

i. Were you comfortable in asking specific questions related to menstrual hygiene

to female community champions?

9. Did the sensitization meeting have include women, disabled, or elderly people?

a. Were there any challenges that prevented you from participating? Prompt: hours, distance

IV. Changes

6. Please describe how you have interacted with community champion/village head/chief on

WASH-related topics.

7. How are you practicing what you learned in the meeting? Please provide examples.

8. Have you observed any changes in behavior by your neighbours or other community members

after the sensitization? Please explain.

V. Future plans/actions

9. Do you have any recommendations to improve community sensitization meetings?

10. Is there anything else you like to mention before we end the interview?

Thank you



Education: Capacity development in the Teaching at the Right Level

approach for scale-up of the Catch Up intervention





education sector, specifically the Catch Up intervention—one of the latest CD interventions supported by

UNICEF—and which uses the Teaching at the Right Level (TaRL) approach. The purpose of this study is

to explore the relevance, effectiveness, perceived impacts and sustainability of the Catch Up intervention.

The purpose of this Key Informant Interview is to better understand how the Catch Up intervention was

implemented, the process of designing the Catch Up Intervention, and whether this intervention led to any

specific changes. In addition, we also want to capture the engagement between stakeholders who were

involved in this intervention, explore anticipated outcomes and unanticipated interactions.























National Level KII—Catch Up/Teaching at the Right Level (TaRL)

I. Introduction & Background

1. Please tell me a bit about your work with [UNICEF/MoGE] on education.

2. What are the goals for Catch Up/TaRL? What is the initiative trying to achieve?

a. Who is implementing Catch Up/TaRL?

II. Purpose & Design

3. How did you determine that teachers (including head teachers) and MoGE staff needed

support to improve teachers’ skills to teach students at their individual levels?

a. Was any type of needs assessment conducted? If so, who conducted the needs

assessment? What capacity gaps were identified? Probe: teachers, MoGE staff

b. How was training approach for teachers and MoGE staff developed? For example,

what resources were used to develop the training plans?

4. Is there a conceptual framework or theory of change underlying the CD support? If yes,

please describe.

5. Is the Catch Up/TaRL training appropriate given the constraints of many Zambian

schools (e.g., resources, class size)? Please describe.

6. Are there any other organizations/programmes providing training similar to Catch

Up/TaRL? Please describe.

III. Activities

7. What specific CD supports (training, mentoring/coaching, provision of information, etc.)

are being provided for teachers (including head teachers) and MoGE staff?

a. What was the purpose of the CD support for teachers and MoGE staff?

b. What kinds of capacities needed to be developed in teachers/MoGE staff?

c. How was the CD support delivered? (probe for modality, frequency, follow-up,

differences at each level)

d. Who trained MoGE staff and teachers?

8. Did teachers/MoGE staff receive any training or support to use technology related to

teaching at the right level? Please describe.

9. Was training at the central level? Or was a training of trainers (ToT) approach used?

Please describe. Probe for training of master trainers at the district level.

10. Can you tell me a little about what worked well / didn’t work well with the ToT

approach? Probe for information dilution, etc.

11. What steps were taken to ensure more women and disabled staff were also participating

in the trainings?


b. How were any presence of gender bias accounted for in the delivery of the

trainings and in the content of the trainings?

c. Were there any gender-related challenges that prevented you from participating?

Prompt: Were trainings conducted during after work hours, which may have

prevented you from actively participating due to domestic and child caring

responsibilities?



12. How, if at all, did the training of teachers/MoGE staff in different

provinces/districts/schools within Zambia vary? Was it adapted to meet the needs of

different locations (e.g., rural schools, lower performing schools, larger schools)? How?

IV. Perceived Changes

13. Did teachers or MoGE staff provide feedback on Catch Up/TaRL trainings? How would

you characterize this feedback?

14. Is there any monitoring and evaluation (M&E) of the CD support to teachers and staff?

a. If yes, can you tell me about the M&E plan?

b. What CD activities are monitored/evaluated? Is feedback incorporated in

subsequent training cycles?

c. What indicators are you using to measure capacity development?

d. How is M&E data collected?

e. Do MCDSS staff do any self-assessments?

15. Have you noticed any changes in teachers’ capacity (knowledge, skills, behavior) and/or

MoGE staff resulting from these trainings on Catch Up/TaRL? Please describe.

16. Do you have any data (M&E data, evaluations/other studies, self-assessments, etc.) to

support your belief that capacities improved/did not improve? Please describe.

V. Future Plans

17. Are teachers and MoGE staff held accountable for the training/capacity building support

they’ve received? If yes, how so?

a. Is their performance monitored and assessed following trainings?

18. Are teachers/MoGE staff they held accountable for fulfilling their roles and

responsibilities? If yes, how so?

19. What further CD (or additional follow-ups to Catch Up/TaRL) do you think is needed for

teachers/MoGE staff to teach students effectively? Please describe.

20. Are there plans to continue/modify the CD for teachers/MoGE staff?

a. If so, how will this look?

b. Who will support the effort financially?

c. How will it be monitored/evaluated?

21. If no additional CD activities are anticipated, are there plans to periodically assess the

capacities of teachers and MoGE staff? Please describe.

Conclusion

22. How do you think the CD activities we’ve discussed could be improved to better build

the capacities of teachers/MoGE staff?

23. Is there anything you’d like to add that we haven’t covered in the interview?

Thank respondent for their time.



Child protection: Capacity development to improve integrated case

management to strengthen the alternative care system





Child Protection sector, specifically the CD of strengthening case management (CM) for child and family

welfare and strengthening the alternative care system. The purpose of this study is to explore the

relevance, effectiveness, perceived impacts and sustainability of this CD intervention.

The purpose of this Key Informant Interview is to better understand how this implementation was

implemented, the process of designing the intervention, and whether the CD intervention led to any

specific changes. In addition, we also want to capture the engagement between stakeholders who were

involved in this intervention, explore anticipated outcomes and unanticipated interactions.























National Level KII—CM & Alternative Care

I. Introduction & Background

1. Please tell me a bit about your work with [UNICEF/MCDSS] on child protection.

2. What are you trying to achieve through strengthening CM?

3. Can you also tell me a bit about the goals of the alternative care framework (ACF)?

II. Purpose & Design of Capacity Development

4. How did you determine that MCDSS staff and child protection officers needed support to

implement CM and the alternative care framework?

a. Was there any type of needs assessment conducted? If so, who conducted the

assessment? What capacity gaps were identified?

b. How was the CD approach developed? For example, what resources were used to

develop the training plans?

5. Is there a conceptual framework or theory of change underlying the CD support being

provided to MCDSS staff? If yes, please describe.

III. Capacity Development Activities

6. What specific CD supports (training, mentoring/coaching, provision of information, etc.)

are being provided for MCDSS staff working on CM and the alternative care framework?

a. What was the purpose of the CD support?

b. What kinds of capacities needed to be developed and among whom?

c. How was the CD support delivered? (modality, frequency, follow-up)

7. Did MCDSS staff receive any training or support to use technology related to

CM/alternative care framework? Please describe.

8. If training was provided, was it all done at the central level? Or was a training of trainers

(ToT) approach used? Please tell me a little about this.

9. Can you tell me a little about what worked well / didn’t work well with the ToT

approach? Probe for information dilution, etc.

10. What steps were taken to ensure more women and disabled staff were also participating

in the trainings?


11. How was any presence of gender bias accounted for in the delivery of the trainings and in

the content of the trainings?

a. Were there any gender-related challenges that prevented you from participating?

Prompt: Were trainings conducted during after work hours, which may have

prevented you from actively participating due to domestic and child caring

responsibilities?

IV. Perceived Changes

12. Is there any monitoring and evaluation (M&E) of the CD support to MCDSS?

a. If yes, can you tell me a little about the M&E plan?

b. What CD activities are monitored/evaluated?

c. What indicators are you using to measure capacity development?

d. How is M&E data collected?

e. Do MCDSS staff do any self-assessments?



13. Have you noticed any changes in MCDSS staff capacity resulting from these CD efforts?

Please describe.

a. Whose capacity has changed?

b. What skills/knowledge changed?

14. Do you have any data (M&E data, evaluations/other studies, self-assessments, etc.) to

support your belief that capacities improved/did not improve? Please describe.

V. Future Plans

15. Are MCDSS staff held accountable for the training/capacity building support they’ve

received? If yes, how so?

a. Is MCDSS staff performance assessed?

b. Are their skills/knowledge otherwise monitored?

16. Are MCDSS held accountable for fulfilling their roles and responsibilities? If yes, how?

17. What further CD do you think is needed for MCDSS to use CM to strengthen the

alternative care framework? In other words, what skills/knowledge are still missing? At

what levels?

18. Are there plans to continue/modify the CD for MCDSS staff?

a. If so, how will this look?

b. Who will support the effort financially?

c. How will it be monitored/evaluated?

19. If no additional CD activities are anticipated, are there plans to periodically assess the

capacities of MCDSS staff at various levels to deliver CM using the alternative care

framework? How so?

Conclusion

20. How do you think the CD activities we’ve just discussed could have been improved to

better build the capacities of MCDSS staff? Probe in terms of relevance and delivery.

21. Is there anything you’d like to add?

Thank respondent for their time.


AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG C-1

Annex C. Evaluation Matrix

Relevance

Q1. To what extent do the intended outcomes and the relevant outputs address the national priorities as indicated in the National

Strategies that culminate into the realization of vision 2030.

1. CD Intervention 2. Indicators 3. Data source

All

• Extent to which CD interventions addressed

Vision 2030 goals for economic and human

development in their approaches

• Extent to which implementers collaborated

with Zambian government to design CD

interventions

• Extent to which CD interventions were

designed to fill gaps identified in each sector

Document review:

• UNICEF’s global and regional strategies by sector

• Background information on CD activities

• CD intervention training materials

Primary data collection:

• Institutional/system-level KIIs

Q2. To what intent are the key strategies in capacity initiatives aligned with UNICEF’s Global/Regional priorities.

Q3. Has UNICEF been able to adapt to the changing contexts- social, political and economical to address capacity/ systems

strengthening needs in the Country?

Q4. Has investment in capacity and systems strengthening across the Programmes been relevant to the needs of Children, Women

and the other marginalized groups?

All • Whether implementers conducted a needs

assessment ahead of the CD intervention

• Extent to which implementers used HCD

approach in design

• Extent to which CD interventions

incorporated equitable, gender, and human-

rights sensitive approaches

• Extent to which training materials and

content were adapted to account for

contextual factors

• Extent to which staff and community

member training participants’ feedback was

incorporated into trainings

Document review:

• Background information on CD activities

• CD intervention training materials


• Institutional/system level KIIs

• Province/district level KIIs

• Community level KIIs and FGDs



Effectiveness

Q5. To what extent have outcomes been achieved, are there any additional outcome(s) being achieved beyond the intended

outcome? Did the capacity building/system strengthening interventions lead to major unexpected results—both positive and

negative?

Q6. How have the results, both at outcome and output benefited men and women? Have they benefitted equally

Q7. Is the current set of indicators, both outcome and output indicators, effective in informing the progress made towards the

outcomes? If not, what indicators should be used?


All • See table of expected outcomes by

intervention and level

• Extent to which program outputs logically led

to desired outcomes

• Perceived benefits to men and women





Q8. Were the assumptions underlying the programme intervention strategy correct? Which factors, internal and external to the

programme strategies, explain the extent to which results have been achieved?

Q9. What were the facilitating and constraining factors? What are the challenges to achieving the outcomes?

Q10. What are consistent patterns and good practices across capacity building and system strengthening initiatives?

All • Extent to which assumptions underlying

programme intervention were based on needs

assessment

• Extent to which assumptions were vetted by

community-level participants

• Factors and/or practices perceived to

facilitate or hinder intervention’s ability to

achieve outcomes







Efficiency

Q11. Were the resources and inputs converted to outputs in a timely and cost-effective manner? Did ZCO use resources in the most

economical manner to achieve expected equity-focused results? If so how?

Q12. Would it have been possible to achieve the same results at lower costs? If so how? What alternative models exist to achieve the

results at the lower costs?

Q13. What were the most important cost drivers in the delivery of the programme, were costs contained without compromising

results? If so how?

Q14. Were the programme strategies implemented in a timely manner? If not why?


All • Timeliness of financial resources distributed to

province/district/community levels

• Extent of implementers’ (at all levels) access to sufficient

resources to implement the programme as planned

• Fidelity of financial resources distributed and to

maximize benefits to programme beneficiaries

• Efforts to reduce cost through cost-sharing, partnerships,

and synergies with other programmes

• Extent to which efforts to reduce costs negatively

impacted programme delivery

• Extent to which CD interventions were delivered on

schedule and within planned timeline



• Province and district-level KIIs

Q15. Has there been any duplication of efforts among UNICEF’s own interventions and interventions delivered by other

organizations or entities in contributing to the outcomes?

Q16. How well did the programme coordinate with other, similar programme strategies within the Country Office for synergy and

in order to avoid overlaps/duplication?

All • Extent of collaboration with similar programmes to

harmonize and avoid duplication

• Clarity of strategy to maximize collaboration with

partners

• Extent to which beneficiaries and staff perceive added

value of intervention





Sustainability

Q17. How strong is the level of ownership of the results by the relevant government entities and other stakeholders?

Q18. What is the level of capacity and commitment from the Government and other stakeholders to ensure sustainability of the

results achieved?

Q19. What could be done to strengthen sustainability?

All

• Extent to which government allocated additional resources

toward strategies to continue capacity building

• Extent to which donors allocated additional resources to continue

CD interventions

• Extent of continued cross-sector collaboration on aspects of CD

interventions

• Extent to which government entities and other stakeholders use

research to inform decisions about adapting, continuing, or

scaling CD interventions

• Extent to which plans to ensure sustainability were built into

programme design, Theory of Change

• Level of refresher trainings, ongoing knowledge management,

and continued support from master trainers







Perceived Impacts

Q20. Have the capacity building strategies so far contributed or is likely to contribute to medium to long-term social, economic,

technical, changes for individuals, communities, and institutions?


All • Prevalence of perceived changes to medium to long-

term social, economic, and technical capacities

• Types and extent of changes in national and local

strategies to target vulnerable populations

• Extent to which data from CD interventions was

incorporated into system-level data use

• Extent of government plans to continue supporting

CD intervention goals in the medium to long term




Q21. What positive/negative, intended or unintended outcomes have the programmes strategies for systems strengthening

contributed to?

All • Perceived positive outcomes according to

implementers and beneficiaries

• Perceived negative outcomes according to

implementers and beneficiaries

• Perceived positive or negative externalities from CD

programmes






Equity

Q23. Did the programme interventions reach the worse off individuals or communities like it did for the other groups?

Q24. What key barriers-political, economic, social hindered all the affected populations, communities and institutions?


All • Extent to which interventions targeted high

need participants (including low income,

female)

• Extent to which interventions targeted areas

where equity levels are lower

• Extent to which targeting procedures were

followed for each intervention

• Presence or absence of barriers (political,

social, economic) to participating in or

benefiting from CD intervention





AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG D-1

Annex D. Program Summaries

Health: Capacity development in use of the Health Management Information

System and the District Health Information System

In 2013, the Millennium Development Goal Initiative (MDGi) on Accelerating Progress towards

Maternal, Neonatal and Child Morbidity and Mortality Reduction in Zambia was launched

jointly by the Government of the Republic of Zambia (GRZ), the European Union and the United

Nations to improve the availability and quality of health and nutrition services in two provinces

of Zambia. As part of this initiative and to help meet the goal of the MDGi, the Health

Management Information System Strengthening (HMISS) CD intervention was implemented by

UNICEF Zambia to enhance the functionality and effectiveness of the HMIS to provide quality

information based on the needs of end users. Ultimately, this intervention aimed to contribute to

improvements in maternal, newborn, child and adolescent survival, health and nutrition.

The intervention was implemented for two years, from 2016 to 2018 and was facilitated by an

external organization, Akros Inc. and its partner, the University of Oslo. The HMISS CD

intervention was implemented in 11 districts across Lusaka province (Lusaka, Chilanga, Kafue,

Rufunsa, and Chongwe districts) and Copperbelt province (Masaiti, Luanshya, Ndola, Kitwe,

Mufulira, and Chingola districts) of Zambia.

The key objectives of the HMISS CD intervention were to build the human resources capacities

to collect, manage and use data at all levels in order to improve health service delivery, and to

enable staff to make better health-related decisions based on the collected data. The second

objective was to enhance the functionality of the District Health Information System (DHIS2) to

optimize data quality and use, and also to enhance the compatibility and interoperability of the

DHIS2 with related HMIS subsystems. The second objective also involved ensuring that there

were adequate ICT equipment and to enhance its sustainable operationalization. The third

objective was to increase demand for and use of health and nutrition data by MoH staff to inform

health sector planning and management decisions.

The ultimate desired outcome of the HMISS CD intervention was to create a culture for data

sharing and exchange and to create a system environment, facilitated by an improved HMIS

functionality, and which was conducive for data use and decision-making by stakeholders at all

levels of government to improve health service delivery.

Below, the capacity development activities implemented per level is briefly described:

i. Institutional/System level: At the national level, national health officers from MoH received

on-the-job training on ways to clean the HMIS and to provide backend system support. Akros

also organized activities, meetings and workshops for relevant MoH officers to be able to

enhance the functionality, compatibility and interoperability of the DHIS2. Further, as the

MoH’s M & E unit was the custodians of the MOH DHIS2 system, M & E officers provided

guidance to deliver the CD intervention and to ensure its sustainability.

ii. Provincial and District Levels: A Training of Trainers (ToT) approach was used where Akros

trained provincial level Senior Health Information Officers (SHIOs) and district level District



Health Information Officers (DHIOs) to train facility level staff and community-level health

workers on DHIS2 data entry. The SHIOs and the DHIOs at the MDGi districts were trained in

data entry, ways to scrutinize data, on ways to troubleshoot the system, to conduct real-time

monitoring, quality data-checks, data analysis and on how to utilize the data. This training

allowed DHIOs to be part of “mentorship visits” to facilities.

iii. Community/Facility Level: At the community and facility level, staff such as EHTs and

CHWs were trained by DHIOs in data entry, real monitoring of data, data quality checks and

data use. The mentorship visits from DHIOs and supervision visits from the SHIOs helped

enhance the capacity of facility level staff to use data for decision making. These visits were

implemented to train facility level staff on HMIS data collection tools (registers, tally sheets and

summary sheets). community health workers were trained on DHIS2 for real-time monitoring,

data quality checks and timely data use.

Across all three levels, ICT equipment was distributed with a few at the national level.

Table D1 presents a summary of the HMISS CD intervention’s activities, audience, and expected

outcomes.

Table D1. Summary of HMISS CD Intervention and Outcomes

CD Interventions Activities by Level Target Audience Expected Outcomes

Health:

Health

Management

Information

System

Strengthening

Geographic

scope:

Two provinces:

Lusaka and

Copperbelt

11 districts

Institutional/system:

On-the-job training on

cleaning HIMS data,

and system support

• Central health

officers, MoH

• Officers from

the M&E Unit

• Improved ability of relevant MoH staff to

provide data-based maternal, newborn,

child and adolescent survival, health and

nutrition services.

• Improved capacity of to manage and use

data at all levels to improve health service

delivery.

• Enhanced functionality and effectiveness of

the HMIS to optimize data quality and use,

and also to enhance the compatibility and

interoperability of the DHIS2 with related

HMIS subsystems.

• Improved knowledge and capacity on ways

to clean HMIS data and to provide system-

level support.

Province/district

level:

Training of trainers

(ToT) on data entry and

ways to scrutinize data,

troubleshoot the system

and conduct real-time

monitoring and data

analysis

• Province-level

senior health

information

officers

• District health

information

officers

• Increase in the amount of ICT equipment

and enhanced operationalization of the ICT

equipment by relevant MoH staff.

• Improved knowledge and capacity on ways

to conduct data entry, ways to scrutinize

data, troubleshoot the system and conduct

real-time monitoring and data analysis.

• Relevant SHIOs and DHIOS can act as

mentors for facility-level MoH staff.




Community level:

Trainings on DHIS2

data entry and using

data for decision-

making

• Facility-level

staff

• Community-

level health

workers

• EHTs

• Facility and community-level staff such as

EHTs have improved knowledge and

capacity to conduct data entry, on HMIS data

collection tools, in their usage of data and

increased ability to use data for decision-

making.

• CHWs have improved capacity to use DHIS2

for real-time monitoring, conduct data quality

checks and conduct timely data use.



Social protection: Capacity development in beneficiary targeting and selection

for scale-up of the SCT programme

In 2014, the Government of Zambia (GRZ) through the Ministry of Community Development

and Social Services (MCDSS) launched the National Social Protection Policy (NSPP) to protect

and promote the livelihoods and welfare of people suffering from critical levels of poverty and

deprivation and/or who are vulnerable to risks and shocks. In addition, to support the NSPP, the

United Nations Joint Programme on Social Protection (UNJPSP) was established by a joint

partnership of the UN and its agencies, the GRZ and representative governments of five donor

countries.

Through the UNJPSP, the UN provided technical support to the GRZ in implementing the

broader NSPP. The UNJPSP was funded under an agreement between 4 bilateral donors and

GRZ. The three objectives of the Joint Financing Agreement are to: (i) expand social protection

coverage through providing financial assistance to Social Cash Transfers initially (ii) strengthen

integrated delivery systems for social protection through technical assistance that is policy-wide

(iii) pay strong attention to communication, aimed at entitlements awareness and creating

broader political support and commitment to social protection.

The Social Cash Transfer (SCT) programme is Zambia’s flagship government-run social

protection programme, which was established in 2003 to reduce extreme poverty and

intergenerational transmission of poverty. In response to the first objective of the GRZ-UNJPSP

programme, there have been continuous scale up activities of the SCT programme since 2014 to

achieve nationwide coverage. In the initial pilots, the coverage was restrictive because

beneficiary selection was based on a solely poverty-targeted selection mechanism. To improve

coverage and impact of the SCT on beneficiaries, GRZ recognized that the SCT programme

needed to improve beneficiary targeting by moving from a sole focus on poverty to a more

human-rights based approach. which was supported by evidence collected during 2010 and 2012

(Freeland, 2017; UNICEF, 2018).

As a result, the scale-up of the SCT programme from 50 to now all districts in the country in

2017 employed a hybrid targeting model that combines eligibility based on vulnerability

categories with proxy means testing to exclude households above a certain wealth threshold. To

support the 2017 scale-up, a large CD intervention was implemented to develop the capabilities

of enumerators and stakeholders to appropriately screen, select and enrol individuals (Freeland,

2018). The changes in the SCT programme’s selection of households included having a new

comprehensible targeting approach which included life-course vulnerabilities such as disability

of a household member; and the move from community targeting to self-selection. In addition,

mobile technology and digital management information system (MIS) was implemented to

prevent inefficiencies in enrollment and delay full national coverage.

The scale-up of the SCT programme was implemented in phases since 2014. The final phase

with the final selection approach was conducted from 2017 to 2018, covering a 24-month period

at national scale of 115 districts. Within 2017, the programme was brought to all remaining

districts (from 78), and increased the caseload from 173,000 at the beginning of the year to

537,000 at the end of 2017. The training for the officers and stakeholders, enumerators and

community level volunteers, including the relevant training materials such as the Enumerator



Manual for the Training of Trainers and the CWAC training materials were developed in

partnership between the Ministry, UNICEF and Oxford Policy Management (OPM).

The SCT CD intervention included officials from the MCDSS SCT Unit, as well as officers from

the MCDSS M&E Unit. At the province/district level, trainings also included trainings for

district officials, district social welfare officers, and members of the District Welfare Assistance

Committee (DWAC) and the Area Coordinating Committee (ACC) (UNICEF, 2018). Finally, at

the community level, members of the CWAC received trainings to better understand the SCT

programme, including payments, the management information system, case management and

eligibility criteria for beneficiaries. The trainings were also expected to increase CWAC

members’ awareness about their role in community validation and information delivery to

beneficiary households.

Table D2 presents a summary of the SCT CD intervention’s activities, audience, and expected

outcomes.

Table D2. Summary of SCT CD Intervention and Outcomes


Social protection:

Technical support

to develop and

expand the

government’s cash

transfer system

(beneficiary

selection)

Geographic

scope:

Across all

provinces;

115 districts


Training of master

trainers on the SCT,

including selection

and identification of

beneficiaries, and on

the SCT MIS

• Officials from the

MCDSS SCT Unit

• MCDSS M&E Unit

officers

• Expansion of SCT beneficiaries based

on rights-based approaches.

• Improved ability of MCDSS staff to

appropriately screen, select and enrol

individuals based on key life-course

vulnerabilities.

Province/district

level:

ToTs on data selection

and enrolment

• Master trainers,

district government

officials

• District social

welfare officers

• Members of DWAC

• Members of ACC,

assistant district

officers

• Improved capacity on ways to digitally

enrol beneficiaries through mobile

capture, and on the usage of the digital

management information system

(MIS).

Community level:

Trainings on MIS data

entry and digital

enrolment of

beneficiaries

• Members of CWAC

• Chair of CWAC

• Secretary of CWAC

• Enumerators

• Community

members who are

recipients of the

cash transfer

• Community-level staff have improved

knowledge on SCT programme, SCT

payment system, the management

information system, case management

and eligibility criteria for beneficiaries.

• Improved awareness among CWAC

members about their role in

community validation; increased

knowledge on information delivery to

beneficiary households.



WASH: Capacity development in use of the Mobile to Web tool to improve

real-time monitoring, and in community sensitization for scale-up of the

Community-Led Total Sanitation intervention

In 2012, the Zambia Sanitation and Hygiene Program (ZSHP) was implemented by the GRZ,

UNICEF and its partners to enable 3 million people to consistently use improved sanitation

facilities and adopt hygiene practices, and to ultimately decrease the high burden of sanitation

and hygiene related morbidity and mortality in rural areas of Zambia. To meet these targets, the

ZSHP initially utilized a paper-based system which led to slow data processing, transcription

errors, and caused significant delays in data transfers from community to central levels. Further,

accurate monitoring of sanitation and hygiene interventions require precise and timely data. To

address these challenges and enable real time monitoring (RTM), UNICEF and its technical

partner through the Ministry of Local Government and Housing (MoLG) launched an innovative

approach called the Mobile-to-Web (M2W) system to digitalize monitoring of hygiene and

sanitation interventions. The M2W uses mobile phones with simple protocols for reporting and

analysis and is coded in the DHIS 2 system.

In 2016, the GRZ set a target of making Zambia an Open Defecation Free (ODF) country and to

ensure that 60% of the population has access to improved sanitation by 2020. To meet these

goals, the ZSHP also launched an innovative approach called the Community-led Total

Sanitation (CLTS) to mobilize communities to adapt hand washing practices and to eliminate

open defecation.

To meet the above targets across Zambia, the ZSHP also facilitated a CD intervention supported

by UNICEF and implemented by Akros to enhance coverage of its CLTS activities and usage of

RTM systems from 65 districts to 72 districts across 9 provinces in Zambia. One of the key

activities of this CD initiative included community volunteers (community champions) trained to

facilitate triggering in communities. Triggering refers to community members made aware about

the importance of practicing hygiene and sanitation practices. To generate an enabling

environment of demand and leadership for improved sanitation and behavior change within a

community, Chiefs are empowered with data to monitor their chiefdoms.

The CD intervention included trainings on using the M2W tool for officials from several

ministries, including MoLG, MoH, the Ministry of Chiefs and Traditional Affairs (MoCTA) and

the Ministry of Water Development, Sanitation and Environment (MWDSEP) (UNICEF,

2017b). At the province/district level, relevant government officials and members of the health

promotion team from the District Community Medical Office, among others, received trainings

on real-time monitoring and managing the information system. At the community level, chiefs,

village headmen and headwomen, community champions (volunteers) and members of

Sanitation Action Groups received trainings on M2W data entry and on community triggering

(sensitizing and monitoring communities on their sanitation and hygiene practices). AIR will

conduct an evaluation of the CD interventions implemented from 2016 to 2018 for the scale up

of use of the Mobile to Web tool and community sensitization to expand the CLTS intervention.

Table D3 presents a summary of the WASH CD intervention’s activities, audience, and expected

outcomes.



Table D3. Summary of WASH CD Intervention and Outcomes


WASH:

Scale-up of the

M2W tool for real-

time monitoring of

WASH

interventions, and

scale-up of the

CLTS intervention

Geographic scope:

Nine provinces, 72

districts


Trainings and

technology updates at

the ministry level

• Officials from

MoLG, MoH,

MoCTA and

MWDSEP

• Improved capacity to digitize the

monitoring of CLTS intervention

through the mobile to web tool.

• Improved usage of the mobile to web

tool by relevant ministry staff.

Province/

district level:

Trainings conducted

by MoLG to build

local-level WASH

capacity in real-time

monitoring and MIS

management

MoCTA staff support

the CD intervention

through mass ODF

verification and

chiefdom triggering

Chiefs empowered to

view reports, charts

and maps

• Province-level

officials from the

above four

ministries

• Rural Water

Supply and

Sanitation (RWSS)

focal person

• Members of the

health promotion

team from the

District

Community

Medical Office

• Provincial and district level staff of

relevant ministries and recipients of the

CD trainings have improved capacity on

real-time monitoring and on ways to

manage the MIS information system.

• Chiefs have improved capacity in the

usage of tablets to view reports, data,

charts and maps. Chiefs have improved

knowledge to monitor their Chiefdoms.

• Improved capacity of community-level

staff and volunteers (community

campions) in the usage of mobile

phones to report data into the DHIS2.

Community level:

Trainings on M2W

data entry, DHIS2

data entry and

community triggering

Training on

sensitizing and

monitoring villages

• EHTs

• Chiefs, village

headmen or

headwomen

• Community

champions/commu

nity volunteers,

Sanitation Action

Groups

• EHTs, Community volunteers

(community champions) and SAG

members have improved capacity and

knowledge to conduct M2W data entry,

DHIS2 data entry and in community

triggering.

• Community-level recipients of trainings

have improved knowledge and capacity

on usage of mobile phones to enter data,

on M2W data entry and on community

sensitization content and activity.

• Communities and households have

improved knowledge on WASH best

practices.



Education: Capacity development in the Teaching at the Right Level approach

for scale-up of the Catch Up intervention

UNICEF and the Ministry of General Education (MoGE) have collaborated over the years to

tackle low numeracy and literacy rates in the country. To contribute to these efforts, the Catch

Up intervention—one of the latest CD interventions supported by UNICEF—uses the Teaching

at the Right Level (TaRL) approach. The goal of this intervention is to improve teachers’ skills

in teaching students at their individual level, grouping children by their ability rather than by age

and grade, thereby gradually improving children’s learning outcomes (Innovations for Poverty

Action [IPA], 2016, TaRL, 2019).

In 2018, MoGE used the Catch Up intervention to provide trainings and mentorship activities for

teachers (head teachers, deputy head teachers, senior teachers) and to build the capacity of

MoGE staff at the national, provincial, district and school level on the TaRL approach (IPA,

2016). As this intervention is ongoing, MoGE’s zonal in-service coordinators are being trained

as mentors and MoGE district staff are being trained as master trainers. AIR will evaluate the CD

intervention on the TaRL approach, for the 2016 to 2018 scale up of the Catch Up intervention.

Table D4 presents a summary of the Catch Up CD intervention’s activities, audience, and

expected outcomes.

Table D4. Summary of Catch Up CD Intervention and Outcomes


Education:

Scale-up of the

Catch Up

intervention using

the TaRL approach

Geographic Scope:

Lusaka City


Trainings on the TaRL

methodology through

the teacher-led model

of the Catch Up

intervention

• MoGE staff at the

national level

• National, provincial, district and

school level relevant MoGE staff

have improved knowledge on the

TaRL approach.



Child protection: Capacity development to improve integrated case management

to strengthen the alternative care system

UNICEF’s child protection programme supports the government of Zambia in developing a

strong child protection system to provide prevention services and adequate responses to

vulnerable children and adolescents (UNICEF, 2019). One of the programme’s key areas is

strengthening integrated case management (ICM) for child and family welfare and strengthening

the alternative care system (UNICEF, 2017a). Stakeholders conceived ICM as a tool for

providing services to vulnerable communities, in order to avoid inadequate and intrusive

approaches that do not holistically meet the needs of vulnerable families (SOS Children’s

Villages International, 2013; UNICEF, 2017a). The ICM model recognizes that the rights and

needs of children who face multiple risks are best addressed within a coordinated and integrated

approach (UNICEF Zambia, 2019).

The intervention that aimed to strengthen the alternative care system was facilitated by UNICEF.

The goal was to support MCDSS to implement the alternative care regulatory framework at the

national level, through a strengthened ICM, for vulnerable children and adolescents who have

been and are at risk of being separated from their families. AIR will conduct an evaluation of the

CD intervention implemented to strengthen the alternative care system through improvements in

integrated case management from 2016 to 2018.

Finally, across the selected CD interventions, we will analyze how the implementation processes,

activities, and training content used gender inclusive, rights-based, and equitable approaches. For

example, were the training activities implemented at a time and location that enabled all

participants, irrespective of gender and ability, to attend trainings? The data collection will

clarify whether and how implementers of CD activities incorporated these cross-cutting issues

into CD activities.

Table D5 presents a summary of the Strengthening of the Alternative Care System CD

intervention’s activities, audience, and expected outcomes.

Table D5. Summary of Strengthening of the Alternative Care System CD Intervention and

Outcomes


Child protection:

Strengthening the

alternative care

system

Geographic

Scope:

Lusaka City


Support and trainings

on improving use of

the ICM system

• National-level

MCDSS staff and

UNICEF child

protection officers

• National-level MCDSS have

improved knowledge on the use of

ICM system

LOCATIONS

Domestic: Washington, DC (HQ) | Monterey, Sacramento, and San Mateo, CA | Atlanta, GA | Honolulu, HI | Chicago and Naperville, IL

Indianapolis, IN | Metairie, LA | Waltham, MA | Frederick and Rockville, MD | Chapel Hill, NC | New York, NY | Columbus, OH | Cayce, SC

Austin, TX | Arlington and Reston, VA | Seattle, WA

International: Algeria | Ethiopia | Germany | Haiti | Zambia

9165_09/19

Established in 1946, the American Institutes for

Research (AIR) is an independent, nonpartisan,

not-for-profit organization that conducts behavioral

and social science research on important social

issues and delivers technical assistance, both

domestically and internationally, in the areas of

education, health, and workforce productivity.

MAKING RESEARCH RELEVANT

AMERICAN INSTITUTES FOR RESEARCH

1000 Thomas Jefferson Street NW

Washington, DC 20007-3835 | 202.403.5000

www.air.org

evaluation of unicef zambia’s capacity development ... · final report for the evaluation of...

Documents