macedonia conditional cash transfer project (cctp)
TRANSCRIPT
Macedonia Conditional Cash Transfer Project (CCTP)
Project Preparation
Monitoring & Evaluation of the CCTP
Terms of Reference
I Overview
The Government of Macedonia (GoM) wants to explore Conditional Cash Transfers
(CCT) as a means of streamlining the country’s social assistance system, while at the
same time improving the country’s performance on education, health and social
protection indicators.
Currently, the country possesses a relatively small but complicated and not transparent
social assistance system. The GoM is looking to improve and rationalize this system,
including introducing CCTs for poor families to break the inter-generational poverty
cycle.
The Conditional Cash Transfer project, supported by the World Bank and planned for
FY09, will aim at alleviating poverty while working towards improving the future
opportunities of recipients. This can be done through conditioning existing cash benefits
on behaviors that build human capital, such as pre- and primary school attendance, take-
up of primary health care, especially during the pre- and peri-natal phases, and attendance
of parenting classes, health care of mothers, children and elderly. Target groups are
benefit recipients, such as the economically vulnerable (currently receiving the Social
Assistance), and parents of children with special needs (receiving the so-called Special
Allowance). Through this targeting, CCTs de facto benefit the marginalized, including
rural populations, those with special needs and minorities (such as the Roma).
Under this project, the Government of the R.of Macedonia seeks to unite a team of
international experts and local consultants. International experts are sought in the areas of
Social Safety Net assessment, Benefit simulation modeling, Program Administration and
Monitoring and Evaluation of the CCTP. Each of the international experts will be
supported by a local consultant.
II General Objectives:
The consultant should design the Framework for the Monitoring & Evaluation of
the CCTP (‘Results Framework’), i.e. selects a set of Monitoring indicators to
track progress on the project, and prepares the Impact Evaluation. This includes:
Advising, managing and using the resources of a Working Group on Impact
Evaluation nominated by the GoM (‘IE WG’). The IE WG consists of the
Head of the State Statistical Office, the Minister of Labor and Social Policy,
and technical staff in charge of statistics of the Ministry of Labor and Social
Policy, the Ministry of Education and the Ministry of Health; Health
Insurance Fund and the Employment Agency of RM.
Designing and planning the rollout of any different CCT interventions in the
country; thereby determining treatment and control groups for a piloting;
Designing questionnaires for a baseline and a follow-up survey in co-
operation with the IE WG and the Survey Firm;
Hiring, instructing and supervising a Survey Firm with local expertise, to
conduct the baseline survey;
Co-operating with and building the capacity of a local academic consultant
hired separately by the GOM to support the work on the Impact Evaluation;
III Scope of services:
Note: The consultant at the initial beginning of the assignment shall familiarize himself
with the Macedonian Benefit system.
The Consultant should prepare a set of Monitoring Indicators to track the progress
of the project:
o The set will comprise realistic and meaningful measures for inputs,
outputs and outcomes of the CCTP.
o The Monitoring System will be the main instrument of measuring the
interventions under the project. Therefore, it needs to be compatible and
easily mergeable with the baseline and follow-up surveys for the Impact
Evaluation (see point below). This includes using at least one identifier
that is common to both instruments, i.e. the same school codes, postcodes,
municipality-codes, household keys, and regional keys. In this way, the
Monitoring System and the surveys can be merged at the interface of the
common identifiers.
o Together with the IE WG, the consultant will also determine the data
sources and collection channels for regular monitoring, from before the
start of implementation of the project.
o In order to facilitate this output, the Labor Market Analyst will inform the
Impact Evaluation expert about the design of the CCTP, as he progresses.
The World Bank team will share a framework for monitoring systems.
Baseline and Follow-up survey questionnaires to support the Impact Evaluation of
the CCTP.
o The consultant, in co-operation with the IE WG, and with a professional
survey firm (see point below), will choose a questionnaire for a baseline
and a follow-up survey of households, to measure the impact of the CCTP.
o The questionnaires should cover indicators for the intended impact of the
CCTP, e.g. health indicators, school attendance and other welfare
measures. It should also cover a comprehensive array of socio-
demographic background variables.
o It may be possible to use the Multi-Indicator Cluster Survey (MICS 2005)
already carried out by Unicef and the State Statistical Office as a baseline
for the CCTP. The consultant should lead the decision on this and jointly
decide with the IE WG, whether this survey is suitable.
o Any data collection for the Evaluation of the CCTP should be harmonized
with already planned data collections by the State Statistical Office. The
IE WG can advise on this.
Selection and instruction of a professional high-quality survey firm
o The consultant will draft Terms of Reference for the research firm to carry
out the survey, and assist the IE WG in compiling a shortlist of adequate
firms. He/she will also assist the IE WG in the selection of a research firm.
o Once the firm is selected, the consultant will involve it in the dialogue on
the impact evaluation and instruct it on the CCTP. Jointly with the firm’s
statistical team, he/she and the IE WG will decide on a sample size and
sampling strategy for the surveys.
A situation analysis and diagnosis of current issues in the social sectors that could
be usefully addressed by Conditional Cash Transfers, using existing data in
Macedonia (e.g. the MICS 2005,) such as e.g.
o discrepancies between rich and poor in pre-school, primary and secondary
education enrolment, as well as
o in peri-natal, early childhood and other existing health indicators; and
ante- and peri-natal health behaviour of parents (e.g. immunizations,
check-ups, nutrition etc.), and in general parenting behaviour.
o The consultant, together with the local consultant and support from the IE
WG, will analyse the relevant Macedonian data. Any previous analysis
regarding this topic, by Unicef or the World Bank will be shared with the
consultant.
A rollout plan for the CCTP.
o The consultant will decide, jointly with the IE WG, on an appropriate roll-
out strategy for the CCTP.
o This comprises selecting the treatment population and municipalities for a
first pilot of the CCTP during two years, as well as an adequate control
group.
o Roll-out in the form of a randomized experiment is highly recommended.
An Identification strategy to evaluate the CCTP.
o Together with the IE WG and the Labor Market Analyst, the consultant
will decide which of the interventions under the CCTP will be evaluated.
o Based on the chosen roll-out strategy, and the consequent treatment and
control groups, the consultant will develop a strategy to arrive at a
statistically sound identification of the impact of the selected
interventions.
A plan to disseminate the results of the Evaluation
o Jointly with the IE WG, the consultant will plan a strategy of
disseminating the results of the Impact Evaluation to the wider
Government, Academia and the public.
o The actual format of the dissemination process is subject to agreement by
the IE WG, but should include stakeholder workshops, cross-ministerial
presentations and dissemination of the data, international and national
academic publications and graduate school workshops in Macedonia.
Active participation in two workshops to build capacity.
o The first workshop, organized by separately hired consultants, will
introduce the IE WG and other selected officials of the GoM to the
concept of Impact Evaluation and its use for policy design. This workshop
will also serve to collect the IE WG’s input on the objectives of the Impact
Evaluation, and the use of the proceeding information within the GoM.
o A second workshop will present the IE WG with the different options of
targeting the CCT and rolling out the project, and their pros and cons.
A Manual including written procedures that will be used by the relevant
institutions for future monitoring of the CCTP.
A contribution to the Project Appraisal Document, summarizing the Results
Framework, comprising:
o the Monitoring Framework,
o the key steps of delivering the Impact Evaluation of the CCTP,
o The World Bank team can provide templates for this section of the Project
Appraisal Document.
The consultant will be expected to share his data and findings with the other
international and national consultants on the team as he/she progresses, and co-
ordinate data requests to the government with them. All international and national
consultants will work together in a team and share their findings.
The World Bank team and the PMU will support the consultant at the beginning
of the assignment with all relevant information and knowledge gathered to date,
as well as any relevant outputs already begun under the CCT project, such as
notes, statistical analyses and spreadsheet models.
IV Reporting obligation and outputs
The consultant shall prepare and deliver the following reports:
Stage I
The consultant will familiarize him/herself with the Macedonian Benefit system, the
background of the CCT project including the GoM’s strategy for social sector reform and
the data available to date. He/she will supervise technical staff from the IE WG in their
work on the diagnosis of issues in the social sectors using the existing data in Macedonia.
He/she will also participate in the first capacity building workshop with the IE WG, and
facilitate an agreement on the objectives of the Impact Evaluation. Jointly with the IE
WG, he/she will also decide next steps on a baseline survey and the selection of a survey
firm.
Outputs of Stage I:
A brief report summarizing the Diagnosis of issues in the social sectors, as
described in Section IV. This work should be fed back to the other consultants,
especially to the Labor Market Analyst on the team.
Participation in the separately organized capacity building workshop (with the IE
WG).
o This workshop will provide an accessible introduction to the concept of
Impact Evaluations and their use in policy making.
o The workshop will also serve to collect the IE WG’s input on the policy
questions that the Impact Evaluation might address, and
o It will further serve as a forum to agree on the surveys to be carried out for
the Impact Evaluation, and to kick-start the selection process of a survey
firm.
For the orientation of the IE WG prior to the workshop, the consultant will
prepare and circulate an agenda and questions to discuss, and a specific data
request if needed.
Terms of Reference for a survey firm.
Stage II
Representatives from the MLSP will draft the Monitoring Framework, with the
support and under the supervision of the Impact Evaluation consultant.
Meanwhile, the IE WG will select a survey firm to carry out baseline (if needed)
and follow-up surveys.
Under the supervision of the Impact Evaluation consultant, and in co-operation
with the survey firm, the local academic consultant will draft baseline and follow-
up questionnaires, and agree them with the IE WG.
Likewise, the team formed by the local and international consultants, the IE WG,
and the survey firm, will start work on a roll-out plan for the CCTP and a
sampling strategy for the survey.
Outputs of Stage II:
A set of Monitoring Indicators, as described in Section IV. The
Monitoring framework must be designed with a view to the CCT concept
put forward in the Labor Market Analyst’s preliminary report, to be
delivered during Stage II.
Questionnaires for a baseline survey (unless the MICS 2005 will serve as
baseline) and a follow-up survey of households with the same indicators
An agreed sample size.
A brief presentation at a second workshop, to present the IE WG with the
different options of targeting the CCT and rolling out the project, and their
pros and cons. This presentation should co-ordinate with and build on the
presentations of the other international consultants at the workshop. These
presentations will cover different scenarios for the design of the CCTP.
Stage III:
The team will finalize the roll-out plan for the CCTP and the sampling strategy for the
survey, and decide which of the interventions under the CCTP will be evaluated.
The consultant, together with the local academic consultant, will design an identification
strategy for an Impact Evaluation, based on the rollout plan for the CCTP and the data to
be collected by the survey firm.
Jointly with the IE WG, the consultant will plan a strategy of disseminating the results of
the Impact Evaluation to the wider Government, Academia and the public.
Outputs of stage III:
A written, transparent and user-friendly contribution to the Project Appraisal
Document, summarizing
o the Monitoring Framework;
o the rollout plan for the CCTP;
o data collection for the Impact Evaluation;
o an Identification strategy to evaluate the CCTP; and
o A plan to disseminate the results of the Evaluation.
The GoM and the World Bank team will provide the consultant with templates for
the structure of this contribution.
The consultant should submit the above listed reports in Macedonian and English
language. All reports (outputs) must be accepted by CCT Working Group. The
acceptance of the each stage reports is precondition for continuing the consultant’s work
into the next stage.
V Qualifications & Requirements:
Higher level University degree either in Economics, similar quantitative discipline
or public health with at least seven (7) years of experience in Impact Evaluation.
Proven experience and knowledge of labor market and social protection
economics and policy.
Proven analytical, problem-solving and quantitative skills, including statistical
packages.
Proven ability to build capacity of government officials.
Excellent oral and writing skills in English.
Publication track record is desirable.
Excellent interpersonal communication skills, demonstrated ability to work
cooperatively with clients, and ability to liaise tactfully as a member of a
multicultural team.
Proven ability to interact actively with clients.
VI Duration of the assignment
It is expected the consultant to perform the assignment within a period of 4 (four)
months, starting from the date of the contract signature.
The first ranked consultant should submit technical proposal containing time table for the
tasks to be accomplished under the assignment, but should also take into consideration
that his/her work is closely connected with the work of the other consultants hired under
the CCTP, and that the proposal may be a subject to additional adjustment.
Subject to good performance of the consultant, and willingness of both parties to continue
the relationship beyond project preparation, the consultant will be asked to carry out the
actual Impact Evaluation two years after the start of project implementation. – This is
because the nature of an Impact Evaluation makes it desirable that the same team be in
charge of its designing, planning, and realization.
VII. DATA, LOCAL SERVICES, PERSONNEL AND FACILITIES TO BE
PROVIDED BY THE CLIENT
1. The Ministry of Labor and Social Policy will engage a local consultant (financed
by the PHRD Grant for preparation of the CCTP) that will support the
international consultant during the performance of his/her assignment. The
consultant will be able to use the resources of the Ministry of Labor and Social
Policy, Ministry of Education and Science, Ministry of Health, Health Insurance
Found, and Employment Agency of RM.
2. The consultant will be given all relevant data that the: Ministry of Labor and
Social Policy, Ministry of Education and Science, Ministry of Health, Health
Insurance Found, and Employment Agency of RM, have. Considering that there is
a possibility the documents not to be available on English language, the
Consultant should in his/her financial offer allocate funds for translation. In the
same time the local consultant that will be engaged additionally will participate in
the translation/interpretation.
The consultant will be provided with suitable office space. Providing a computer (lap
top) is a responsibility of the consultant.
ANNEX A
List of relevant documents
1. Law on social protection and relevant bylaws www.mtsp.gov.mk
2. Law on child protection and relevant bylaws www.mtsp.gov.mk
3. National strategy for Roma
4. Poverty assessment prepared by State Statistic Office and World bank
5. Part of the IPA operating program
6. Law on primary education www.mon.gov.mk
7. Law on secondary education www.mon.gov.mk
8. Law on education inspection
9. Law for Bureau for education development
10. Law for specialized education
11. Ministry of Health www.zdravstvo.gov.mk
12. Health Care Law and all relevant bylaws www.zdravstvo.gov.mk
13. 14 National Vertical programs: - available on
http://www.zdravstvo.gov.mk/documents/documents/programi%2029.3.2007.doc
a. immunisation
b. brucellosis
c. control of tuberculosis
d. HIV/AIDS
e. Check-ups for school children and students
f. Program for reproductive health
g. Program for mother and child health care
h. Program for blood donation
i. Program for preventive health care
j. Program for uninsured persons
k. Program for drug adictors and other addictions
l. Program for dialysis
9. Health Sector Management Project – Technical Reports, Health Strategy of
Republic of Macedonia 2006-2015 available on http://www.moh-hsmp.gov.mk/
10. Health Sector Transition Project – Technical documents (especially for Pernatal
care) available on www.zdravstvo.gov.mk
11. Health Insurance Law and by-laws, Basic Benefit Package - Health insurance
Fund www.fzo.org.mk
12. Republic Institute for health protection www.rzzz.org.mk – Health map, Report
on health status and health care system and other technical Reports
13. Medical Map can be used with data on health status and health care on National
level, regional and municipality – promoted on 30.05.2007, available in Republic
Institute for Health Protection and in the Ministry of health
14. World health organization Health for all database should be used with HFA
indicators for Macedonia, updated in 2006 with data for 2005 www.who.int
Existing Data Sources
This is a non-exhaustive list of data sources existing in Macedonia, as far as relevant
to Cash Transfers and their beneficiaries.
Surveys
Multiple Indicator Cluster Survey (MICS), carried out by Unicef in 2005-2006 and
published 2007, with a sample size of 5379 households. This household survey
contains valuable indicators on maternal and child health, nutrition, health system use
and self-medication, in-house and out-of-house educational activities, as well as a
rich array of socio-demographic indicators covering age, gender, wealth quintile,
ethnicity, region, urban/rural. Individual-level data is due to be released by Unicef in
May 2007. At present, summary tables are available which relate the key indicators to
the demographic background variables.
Household Budget Survey (HBS), carried out by the State Statistical Office (SSO)
annually, with a sample size of 5050. The survey covers household expenditure,
income and socio-demographic background, without ethnicity. Currently, the last
available processed year of the HBS is from 2005, but 2006 data should be available
from May 2007.
Labor Force Survey (LFS), carried out by the SSO annually, with a sample size of
nearly 10,000, covering people’s waged activity, sectors, salaries and socio-
demographic background, again without ethnicity. The last available year is currently
2005, but 2006 should be available from May 2007/ The SSO does not usually release
individual data, but can produce summary tables on demand.
More information about data handled by the State Statistical Office can be found on
www.stat.gov.mk
Administrative Data
The current legislation on the Macedonian Benefit system can be found under
http://www.socialcohesion.coe.int/MISSCEO/IndexCountry.aspx?year=2007
Benefit Accounting Data for Social Assistance and Child Allowance is held in the
Accounting Department of the MLSP. It contains the total amount spent on these
benefits and the number of beneficiaries by level of benefit, and by the main criteria
of eligibility, such as age for children.
Benefit Accounting Data for Unemployment Benefit is held in the Accounting
Department of the National Employment Agency. It contains the total amount spent
and the number of beneficiaries by level of benefit, and by the main criteria of
eligibility, such as time out of work, and time to pension.
Relevant previous Donor Activities
World Bank
Social Protection Implementation Project (SPIL).The objectives of SPIL are to
improve the effectiveness and efficiency of social protection system through
improved administration and long-term sustainability of the pension system and
improved targeting and administration of cash benefits. The project includes three
components: (i) pension system administration; (ii) social protection program
support; and (iii) project management, monitoring and evaluation. The first two
components consist of multiple subcomponents. Component (i) comprises support for
governance and management reform in the PDF, the development of the pension
supervision agency, MLSP capacity building for pension system analysis, policy
development and public education, and pension system custody capacity development
at the NBRM. Component (ii) comprises support for the policy framework for
streamlined, cost effective and better targeted cash benefits, implementation of local
social planning and deinstitutionalization of social work services, and establishment
of inspection, supervision and quality assurance functions; and information
management systems and IT capacity strengthening.
Foundation Open Society Institute – Macedonia FOSIM
Roma Education Program (REP). The goal of the REP is to help Roma students to
remain in the education system and to improve their performance and attendance
through incentives at all educational levels, from pre-schools to university. Students
receive cash benefits through the banking system in four yearly installments, against
proof of attendance from the teachers’ registers. The program further supports
teacher-mentors for students, who serve as a major liaison for the program in schools.
So far, the evidence suggests a significant increase in retention, depending on the
grade from 7-17%. The REP is managed by FOSIM and financed by USAID,
Pestalozzi Children Foundation, OSCE and the Hungarian Embassy.
Unicef
Early Childhood Development (ECD). Unicef currently develops a comprehensive
ECD program, covering parenting courses, vaccination, satellite childcare centers
attached to a kindergarten, materials and standards based on the Northern Irish
‘Lifestart’. The program is the concept phase, and a Working Group of
representatives of the MES, the MOH, the MOF and the Local Government
Association has just begun identifying project components.
Child-friendly Schools. This project has developed a set of standards that a ‘child-
friendly’ school should fulfill in terms of health & safety, conflict solution strategies,
gender, participation by parents and the community, multiculturalism and child rights.
The standards have been successfully applied to a set of pilot schools in order to foster
social integration of the poor, Roma, and children with SEN and are ready to be scaled
up.
GOVERNMENT OF JAMAICA/WORLD BANK
SOCIAL PROTECTION PROJECT
TERMS OF REFERENCE
PROCESS EVALUATION AND DEVELOPMENT OF SERVICE STANDARDS
PROGRAMME OF ADVANCEMENT THROUGH HEALTH AND EDUCATION
I. BACKGROUND OF PATH In an attempt to reform the Social Safety Net (SSN), the Government of Jamaica has embarked upon a major social security initiative that has replaced the three key existing
social assistance programmes (Food Stamps, the Outdoor Poor Relief and the Old Age and Incapacity Programmes) with the PATH.
The overall objective of the PATH is:
To ensure that the vulnerable groups in society are assisted by the state. To attain economies through implementation of a single system of benefits. To increase transparency in the selection of beneficiaries.
The PATH is currently funded by a loan of US$40 million from the World Bank and
supplemented by US37.5 million from the Government of Jamaica. A further US$40 million will be provided by the World Bank under a new Social Protection Project (2008
– 2013) of which the PATH will benefit. Since its launch in 2002, the PATH, administered by the Ministry of Labour and Social
Security has become the country’s main social assistance programme, providing cash
transfers to almost 280,000 beneficiaries in 2008. The transfers are made through two
categories of grants (i) education and health grants for poor children 0 – 17 years old
conditioned on school attendance and preventative health care visits and (ii) social
assistance grants for poor pregnant and lactating women, the elderly poor, poor persons
with disabilities and destitute adults under the age of 65.
The base benefit under the programme during 2007 was J$530 (or US$8) per eligible
person per month with beneficiaries receiving payments of J$1,060 every two months.
This base benefit was increased to J$650 per month per beneficiary effective April 1,
2008. Effective December 2008, the Government of Jamaica introduced a differentiated
scheme of PATH benefits to compensate secondary school students for higher
opportunity costs, (in particular for boys) and to motivate them to finish high school and
continue on to tertiary education or training.
II. MONITORING AND EVALUATION SYSTEMS
1
The PATH has been monitoring and evaluating its programmes through its planning & monitoring unit, impact evaluations, internal and external audits:
i.) Planning & Monitoring Unit – The planning & monitoring Unit with information
provided by the Management Information System (MIS) prepares progress
reports in addition to conducting mini research projects to gather information to assist with evaluating the performance of the programme.
ii.) Impact Evaluation – The impact evaluation used a quasi experimental design
(regression discontinuity) to compare households just below the program
eligibility threshold (just-eligible) with households just above the eligibility
threshold (near-eligible). The impact evaluation was complemented by a
qualitative analysis of program implementation and an analysis of benefit
targeting. Data for the evaluation were obtained in several ways: (i) household
surveys were conducted at several points in time, including a baseline survey
of participant and comparison group households and a follow-up survey of the
same group after the participant group had received several disbursements of
benefits; (ii) qualitative interviews and focus groups with program staff, as
well as the staff of schools and health care facilities serving program
beneficiaries and (iii) data extracted from the Management Information
System (MIS) used to operate the program.
iii.) Internal & External Auditors – The operations of the PATH are audited by both
the Ministry’s Internal Auditors and the External Auditors of the Accountant
General’s Department. This is to ensure that the PATH complies with the
Government of Jamaica guidelines set out in the FAA Act and the World
Bank Financial Accounting Reporting and Auditing Handbook (FARAH).
In addition the Programme wants to conduct periodic process/implementation
evaluations, develop service standards and monitoring mechanisms for compliance with standards:
iv.) Process/Implementation Evaluation - the objective of this is two fold. It is to
detect if the Programme is following the guidelines as set out in the
operational manual and measures the complaints, expectations and satisfaction
levels of the stakeholders. In order to do this, the Programme will hire an
external firm to execute qualitative and quantitative assessments. From the
results of these assessments, it is expected that the Project directorate will be
better guided in its decision making, resulting in improvement and
adjustments for the operations of the Project.
v.) Development of Service Standards and Monitoring Mechanisms for compliance
with Standards – This activity is aimed at establishing continuous improvement in the operations of PATH at the local level and will draw on
2
information from the enhanced Management Information System (MIS) that
will facilitate the development of periodic reports about the standards
compliance for parish offices. This will allow operations staff, in these offices
to understand where the gaps exist and can facilitate improvement in their
performance and service delivery. After the process evaluation is carried out,
the consultant will assist the PATH management to establish a set of service
standards. The standards will be focused on key aspects of the service
delivery.
This TOR has been developed to contract and execute the components which are stipulated under numbers iv and v.
III. ASSIGNMENT OBJECTIVE The main objective of this consultancy is to validate the accuracy of the information
managed by the Programme and to measure the efficiency of the PATH project
processes. The process/implementation evaluations should provide feedback for program
improvement by answering questions about the processes that Programme management
directly control to achieve results. The outcomes of the process evaluation should feed
into the development of standardized service standards that can be used to assess service
delivery in the PATH.
The specific objectives are as follows:
1. To review the efficiency of the project cycle phases: targeting, enrolment,
payment, compliance with conditions, case management. 2. to review the level of services that are provided by the schools, health centres
and post offices 3. To review the quality of the services provided by PATH to the service
providers (health centres, schools, post offices & Bank) and how the
beneficiaries feel about the quality of service that is provided by PATH and the service providers.
4. To develop standardized service standards for the PATH that can be employed in the parish level.
5. To develop guidelines on how to monitor the service standards established.
IV. OVERVIEW OF ASSIGNMENT FOR THE PROCESS EVALUATION
The following gives an overview of some of the areas where assessments should be conducted during the process evaluation.
1) Project Cycle
3
Application It is essential to ascertain whether the procedures for application are being followed as
stipulated by the Operations Manual. Specifically, are the application forms being
completed by the staff of the Ministry of Labour and Social Security and how accurately
is this being executed? How long does it take to process an application from the point of
interviews through to a qualified applicant receiving payment? Are the potential
beneficiaries receiving clear information on the application process?
Verification Upon successful application to PATH, it is essential to know if the social workers in the
Parish Offices are actually going out into the field to verify the information that is
collected and captured on the application forms and how long it takes for a social worker
to visit the household and verify the information. It is essential to know if the new
beneficiaries were given training on the rules of PATH and how the Programme operates
and what are their obligations. Of equal importance, this consultancy should assess the
quality of the training that was delivered to the beneficiaries by the Social Workers.
Payments
Cheque Payments - Payments are to be made on the 15th
of every even month (February,
April etc). It is important to know if payments are made in a timely manner, if postal offices/points of services treat beneficiaries in a respectful way. For beneficiaries who do not collect their cheques, it is important to identify if there are any operational reasons for them not collecting.
Keycard Cash – Examine the application process for key card cash if procedures are followed and service delivery is adequate. If persons are receiving their payment via
keycard cash by the 15th
of every even month and the means of communication in terms
of payment posted to the accounts. Whether or not the correct amounts are posted to their accounts and any concerns with points of purchase.
Compliance Service providers are required to complete and submit their compliance reports within a
given period. Are these service providers complying with the schedule given and what
are some of the problems encountered by the service providers? It is also important for
this consultancy to ascertain the perspective of the social workers who are charged with
the responsibility of ensuring that the compliance reports are submitted to PATH within a
stipulated time.
Case Management Is PATH actually facilitating the appeals process island wide? Are complaints being dealt with in a timely manner? Are changes to the status of the family, which affects the level
of payment received, being addressed by PATH? Additionally, are visits being made to
the beneficiaries by the social workers?
2) Provision of Services
4
Education Identify difficulties/problems that teachers may experience in providing the required attendance information to PATH and problems experienced by staff of the PATH
Secretariat in collecting/using the compliance reports submitted by schools. How can PATH improve its current operations and services which it gives to beneficiaries?
Health Identify problems that are experienced by health centre personnel in providing health
services to PATH beneficiaries. Highlight the difficulties, if any, experienced by
parents/caregivers of PATH beneficiaries who are required to take their children to the
health centres as stipulated by PATH. Highlight problems that may be experienced by
staff of the PATH Secretariat in collecting/using the compliance reports submitted by
health centres. How can PATH improve its current operations and services which it
provides to beneficiaries?
Post Offices Pinpoint challenges experienced in the process of administering payment/cheques to the beneficiaries of PATH. Identify problems experienced by the beneficiaries in collecting
their cheques from the post offices. How can PATH improve its current operations and services which it offers to beneficiaries?
National Commercial Bank (NCB) Identify any problems (potential or realized) that are experienced by the NCB in providing services to PATH beneficiaries. Highlight any difficulties PATH beneficiaries
may be having in utilizing their ATM cards. Unfold any concerns that the MLSS may have with the services of NCB as the provider of the keycard cash system.
3. PATH Services
Ministry Level Are the ministries involved in the execution of PATH satisfied with the level of communication between PATH and themselves? Are the communication channels
adequate to facilitate dissemination of information between the ministries?
Operational Level Are the field staff who work in the schools, post offices, health centres satisfied with the
level of training that has been given to them re: PATH. Do they think that there are
adequate channels in place to ensure the free flow of information about PATH and its
functions to them? Are they satisfied with the operations of the Parish Offices and PATH
as a whole? Is the Bank/MLSS satisfied with the level of communication between
themselves in order to provide service to the beneficiaries? Beneficiaries
5
Are the beneficiaries satisfied with the Programme? What are their feelings and
perception about the application, selection and training processes? How do beneficiaries
feel about the avenue for reporting complaints about issues related to PATH (appeals
committees, parish office social workers)? Do they think that PATH has adequate
machinery to deal with these complaints? How do they view the PATH appeals process?
Is it operating in a manner that seems to be fair? What are their impressions about how
they are treated as beneficiaries of PATH by the staff and other stakeholders? How can
PATH improve its current operations and services which it gives to beneficiaries?
V. METHODOLOGY
The consultant will utilize different methods to carry out their investigation in the field. These are inter alia:
i) Field work using questionnaires on representative samples of stakeholders
Once the area of study is identified then the consultant will be required to
prepare questionnaires that are relevant to the stakeholders that are being
understudied. The consultant will then be required to identify representative
samples to which the questionnaires will be applied. The firm will undertake
the field work, collect and process the information, analyze the results and
finally make inferences to the entire population of the given stakeholders
being studied.
ii) Fieldwork using questionnaires/interviews on targeted and problematic
stakeholders. Once the area of study is identified then the consultant will be required to
prepare questionnaires/guided questions for interviews that are relevant to the
stakeholders that are being understudied. The consultant will then be required
to identify samples to which the questionnaires/interviews will be applied.
Undertake the field work, collect and process the information, analyze the
results of the problematic target group.
iii) Qualitative fieldwork The consultant will identify the persons who will participate in the focus
group sessions and interviews. The firm will be required to prepare the guided
questions which will guide these meetings. The consultant will also be
expected to organize the focus group sessions and execute the work, record
the proceedings of the meeting, analyze the results and produce the report.
The consultant has to be aware and be very knowledgeable of the operations of PATH.
Using all these tools the consultant will be able to address the issues identified for the specific period and so the report has to include the following:
i) methodology utilized
6
ii) the main results
iii) specific recommendations
iv) operational recommendations to improve the process being analyzed
VI. OVERVIEW OF THE SPECIFIC ASSIGNMENTS FOR THE PROCESS
EVALUATION
a. First Assignment For the first assignment we need a comprehensive review of the project cycle including
application, verification and appeals process. For the first and other assignments we need
an overview and assessment of the satisfaction level, main preoccupations, and main
expectations of the beneficiaries and stakeholders with the Programme’s operations. For
these the consultant has to design a questionnaire which will be used to collect data from
a sample of beneficiaries and other stakeholders.
N.B The field work will be done for samples of beneficiaries and relevant stakeholders with estimation errors no higher than 10%. The samples should cover about 1% of beneficiary households, 3% of schools, 5% of health facilities, 3% of post offices, 10% of
parish and PATH officers. At least one person who is knowledgeable of the operations of PATH and the relevant institution should be interviewed.
The activities to be done are with the aim: to identify process flaws, inefficiencies and sources of error;
to conduct “as is” analyses of existing processes aimed at identifying and prioritizing processes to be redesigned;
to define each process in terms of the required operations, actors, inputs, outputs, and interfaces
to develop blueprints of PATH’s existing and proposed operating processes, including Parish Offices’ operations;
to streamline and to improve processes between PATH and its Parish Offices’ operations;
c. Second Assignment Specifically of this second assignment the Programme is interested in learning more about the Compliance process.
Since the first compliance check which was executed in March 2003 to present, PATH
has continued to experience high levels of non-compliance among its children
beneficiaries who fall in the age range 1<6 years old and 6-17 years old. Levels of non-
compliance for the children 1<6 years old have ranged from a low of 34% (13,246
individuals) to a high of approximately 45% (17,389 individuals). Of equal importance is
the high level of non-compliance being experienced by the education beneficiaries who
7
fall in the age range 6-18 years old. Data collected has shown that there has been a significant reduction from 30% to approximately 14% among the education beneficiaries.
PATH, however, is still working towards its target of achieving 90% compliance with the
requirements stipulated by the Programme among its beneficiaries. It is with this goal in
mind that the second assignment of this consultancy will be to identify the factors that
result in the levels of compliance among the children who receive health and education
benefits from the Programme.
N.B The field work will be done for samples of relevant beneficiaries (i.e., not complying)
and relevant stakeholders with estimation errors no higher than 10%. The samples to be
covered should be about 1% of beneficiary individuals (300 cases), 1% of schools (10
schools having high levels of non compliance), 3% of health facilities (10 health facilities
having high levels of non-compliance), some parish and PATH officers. All instruments
explained in the methodology section should be utilized for this investigation.
The consultant will be expected to:
i. Identify operational factors that affect the ability of personnel from the schools, health centres and PATH to accurately record and prepare compliance reports
for submission to PATH. ii. Identify if issues of non-compliance are related to the services provided by the
school, health centres, PATH. iii. Document the views and opinions of the non-compliant beneficiaries about
the conditionalities that are enforced by PATH. iv. Highlight why non-compliance rates continue to be high among the PATH
beneficiaries.
In addition, the consulting firm will be required to do a comprehensive review of the compliance, payment, complaints and updates activities.
The activities to be done are with the aim: to identify process flaws, inefficiencies and sources of error;
to conduct “as is” analyses of existing processes aimed at identifying and prioritizing processes to be redesigned;
to define each process in terms of the required operations, actors, inputs, outputs, and interfaces;
to develop blueprints of PATH’s existing and proposed operating processes, including Parish Offices’ operations;
to streamline and to improve processes between PATH and its Parish Offices’ operations;
d. Third Assignment
8
In order to assess the quality of services provided by the PATH, the consultant will be
required to do a comprehensive review of the communication channel that exists within
the programme: Undertake a review of the organizational structure of the PATH (Head
Office & Parish Offices) in terms of the roles and responsibilities of staff in executing the
programme.
The consultant will be expected to review the various forms, guidelines/procedures etc.
used by the PATH to determine their effectiveness and where appropriate make
recommendations for change. In addition, a review of the Operational Manual should be
undertaken to highlight any differences between the documented procedures of the PATH
and practiced procedures.
VII. OVERVIEW OF THE SPECIFIC ASSIGNMENTS FOR THE
DEVELOPMENT OF THE SERVICE STANDARDS
The outcome of the process evaluation will provide information on the quality of service
delivery of the PATH, identify bottlenecks and suggest remedial actions to be taken. It is
on this platform that the proposed service standards are to be developed. The proposal should include inter alia:
1) A description of the service(s) PATH intends to provide and the benefits clients are entitled to receive.
2) Development of a service pledge or principle describing the quality of service delivery
clients should expect to receive, focusing on such elements as openness, fairness,
courtesy, professionalism, etc.
3) Specific delivery targets for key aspects of service, such as timeliness, access and accuracy.
4) The costs of delivering the service and
5) Complaint and redress mechanisms that clients can use when they feel standards have not been met.
The consultant is also required to: 1) Consult with clients (beneficiaries and stakeholders) to find out what is important, how
satisfied they are with current service delivery, what’s working well and what needs to be fixed. Consult with the PATH front-line staff to find out how they think service can be
improved within existing resource levels. 2) Develop a communication strategy that clearly and effectively communicates the
service standards, propose the most effective means of communicating the service standards to the PATH stakeholders, include suggestions on how PATH can build on its
9
current communication methods of articulating the standards, and make recommendations on ways to prepare a long-term communication strategy.
3) Conduct a workshop(s) that will train PATH staff in techniques and skills for improving quality and client service and that will empower them to better serve clients.
VII. CHARACTERISTICS OF THE ASSIGNMENT The assignments are for seven months: Five months for the process evaluation and two months for the service standards
In terms of the process evaluations: upon completion of its field work and analysis, the
firm will submit a draft report that will be discussed with the PATH officers. If needed,
PATH will submit its comments to the firm. Having received the comments from PATH,
the firm will finalize the report and make a presentation of its findings in a workshop to
relevant stakeholders. In addition during this workshop PATH will make a proposal as to
how it can address issues identified by the firm. The stakeholders participating in this
workshop will be given the opportunity to express their opinions on the issues being
discussed.
For the service standards, the consultant is expected to have a series of consultations with
the PATH management and stakeholders. Thereafter, they will submit a draft report of
the proposed service standards for discussion. The suggestions will be incorporated into a final report and the relevant staff trained and sensitized to the service standards.
VIII. DELIVERABLES The Consultant will be required to submit the following deliverables for the process evaluation:
i) At the commencement of the consultancy an overall Work Plan for the five months period outlining the methodology, and research tools to be utilized in
carrying out the assignments. Along with this, a detailed work plan for the first assignment.
ii) Detailed work plans for the second and third assignments to be submitted at
the beginning of each assignment.
iii) Draft reports at the end of each of the three (3) assignments outlining
methodology utilized, analysis done, conclusions and recommendations.
iv) Stakeholder workshops at the end of each assignment where participants can
make their comments on the draft reports that are produced.
v) Final report on the three assignments incorporating stakeholder comments
along with the adjustments made to the PATH Operational Manual.
10
The Consultant will be required to submit the following deliverables for the service standards:
i) An overall workplan for the two months period for the development of the service standards
ii) Draft report on the proposed service standards
iii) Final Report on the service standards
iv) Workshop for PATH management and staff.
IX. PAYMENTS
10% Mobilization advance upon the signing of the contract. Process Evaluation
15% upon the completion of the 1st
assignment.
15% upon the completion of the 2nd
assignment. 15% upon the completion of the 3
rd assignment.
15% upon the submission of the Final Report. Service Standards
10% for the draft report on the proposed service standards 10% final report on the service standards 10Workshop for PATH management and staff
X. SUPERVISION/COORDINATION The Consultant will work under the direct supervision of the Project Manager of the
Social Protection Project. He/She will also be expected to have consultations with the
management and staff of the Parish Offices along with functional managers and line personnel within PATH.
11
TANZANIA SECOND SOCIAL ACTION FUND (TASAF II)
TERMS OF REFERENCE FOR CONSULTANCY TO FINALIZE DRAFT QUESTIONNAIRES AND TO COLLECT FOLLOW-UP DATA FOR THE
TASAF II IMPACT EVALUATION
1.0 BACKGROUND The Government of the United Republic of Tanzania is implementing the Tanzania Second
Social Action Fund (TASAF II) through financing from the World Bank. The objective of the
Tanzania Social Action Fund (TASAF) is to enhance the capacity of communities and other
stakeholders (e.g. district and village government, NGOs etc.) to prioritize, implement and
manage sustainable development initiatives, and in the process improve socio-economic
services and opportunities that contribute to improved livelihoods linked to MDG indicators
targets in the National Strategy for Growth and Reduction of Poverty (NSGRP). In order to
assess the impact of the TASAF programs and projects on poverty reduction and/or welfare
improvements in Tanzania, a team of experts from the World Bank have joined monitoring
and evaluation specialists from TASAF to conduct a „joint-learning activity‟ in the form of a
thorough, randomized impact evaluation study. Specifically, this „joint-learning activity‟ will
attempt to quantify the extent to which support to vulnerable groups to implement income
generating sub-projects can improve the welfare of the beneficiary households. A
randomized, controlled evaluation has been designed in which baseline data have been
collected in five study districts and it is envisaged that households will be visited two more
times: one year after baseline data collection and two years after the intervention (i.e. a
panel data set will be collected).
2.0 RATIONALE The Bank‟s recently approved Strategic Framework for Assistance to Africa identified
community driven development projects (CDD) as one of the cornerstones of the Region‟s
strategy for supporting governments to more effectively reach the poor and vulnerable, and
improve governance and accountability in the use of donor and local funds. The strategy
1
noted that community empowerment regularly produces development projects with more
successful implementation and called for an increased scale-up of CDD in the region,
by: (a) expanding the donor funding delivered through CDD mechanisms; (b) expanding
the scope of sectors using CDD mechanisms; and (c) increasing the mainstreaming of
CDD mechanisms through building the capacity of local governments within CDD
projects. For the Africa region, CDD has now become a programmatic approach,
encompassing decentralization, governance reform, empowerment and accountability
efforts as integrated pillars. The number of CDD projects has grown rapidly in the region,
and at least 16 countries have incorporated CDD pillars into their CAS.
Despite the use of CDD mechanisms in the Bank for nearly two decades, interest in
impact evaluation only began in the late 1990s. Since then, a number of reports have
been published, including Rawlings, et al (2000), World Bank (2002), Mansuri and Rao
(2003), and Wassenich and Whiteside (2003). Authors differ in their assessment of
social funds. Issues include: (a) the project choice, cost, and sustainability of social fund-
financed infrastructure vs. infrastructure financed in traditional, top-down development
projects; (b) the effectiveness of targeting mechanisms in social funds vs. alternative
delivery mechanisms; (c) the development role played by the community empowerment
components and (d) how to integrate social fund mechanisms into country-owned
development strategies and local institutions. In particular, social funds have been
criticized for creating a separate, parallel structure, with poor interface with budgets and
sector ministries. The later point is of particular interest in Africa, where the CDD
strategy explicitly calls for creating a permanent institutional structure for investment
directed by empowered communities (World Bank, no date).
A recent review of CDD impact evaluations around the Bank (Wassenich and Whiteside
(2003) identified the Africa region as having the lowest percentage of operations under
evaluation. The review also highlighted the following topics as under researched: (a) project
impacts on, and interaction with, decentralization processes and public sector management;
(b) the comparative performance of CDD versus more centralized approaches in achieving
desired outcomes; (c) the impact of various components within a project to help “unpack the
black box” of evaluation work; (d) long term sustainability of infrastructure outcomes 2
and welfare impacts; and (e) targeting mechanisms. All of these are areas of high strategic importance to the Africa regional program and for the Bank-wide CDD effort. This projects attempts to remedy the evaluation deficit in Africa, both in terms of ZKHWKHU CDD achieves the promise of better service delivery, sustained community empowerment and reduction in vulnerability, and KRZ this is done - how effective are specific components. This study is a collaborative, „joint-learning activity‟. The design of the study, the key hypotheses, and the methodology has been discussed and agreed upon. By visiting households repeatedly, panel data will offer an opportunity for impact
evaluation of TASAF‟s support to specific beneficiary groups (vulnerable households).
Because the data are collected repeatedly from the same households and communities,
panel data can provide baseline data as well as follow up data from which impact can be
assessed. The study, as designed, has a number of unique features, which include:
A random assignment of villages into treatment and control groups, where the
villages in the control group receive their projects with a delay of 12 months.
A detailed listing of each household in each of the villages in the sample to study
village level targeting, benefit incidence, effects of social networks in decision
making, and elite capture.
A stratification of the sampled households in each village (made possible by the
listing exercise mentioned above) into four groups: households with beneficiaries,
households with vulnerable individuals but no beneficiaries, households with no
beneficiaries (and no eligible members), and „prime movers‟ (which include most
influential persons identified by sampled sub-project beneficiaries and the village
executive officer).
Collection of qualitative data on the process by which the beneficiary groups operate. 3
Close integration in the Mkukuta Monitoring System by finalizing questionnaires that
will also form the basis for the National Panel Survey.
3.1 OBJECTIVES The main objective of the consultancy is to field test draft questionnaires for
follow-up data collection, to collect and process the data required to successfully
carry out the TASAF II Impact Evaluation, and to produce a high quality data set. This particular assignment is to implement, starting in April, 2009, follow-up data
collection in five study districts in Tanzania – namely Moshi, Lushoto, Kwimba,
Nzega, and Makete. Upon the satisfactory completion of the baseline data collection,
the consultant might also qualify to implement round 3 of the panel survey.
4.0 EVALUATION INSTRUMENTS
In collaboration with members of the study team (from the World Bank and from TASAF), the
consultant is expected to revise the existing multi-topic household questionnaire that was
used for baseline data collection, and which follows a modular design along that practiced by
the Living Standard Measurement Surveys (LSMS). This household questionnaire should be
revised as necessary to analyze (dynamics of) household welfare and processes of income
generation, as well as household participation in development projects.
In addition to this household questionnaire, a community questionnaire (to assess
community characteristics) will also be implemented. Again, this survey instrument should be
based on the one that was used during baseline data collection and revised as necessary.
5.0 EVALUATION SAMPLE
The evaluation will be implemented in five districts and have a total sample size of approximately 1,500 respondents in 100 villages (15 respondents per village). However, as
4
this is a panel data collection effort, those households which have relocated between
baseline and follow-up data collection will need to be tracked and surveyed at their new
residences. This implies that the number of villages/locations visited by the team will be
larger than 100 villages. Furthermore, as such households may have moved outside of our
study districts, other districts of Tanzania may have to be visited by the survey teams.
6.0 SCOPE OF WORK Instruments for follow-up data collection under this „joint-learning activity‟ will need
to be revised from the versions used for baseline data collection. Then, all the
survey instruments will require formal field testing and finalization. In this case,
the consultant‟s responsibilities to be undertaken will be as listed below:
i. Revise, field test & finalize the various questionnaires & prepare an
enumeration manual;
ii. Translate questionnaires in Kiswahili;
iii. Program the ultra mobile PCs (UMPCs) so that the revised surveys can
be conducted using this technology;
iv. Perform extensive enumerator training;
v. Collect data in all selected enumeration areas using the developed
instruments;
vi. Conduct permanent quality control;
vii. Prepare data sets and submit the same to the evaluation team,
viii. Any other tasks needed for the successful implementation of the data
collection and processing for the impact evaluation.
7.0 SUPPORT TO BE RENDERED BY TASAF TASAF, in collaboration with study team members from The World Bank, will provide
the consultant with support in terms of information and communication with relevant
LGAs to facilitate smooth implementation of this task. As well, they will
5
assign staff to closely work with the Consultant at all times while this assignment is being carried out.
8.0 CONSULTANT’S QUALIFICATIONS The Consultant for TASAF II Impact Evaluation should possess the following qualifications
i. Experience in developing and field testing questionnaires;
ii. Experience in successfully implementing LSMS type panel surveys (track
record of listing integrated in field work; experience with in-field sampling; low
attrition rates in panel surveys; experience in measuring welfare changes over
time; experience in creating identifiers to track respondents; experience in data
entry systems for panel data);
iii. Ability to provide technical backstopping (experience with experimental and
quasi-experimental impact evaluation design) and ability to adjust the design of
the impact evaluation in response to unforeseen circumstances and
opportunities; ability to engage with an academic audience on the design of the
impact evaluation and willingness to adjust evaluation‟s design and
questionnaires in response to informed discussions;
iv. Ability to facilitate follow up work following from the impact evaluation;
v. Ability to revisit the same households 12 after they were visited for follow-up
data collection (subject to satisfactory performance in the first year);
vi. Ability to track households that have relocated;
vii. Ability to implement surveys nationwide;
viii. Ability to data capture immediately using UMPC technology and to provide a
cleaned data set for each study district within one month after the completion of
data collection in each district (excepting data for households that need to be
tracked later);
ix. Knowledge of the dynamics in terms of social, culture, political and economic
aspects of Tanzania, as well as administrative structure and functions of both
local and central government;
x. Residence within the country is crucial; 6
xi. Ability to combine the implementation of this study with related research activities (targeting of TASAF II; experiments with questionnaire design) will be an added advantage.
xii. Ability to produce papers of the highest academic standards will be an added
advantage.
9.0 KEY STAFF The consultant should ensure that the following key staff members are available for this assignment:
i. Team Leader who should possess at least PhD or equivalent in social
sciences particularly in economics and/or statistics. Five years or more
experience in conducting surveys and research in developing countries;
ii. Experienced Survey Team Leaders and Field Supervisors for effective
and timely execution of field interviews. These field staff members
should be selected from among those who worked on baseline data
collection as much as possible.
10.0 TIME FRAME The „joint learning activity‟ for the TASAF II Impact Evaluation is planned for the period
2007-2010. This TOR is limited to the second round of data collection, tentatively
planned to take place between April and November, 2009 with the possibility of an
extension to a third year subject to satisfactory performance by the consultant.
11.0 PAYMENT SCHEDULE Payments to consultant will be effected in accordance with the survey operational plan that will be prepared by the consultant.
7
12.0 OUTPUTS
1. Clean, preliminary data sets after completion of data collection in each district,
and complete data (including those households that are tracked to other
locations) within one month of the end of follow-up data collection;
2. Electronic versions of all survey instruments;
3. Training / enumerator manuals;
13.0 REPORTING REQUIREMENT The consultant shall report to the Monitoring and Evaluation Specialist for TASAF on behalf of TASAF Management Unit.
8
1
Cash Transfer Programme for OVC
Operational and Impact Evaluation of Phase 2
Terms of Reference
June 21, 2007
This version of the TOR up-dates last version dated in 2005. It contains the main
agreements for the design and implementation of the impact evaluation. Specific details
regarding some key operations of the programme are stated in order to clearly reflect the
results expected from the evaluation. The TOR complement the Evaluation Framework
Document (EFD). The EFD presents methodologies and instruments to be implemented
in the evaluation.
A. BACKGROUND AND PURPOSE OF THE EVALUATION
1. Background
Kenya is a country of 34 million people, from which around half are children (under 18
years old). Approximately 15 million are defined as ultra poor, meaning living below a
poverty line of US$ 1 per day. The proportion of Kenyans living in poverty has not been
declining and has actually increased in absolute terms. The crisis of HIV and AIDS has
also worsened poverty in Kenya. It is estimated that 6.2% of Kenyan adults are infected
(1.15 million people), and an estimated 120,000 children are infected. Poverty is mostly
found in: i) the poorest people living in arid areas where the economy revolves around
pastoralists, ii) people who were otherwise poor anyway who have been affected by HIV
and AIDS to the point whereby the economically active members of the household are
either ailing due to HIV/AIDS or have died, and iii) people living in urban slums around
cities and large towns where transient and migrant populations live.
The poverty of orphans and vulnerable children became the subject of discussion in the
course of the parliamentary elections towards the end of 2002, with many parliamentary
candidates pledging to allocate more resources to this group if elected. Approximately
half of those elected, including many cabinet members and senior opposition figures
signed this pledge. Commitment and action has been forthcoming.
HIV/AIDS was declared a National disaster in 1999. The most affected population are the
children. The number of Kenyan children who are orphaned or made vulnerable has
increased as a result of AIDS. In 2003, an estimated 11 percent (1.7 million) of all
Kenyan children less than 15 years had been orphaned according to the Kenya
Demographic and Health Survey (KDHS) compared to 9 % in 1998. Of these, 37 % are
estimated to have been orphaned due to AIDS1. Nationally, 9 % of children have lost
their father, 4 % have lost their mother, and 2 % have lost both parents. Although no
2
recent survey has been undertaken, it estimated that there are currently 2.4 million
orphans in the country.
In recognition of the above issues, the Government of Kenya is in the process of
developing a National Policy and a National Plan of Action for OVCs, a key aspect of
which is the provision of a direct cash subsidy to families caring for OVCs.
Most OVC live in poor communities with elderly grandparents and poor care givers who
can hardly meet their basic needs. Others live in child-headed households where they
have to fend for themselves. The family is the natural and basic unit for growth and
development of children. Given the long term benefits of raising children in families and
the infeasibility of providing institutional care, it is important that interventions for OVC
focus on facilitating community fostering. Strengthening the capacities of the
households/communities to be able to take care of OVC has been identified as the key
priority area in responding to the situation of OVC in the country. It is in this spirit that
the Cash Transfer Program for OVC is taking place to support poor households taking
care of OVC which Phase 1 was introduced in 2004.
Implementation
Phase 1
The Government of Kenya (GOK) submitted a proposal in 2004 to the Global Fund for
HIV, TB and Malaria, a key component of which was to fund the development and
expansion of a cash subsidy scheme for the most vulnerable children. However, the
proposal was not funded; a key weakness of the submission was that it proposed a
programme that had never been tried in Kenya and that there was no base on which to
support the viability of the programme. Therefore, The Department of Children Services
–DCS from the Ministry of Home Affairs –MOHA-, with assistance from UNICEF,
embarked on a bold initiative to demonstrate the feasibility of such a welfare system in
the country. As of December 2004, 500 households in three locations each in the districts
of Garissa, Kwale and Nairobi started being provided with KSh. 500 (approximately
$6.50) per OVC per month on what has been called the phase 1 of a pilot learning process
with the objective of informing the design of a larger scale pilot.
In April 2005, the Ministry of Home Affairs (MoHA) and UNICEF hosted a pre-pilot
review workshop to distil key lessons learned and to facilitate the shaping of a larger pilot
project with a design that could form the basis for nationwide scale-up. Participants
included teams from each district (comprising community representatives, members of
the Area Advisory Council and the District Children‟s Officers), officials from the
Deartment for Children‟s Services (DCS) and other Government agencies (the Ministry
of Health, Ministry of Education, National AIDS Control Council, among others), and
interested partners (including SIDA, DFID and the World Bank). The results of the
workshop showed that the pre-pilot had a positive impact on the welfare of the
beneficiaries mainly in terms of access to education, health, and nutrition. It also stated
that the expenditure of funds was mainly on items such as school uniforms, textbooks,
food, and cooking oil. However, the amount of funds was not enough to cover for all the
3
families‟ basic needs, especially when transaction costs involved in accessing funds was
taken into account (especially transport costs). There has been very little leakage and, as
it has been said, monies have been efficiently spent in a targeted way upon uniforms,
food, rent, drugs demonstrating the poor‟s capacity to address their own problems when
supported to do so.
As phase 1 of the pilot was developed, a growing political pressure was promoting the
expansion of the programme to other areas. As part of the pilot, the DCS started
implementing the learned lessons in 10 additional districts (Bungoma, Trans Nzoia,
Nyandarua, Nyeri, Nakuru, Meru North, Siaya, Kisii Central, Mombasa, Machakos). The
purpose of implementing the programme in these additional districts was to test the
capacity of the government to develop such a programme in a larger number of districts.
2,500 households were targeted. In total the number of beneficiary households reached in
Phase 1, were 3,000.
The operational modalities for this small pilot were not of a type that could be used to
scale-up to national scale. Funds were being transferred from a UNICEF bank account to
the Ministry of Home Affairs and from there to a government account at district level
from where District Children‟s Officers withdrew cash and, usually in great secrecy for
security reasons, or alternatively with armed security escorts provided by the police force,
went to targeted houses to disperse cash. Apart from the difficulties of moving cash
around in this way, there is no way that the usually solitary District Children‟s Officers
could ever hope to scale-up their personal delivery to all targeted households district-
wide. Besides, they have their usual jobs to carry out which is mainly in the realm of
managing children who are in trouble with the law.
However, jumping in at the deep end and actually starting to dispense cash to targeted
very poor families in a number of locations had the intended effect of moving the
theoretical discussion that had been going on for around two years to the realms of how
to move from an existing programme to a programme design that could be scaled-up.
Phase 2
The Pilot for the Cash Transfer Programme for Orphans and Vulnerable Children is
scaling up from the initial 3,000 OVC (phase 1) to around 8,000 (phase 2) during 2007.
Apart from the initial 13 districts the programme will also be piloted in 4 new docor
funded districts in the Nyanza Province (Kisumu, Homa Bay, Migori and Suba) where
the HIV/AIDS prevalence is higher than in the rest of the country and in 2 additional
government funded districts (Embu and Busia).
Negotiations were taking place with commercial banks in Kenya and the Post Office with
regard to how cash could be transferred through them with a low overhead cost to their
remote locations from which targeted households could withdraw their money. At the
same time experts in setting-up cash transfer programmes of national scale were being
recruited and familiarized with Kenya to refine targeting procedures, design management
information systems, develop operational manuals for use at local level, and to work
intensively with the DCS to develop a dedicated secretariat for the management of the
4
expanded pilot programme, known as phase 2. A major strategy for the strengthening of
the GOK‟s capacity has been put in place.
The primary objective of this phase of the pilot phase is to evaluate the potential role of
cash transfers as an instrument to keep orphans and vulnerable children within their
families and communities. Other objectives include promoting better nutrition and health
and school enrollment, attendance and retention. Important improvements in the design
of the different processes have been made including the targeting process and the
delivery mechanism.
It is expected that the design of the programme will be adjusted with the lessons learned
from the implementation of Phase 2 and the results from the Operational and Impact
Evaluation. Emphasis is placed on also informing the design of an elaborate monitoring
and evaluation system for use in a scaled-up cash subsidy programme. The outcomes of
this process will also inform the National OVC Policy development in regards to
community-based safety nets for orphans and vulnerable children. The information will
be especially invaluable for partners and donor agencies looking to support such an
initiative.
The target population of the programme is 100,000 households at a national scale. With
an average of 3 OVC per household the coverage would be of around 300,000 OVC.
The Operational Manual
An Operational Manual (OM) has been developed to describe the operations of the
Programme. Its purpose is to provide a set of instructions on the organisation,
procedures, and resources for an efficient and effective implementation of the Programme
in order to:
Enable stakeholders to understand the scope, content, organisation, and
activities of the Programme.
Indicate the performance expected of the management, operations and
personnel.
Ensure that the requirements for transparency, equity, compliance and
accountability are met.
Guide the operations of the functions to be performed under the project to
ensure consistency, timeliness and accuracy.
The manual provides all guidelines and instruments being implemented in the
programme. However, if the need arises for it to be modified or adjusted, the Central
Programme Unit in charge of the Programme is responsible of up-dating it. Changes
should be made after being discussed and if the pertinent stakeholders are in agreement.
For this purpose, proposed changes will need to be circulated and finally approved by the
National Steering Committee for OVC.
The existing OM is dated in November 2006. It reflects the operations and instruments
being implemented in Phase 2 in the 7 donor funded districts (Kwale, Garissa, Nairobi,
5
Suba, Homa Bay, Kisumu and Migori). At the end of Phase 2, the OM will be up-dated in
order to reflect the learning lessons from Phase 2 and the decisions done at the design and
implementation level for the next phase. These terms of reference and the evaluation are
closely attached to the implementation of the OM in the 7 mentioned districts.
Programme Objectives
As stated in the Operational Manual, the main objective of the OVC-CT programme is
to provide a social protection system through regular cash transfers to families living with
OVCs in order to encourage fostering and retention of OVCs within their families and
communities and to promote their human capital development.
Based on the overall objective of the project, the specific objectives for the household
beneficiaries are as follows:
Education:
Increase school enrolment, attendance and retention for 6 to 17 year old children 2 in
basic school (up to standard 8th
).
Health:
Reduce the rates of mortality and morbidity among 0 to 5 year old children3, through
immunizations, growth control and vitamin A supplements4.
Food security:
Promote household nutrition and food security by providing regular and predictable
income support.
Civil registration:
Encourage caregivers to obtain identity cards within the first six months after
enrolment
Encourage caregivers to obtain birth certificates and identity cards for children
2. Purpose of the evaluation
The purpose of the evaluation is to establish the efficacy and efficiency of the OVC cash
subsidy pilot project during phase 2. Specifically the evaluation will look at determining
the effectiveness of the processes used in the pilot with regard to:
the targeting and eligibility criteria
cash disbursement and accountability mechanisms
the transaction costs associated at each stage of the project
subsidy utilization
the effect that the cash subsidy has had at the community, household (including
foster families) and individual child levels
2 Children up to 17 years old could still be enrolled in basic school
3 Health focuses on immunization, nutrition and children illness
4 As discussed and agreed with MOH as per health regulations
6
The impact of conditions related to the cash subsidy provision
B. KEY EVALUATION QUESTIONS
The overall objective of the evaluation is to answer the following questions:
Are cash transfers reaching the most vulnerable children and having a substantial
impact on their welfare (both human development for the child and wider social
benefit for the household)? How much of an impact are cash transfers having?5
Does the impact justify the cost of the programme? Would a national programme
be affordable and fiscally sustainable? On that basis, should the programme, or a
variant of it, be scaled up to a national level?
If the programme is to be scaled up, which aspects of its operation must be
modified or strengthened for it to operate effectively at a national level? Which
aspects of good practice should remain the same and be replicated?
What is the impact or incentive effect of imposing conditions on beneficiaries,
versus not imposing conditions? What is the cost of doing so, for both
households and the government? Does any additional impact warrant the
additional cost? If households fail to comply with conditions, why is this so?
The evaluation focuses on three aspects. First, to evaluate the welfare and economic
impacts of the pilot amongst those who benefit from it; second, to evaluate the
operational effectiveness of the pilot including cost evaluation; and third, to evaluate the
extent to which the programme reaches those in greatest need (targeting effectiveness).
The evaluation will review the following specific questions:
The disbursement mechanisms -How was the cash disbursed to the families?
Was the method used to channel the funds to the beneficiaries cost effective and
safe? Who was responsible for disbursing the cash was this an effective
mechanism-How long did it take for households to access the money? Were any
delays encountered and why? Costs incurred by the participating households to
access the cash subsidy. How was the disbursement of funds monitored? How
much did it cost to monitor the disbursement (personnel, time, financial) Was the
community knowledge and familiarity and access to with banking systems and
processes
Cash subsidy utilization- Who received the cash subsidy within the household
and who was responsible for decisions regarding its use? Were the children
consulted in the use of the cash subsidy? Were their priority needs recognized
and prioritized? If not, how were the choices made? What was the money spent
on other than food and clothing and specifically did any of the families spend the
money on ARVs. What was the health expenditure on?
5 The original proposal for the evaluation also asked: is cash the most appropriate instrument for achieving
these welfare impacts?
7
Implementation capacity? – Is there sufficient government capacity to
undertake the activity- how much of the District Children‟ Officer‟s time is spent
on this activity- technical skills and knowledge and understanding of the project
to effectively implement it. What is the headquarter capacity to oversee the
activities being implemented in the three areas. Capacity of the Area Advisory
Councils (AAC) to monitor the activities.
Social impact “social footprints”- what effect the cash subsidy has had on the
status of the household in the community – the family dynamics with regard to
control of resources- have changes been noted in the power relations within the
household as a result of the cash subsidy,
Differences encountered in the project areas with regard to cultural practices,
geographic location that hindered or facilitated the processes involved.
Is there any improvement to the child’s situation? Health status of under 5s,
School enrollment, child labour, early marriage, Birth registration, Child school
attendance, was the child fully immunized? Record of attendance health facilities
etc…
Did the cash subsidy pre-pilot have any effects on the market with regard to
prices of commodities etc? Would it be possible to establish how much of a cash
subsidy would lead to changes in the markets
Did the cash subsidy conditionality lead to any changes in community demand
for services (health education); were facilities able to cope with the increased
demand?
The evaluation will also focus on the extent to which funds may be diverted away from
their intended recipients and, measure the efficacy of support mechanisms and changes in
the livelihoods of OVCs.
Conditions vs. no conditions
Discussions on the design of cash transfer programmes in Africa have recently focused
on question of imposing conditions on those qualifying for transfers. Cash transfers that
are conditional, for example, on school attendance or taking young children for regular
health checks have the potential to both enhance children's development and also reduce
income poverty. Many of the established programmes in the Caribbean and South
America impose such conditions on recipients6 and have demonstrated significant
impacts in education, health and nutrition indicators. However, it is not clear in the
literature whether it is the effect of the cash transfer, or the conditionality itself which is
leading to the increased school or clinic attendance.
Regarding imposing conditions in Africa, some argue that the Latin American and
Caribbean experience cannot directly serve as a blueprint for low-income African
countries7. In part the argument cites supply-side constraints, especially in health and
education infrastructures. Supply-side deficits are especially significant in rural areas
6 Barrientos, A., and DeJong, J., (2004).
7 Schubert, B. & Slater, R. (2006). Social Cash Transfers in Low-Income African Countries: Conditional or
Unconditional? Development Policy Review, 2006, 24 (5): 571-578
8
with poor access to services when distances to facilities are significant, the transport
infrastructure is deficient, basic drugs and other material resources are in short supply and
the problems are often compounded by high levels of teacher and health personnel
absenteeism.
A key consideration with respect to conditions versus no conditions is the extra cost of
maintaining a management information system that requires a large amount of data in the
first instance to help decide whether a family is eligible or not, and then to monitor
compliance with any conditions that the authorities wish to implement. The case against
conditions is that the extra cost of the management information system is not worth it.
Savings from forgoing the system could be dispersed to more poor people. The case for
the management information systems is that they improve the efficiency of targeting and
improve the uptake of social services to such an extent, that it is worth maintaining them.
The Decision taken by the Government of Kenya and the main stakeholders is to test a
programme with conditions and compare it against a programme without conditions. The
following chart presents the options for operations to be implemented as part of the
scheme:
1. Training at enrollment. As part of this registration process, the selected caregivers
are trained on the operational processes regarding the programme and are told what the
subsidies should be used for (e.g. health, education, nutrition)
2. Monitoring school and health attendance. The programme will periodically collect
information on the attendance to health and school facilities.
3. Case management. With the information from the monitoring process, the programme
develops a case management process in order to enquire about the reasons for non
attendance and to encourage attendance to health and school facilities.
4. Monetary penalty. A determined amount is deducted for non compliance to the
conditions.
5. Expulsion. After a determined number of periods of non compliance, the beneficiary
households are expelled from the programme.
The following table presents the agreement reached:
WITH CONDITIONS
GROUP
WITHOUT CONDITIONS
GROUP
Training at enrolment Yes Yes
Monitoring Yes No
Case management No special case management
will derive from the
monitoring information.
The normal procedures
No. The normal procedures
from the DCS will be
applied as provided by The
Children Act.
9
from the DCS will be
applied as provided by The
Children Act.
Penalties due to non
compliance
Yes No
Exit due to non
compliance
Yes No
Beneficiaries will not be subject to conditions if shortages of health and school services
exist. Justifiable cases for non compliance with the conditions are also established.
C. EVALUATION PROCESS AND METHODS
The evaluation strategy will use a quasi-experimental research design (including pre/post-
test, case reference and control community) to gauge performance of the pilot
programme.
Given that the evaluation will compare a programme with conditions against a
programme without conditions, information will be collected from 4 treatment and
control groups as follows:
TREATMENT WITH CONDITIONS
CONTROL
TREATMENT WITHOUT CONDITIONS
CONTROL
The following table presents the districts with conditions and those without conditions:
Districts with conditions Districts without conditions
Kwale Garissa
Kisumu Suba
Homa Bay Migori
Nairobi (one sub-location) Nairobi (one sub-location)
2 locations are selected as treatment locations and 2 locations are selected as control
locations. The decisions referring to the difference in treatment for the “with condition
group” and the “without condition group” will remain in those locations as long as the
evaluation is taking place. This also means that the control locations are not supposed to
be included in the programme before the information for the first follow up of the impact
evaluation of Phase 2 is collected.
The evaluation has two measurements in time:
1. The Base Line, for which collection is ongoing (March to June 2007)
10
2. One Follow-Up (March to June 2008).
The moment for the follow up can be changed depending on the performance of the
implementation of the programme. If, for instance, all payments are done with the
periodicity expected, the follow up should be done 12 months after the base line. If that is
not the case, or other problems occur that affect the performance of the programme, the
situation should be analyzed and decisions on the time for the follow up should be made
accordingly.
Quantitative data gathered through this process, together with data from the 2002
GOK/UNICEF baseline surveys in the same communities, will be used to test out, review
and document targeting modalities, efficacy, unit costs, efficiency of monitoring, role of
institutions including communities and the government, administrative considerations,
scale and partnerships.
The consultant will also be expected to collect qualitative data and elicit information
from program managers, community workers and communities at large through in-depth
interviews and focus group discussions.
The operations research methodology would be used to evaluate and predict the effects of
the „systems analysis‟ undertaken above on the situation of the OVCs, especially in terms
of outcomes in their survival (birth registration, immunization, nutrition, etc)
development (IECD, school attendance, etc) and protection (sexual abuse, child labor,
etc) rights. It is envisaged that through the additional use of qualitative research
techniques, the evaluation component will gather beneficiary (and non-beneficiary)
perceptions of the social safety net and gauge the extent to which they interact with
informal risk mitigation and coping schemes.
D. OUTPUTS
1. Evaluation Framework Document
2. Comprehensive Base line report; raw data sets (in SPSS format) and completed
questionnaires
3. Comprehensive evaluation report; raw data sets (in SPSS format), completed
questionnaires and monitoring and evaluation instruments.
E. SELECTION OF CONSULTANTS
The consultancy firm Oxford Policy Management was selected after a tendering process
following UNICEF rules and regulations. The consultancy firm was contracted to
undertake an independent evaluation of the OVC Cash Transfer Programme. The first
contract was signed in 2005.
11
F. STEERING COMMITEE FOR THE EVALUATION
Conformation
This evaluation is managed by a Steering Committee comprising of the undersigned
individuals representing the following organizations:
- Department of Children Services (OVP and MOHA)
- UNICEF
- DFID
- World Bank
Responsibilities
The following responsibilities are assigned to the Steering Committee:
1. Approve the TOR for the evaluation.
2. Manage the evaluation process including feedback within their respective
organizations.
3. Receive and discuss the answers to the evaluation questions delivered by
the consultant evaluators
4. Support the refinement of the programme ensuring good use of the
evaluation results
Secretariat
UNICEF will undertake the secretariat of the Steering Committee for the Evaluation
under the supervision of the Child Protection Section and the SPME.
Yemen Social Welfare Fund
Consultancy Service
Design of a Monitoring and Evaluation System
Terms of Reference
A. Introduction
Country Background
The Republic of Yemen, with a population of about 22 million people, and a GDP per capita of
about US$650, is among the poorest countries in the world. Forty percent of the rural population
and 20 percent of the urban population live below the poverty line. Despite steady progress at the
macro level, Yemen has some of the worst social indicators in the MENA region: a 28 percent
literacy rate among women, 18 percent malnutrition, girls primary enrollment at 55 percent, and
access to safe water at 31 percent. Poverty is pervasive in rural areas, where much of the
population resides. The dispersion of the population and the difficult topography pose a serious
challenge to delivering social services: Yemen has around 35,000 official villages (with
approximately 100,000 human settlements), many with less than 100 households, which makes
the provision and maintenance of social services very expensive. The country’s fiscal revenues
depend heavily on the oil sector (for nearly 75 percent of income) and production forecasts
indicate that existing reserves will be depleted in 8-10 years. Yemen is an oil-dependent
economy with oil revenues projected to decline and with no good short to medium-term
alternative income sources to deliver social services.
Social protection is a main pillar in Yemen’s poverty reduction strategy. However there is no
comprehensive risk reducing mechanisms (such as labor market policies and programs). The
government has established a number of safety net mechanisms, but most of the existing safety
net programs have limited coverage, are not well targeted and are inefficient in terms of providing
adequate protection for those in need, and have problems of management, governance and high
administration costs.
The World Bank has been involved in social protection in Yemen for over ten years. Most
recently the Bank supported the Government of Yemen to review its Social Protection Strategy
and proposed several recommendations in support of further effectiveness and efficiency of
Yemen’s social protection mechanisms. Key recommendations of the recently developed Social
Protection Strategy included some reform elements of the country’s cash assistance mechanism,
the Social Welfare Fund.
Program Background
The Social Welfare Fund (SWF) is the largest public cash transfer based social safety net in
Yemen. It was established in 1996 by the Ministry of Social Affairs and Labor and has gradually
expanded to provide cash transfers to some three million poor and vulnerable Yemenis. In April
2008, and in response to the recent food crises, the Government of Yemen (GOY) mandated an
increase of the caseload to cover up to seven million Yemenis, and doubled the cash benefits.
The SWF has suffered from a combination of low benefits and poor targeting, resulting in
chronically low coverage of the poor and little impact on overall poverty. The ongoing World
Bank policy support for Yemen Social Protection Strategy highlighted in its recommendations
Yemen’s need to improve risk mitigation and cushion the adverse impact of high and more
volatile food prices on poverty. The SWF is foreseen by the GOY as the key instrument for
redistribution of the expected savings from the energy subsidy reform once implemented. As a
result, the government has authorized a program of fundamental reforms to SWF, including
improving the targeting of the poor, strengthening the service delivery capacity, and
implementing a new legal and policy framework. The implementation of reforms and the
expansion of the program is entering a critical period requiring significant technical assistance,
capacity building and training support.
The SWF is financed by the Government, and operated through a central office in Sana’a, 22
branch offices located in each governorate, and 172 district teams. The total number of staff is
1613. Limited technical assistance has been provided by the EC for several years, and recently
the World Bank has begun to provide complementary guidance on program targeting and
supporting the national survey targeted to nearly doubling the current beneficiaries among poor
Yemenis suffering from high food and fuel prices.
Monitoring and evaluation of the program impact has been identified as a key weakness of the
program. The Government of Yemen is seeking World Bank support to its efforts for the
establishment of a best-practice Monitoring and Evaluation (M&E) system, and to develop the
capacity of SWF staff in the area of impact evaluations and systematic monitoring and evaluation.
The World Bank received a Japanese trust fund (CTF) to support responding to the SWF need to
develop its monitoring and evaluation capacity.
B. Objectives of the Consultancy
The objectives of the consultancy are:
to develop a M&E system for the SWF based on the program objectives as mandated by
its law,
to define performance and impact indicators that will enable assessing the SWF program
effectiveness and efficiency and inform decision making process towards continuous
improvements, and to enhance its impact as a national safety net instrument;
to support capacity building of SWF staff in the design, implementation, and maintenance
of the program M&E system.
C. Scope of Work
The consultant will specifically undertake the following tasks:
o Undertake a situational assessment and formulate a work program designed to
achieve the expected consultancy objectives; o Review current planning, monitoring and evaluation practices of SWF, and describe
resources, deliverables and expected outcome of SWF. Highlight the implications of
these practices on targeting, efficiency and effectiveness of SWF programs and
processes;
o Formulate sets of key management, process, and impact indicators that will be
used to assess the effectiveness and efficiency of the SWF program. Specify
indicators at different levels as administrative level, district level and beneficiary
level if possible; o Propose a reformed Monitoring and Evaluation system to inform the planning and
decision making process, the design and cost-effectiveness of SWF interventions,
focusing on the process evaluation and assessment of targeting accuracy;
o Develop or harmonize and consent appropriate data collection tools for the
program monitoring and M&E mechanisms / procedures. o Train SWF relevant staff at the central office and branches on the implementation and
maintenance of the developed M&E system;
o Assess the capacity of the SWF staff responsible for monitoring and evaluation,
and provide recommendations for any further training/follow up needed to ensure
adequate capacity is place to enable them to take the lead in M&E of the program;
o Define needs for surveys and/or studies that supplement the routine data
collection for purposes of impact evaluation;
o Develop terms of reference for an impact evaluation study that will build on the
baseline data collected by the ongoing nation-wide social survey;
o Develop M&E Operation and Procedure Manuals detailing all tools, methods and
techniques relating to M&E activities;
o Collaborate with the MIS department to develop procedures and ensure software
availability for electronic recording and transmission of all required M&E data;
o Create an electronic database for monitoring the performance of the SWF;
o Prepare an end of consultancy report including findings, achieved tasks and
activities, and recommendations. The final report is to include all documents,
tools, and guidelines developed under the consultancy.
D. Deliverables
The following deliverables are required.
A consultant report describing all activities undertaken under the consultancy and
the consultant’s findings and recommendations for future action;
Documentation of all elements of the agreed M&E system;
M&E guideline with sample templates and forms;
An outline of proposed surveys/studies to be undertaken to enable program
monitoring and evaluation;
Terms of Reference for the implementation of an impact evaluation study;
Simple electronic database for the performance of the program
.
E. Work Program and Timetable
The consultancy is for 75 working days spread over the period January 1 – June 30, 2009,
including three field trips to Yemen. First trip (10 days) is expected in January 2009, the second
field trip (15 days) is expected in March 2009, and the third trip (10 days) in May 2009. The final
report, including all deliverables are expected no later than June 20, 2009.
The consultant will report to Ms. Afrah Al-Ahmadi, Sr. Human Development Specialist, TTL, at
World Bank , Yemen Country Office ([email protected]).
The consultant will be assisted by a national consultant who will extend his support to SWF
throughout this consultancy and beyond. The consultant will work very closely with the SWF
staff, more particularly the monitoring department staff. The consultant will also coordinate with
the EU project team located in the SWF premises, the Social Fund for Development, which is
implementing an institutional capacity building project for the SWF.
F. REQUIRED QUALIFICATIONS AND EXPERIENCE
Post-graduate degree in social sciences, development studies, economics, or relevant
discipline, and formal training in monitoring and evaluation
International experience in designing and conducting monitoring and evaluation systems,
survey designs, survey analysis, and impact evaluation studies.
Proved record of capability in carrying out similar task, and a minimum of 5 years
experience on the design of M&E systems
Fluency in written English Language
Familiarity and experience with cash assistance programs would be an advantage
REPUBLIC OF TUNISIA
SKILLS DEVELOPMENT AND COMPETITIVENESS POLICY LOAN (SDCPL)
Monitoring and Evaluation
Terms of Reference for Consultancy
Background
Currently, the main preoccupation of the Tunisian Government is to create enough jobs to absorb
a rapidly growing and more educated labor force. Aligned with these priorities, job creation and
skills development have a prominent role in the country‟s 11th Development Plan (2007-2011).
In this plan, the Government emphasizes that more jobs will have to come mainly from more
investments in the private sector and faster growth. Good quality jobs, moreover, require that
these investments take place in high-value added sectors and are accompanied by innovations that
boost productivity. Tackling the multi-faceted aim of both more jobs, and better jobs, requires
parallel developments in various spheres of the economy, in terms of macroeconomic and
investment policies, labor regulations, education policies, and the corresponding social protection
system.
It is in this context that the Government of Tunisia (GOT), with the support of the World Bank,
has decided to embark on a cross-sectoral Program to reform both the competitive environment
for companies and the formation of professional skills. The Program aims at accumulating human
capital, and promoting innovation and productivity growth, with a view to improving
international competitiveness and decreasing unemployment rates in the country.
The main focus of the program will lie on a reform of the vocational education and training
(VET) system, comprising actions in four policy areas. While the concrete detail of the actions is
still being agreed, the strategic lines include:
1 Partnership between the training system and the private sector in priority sectors. The
government has been developing this partnership, conceived during the MANFORME reform in
1995, and implemented in stages with partial results until now. An inter-ministerial committee
with strong participation of private sector representatives will be set up to adopt joint policy
measures to:
promote private participation in the delivery of in-service and pre-service training, and
define the regulatory and legal framework to consolidate private (entrepreneurial and
professional associations) participation in the management of Training Centers.
In addition, the Ministere d‟Education et Formation (MEF) envisages two demand-side measures
to support the take-up of training in priority sectors (e.g. sectors with skills deficits):
a scholarship for VET in public institutions, covering boarding, materials and a pocket
money, directed at students in need (where need will be assessed through parental
income, fixing a strict eligibility limit.) The scholarship has been decided already.
a voucher for VET in private institutions, envisaged to be designed similar to the
scholarship. The voucher and its design are still under discussion.
2 Reform of the Financing mechanisms used for pre-service and in-service training.
Tunisia‟s main instrument to promote in-service training among enterprises is based on a training
levy (Taxe de la Formation Professionnelle, TFP), which is a payroll tax1 designed as a tax rebate
system administered by the CNFCPP. Manufacturing firms pay 1% of their payroll and non-
manufacturing firms pay 2%. The amount of the tax rebate is calculated on the basis of the
enterprise contribution to the TFP and can be used to finance the identification of training needs,
training and operation costs of training departments.2 The complexity of procedures to obtain tax
rebates has been a limitation to the effectiveness of this instrument, especially for small and
medium enterprises. 3 Given the limited impact the tax-rebate system has had on small and
medium enterprises, an additional instrument, the Programme National pour la Formation
Continue (PRONAFOC), was introduced in 1995 to promote in-service training among SMEs
with less than 100 workers. 4
To increase efficiency and equity in the tax collection and the use of financial resources, the
government will conduct an analysis of the current financing mechanisms, an analysis of the cost-
effectiveness of the in-service programs and the existing criteria for training tax collection. The
results of these analyses will be used to design a reform of the financing mechanisms.
3 Promote responsiveness to changes in the national and global economy. Under this
heading, the GOT has engaged in a series of reforms since the early 1990s, conducive to prepare
local firms to position their goods and services in EU markets. Core of this “Mise a Niveau”
strategy has been the reform of the VET system to make it demand driven and responsive to local
and international jobmarket needs. To consolidate these reforms the GOT will:
implement ISO 900 certification for all institutions of the VET system: Training Centers,
public and private, ATFP, CENAFIF, CNFCPP.
Introduce a mechanism to promote accountability for results, at all levels of the VET
system: Training Centers, ATFP, CENAFIF, CENFCPP. To define this mechanism, the
government will develop indicators to measure results to be achieved by the relevant
institutions on a yearly basis.
Develop an economic observatory to be articulated with the current labor observatory to
serve enterprises and training centers in understanding global and local economic and
technological trends, and improve their capacity to adapt their training programs
correspondingly.
4 Promote a Life-Long Learning Policy Framework. This is a long-term goal, and the
government has already initiated discussions on the policy options to be followed. In particular,
the government will develop a unified framework for qualifications and quality assurance for all
levels of education - basic, VET and higher education -.
1 Manufacturing firms pay1% of their payroll, and non-manufacturing firms pay 2% of their payroll. Export-oriented
and agriculture firms are exempted from paying the TFP. 2 Eligible expenses to be financed by the TFP rebate include services for the identification of training needs,
elaboration of training plans, actual delivery of training for personnel, and to cover the costs associated with the
internal structures to the firm devoted to training of its personnel.
3 Companies pay the TFP out of their payroll, undertake training activities, validate these expenditures with CNFCPP
and get their tax rebates based on eligible expenditures. Small enterprises with small payrolls pay small amounts of
TFP and since their training expenditures eligible for tax rebates are limited by the size of the TFP paid, their incentive
to undertake training and apply for a tax rebate is less compelling. 4 The focus on SMEs was important since they represent the largest segment of enterprises and employment in Tunisia;
SMEs can benefit from PRONAFOC either as individual firms or as a group of firms and can obtain financing for
assessing training needs diagnostics, training plans, the provision of training services and evaluating activities.
Methods and Scope of Services of the Consultancy
The consultant should design the Framework for the Monitoring & Evaluation of the
SDCPL („Results Framework‟), i.e. select a set of Monitoring indicators to track progress
on the Program; and prepare an Impact Evaluation. It is understood that activities in all
four areas of the SDCPL can be monitored, but only some of them, such as the
scholarship and the education voucher under area 1) can be impact evaluated.
This includes:
Designing a Framework of Monitoring indicators to track progress on Program areas
1-4;
Advising, managing and using the human resources the MEF has tasked with
assisting the work on the Monitoring Framework;
Together with the Project Management Unit (PMU) at the MEF, identifying those
activities under the SDCPL that can be impact evaluated;
Designing the targeting and implementation of these activities, thereby determining
treatment and control groups for a pilot;
Designing questionnaires for a baseline and a follow-up survey in co-operation with
the PMU and the Observatoire d‟Emploi (who will be tasked with the survey work);
Instructing and cooperating with the Observatoire d‟Emploi, to conduct the baseline
survey;
Drawing on the resources and building the capacity of the technical equipe of the
PMU at the MEF; and
Liaising with and reporting to the MEF‟s Project Management Unit (PMU), headed
by M. Brahim,Toumi;
Deliverables
Under this assignment, the consultant will be expected to deliver the outputs detailed below. For
his/her work, the consultant will be able to draw on free support through the technical equipe of
the PMU. He/she will also be able to draw on resources from the Observatoire d‟Emploi.
The Consultant should prepare a set of Monitoring Indicators to track the progress of the
Program:
o The set will comprise realistic and meaningful measures for inputs, outputs and
outcomes of all four areas of the SDCPL.
o It will also show current baseline values of these indicators, and ambitious but
achievable target values for the end of the Program;
o The Monitoring System will be the main instrument of measuring the
interventions under the Program. Therefore, it needs to be compatible and easily
mergeable with the baseline and follow-up surveys for any Impact Evaluation
(see point below). This includes using identifiers that are common to both
instruments, i.e. the same training center codes, postcodes, municipality-codes,
household keys, and regional keys. In this way, the Monitoring System and the
surveys can be merged at the interface of the common identifiers.
o Together with the PMU, as well as the Observatoire d‟Emploi, the consultant will
also determine the data sources and collection channels for regular monitoring,
from before the start of implementation of the Program.
o The World Bank team will share a framework for monitoring systems.
Baseline and Follow-up survey questionnaires to support the Impact Evaluation of the
SDCPL.
o It may be possible to establish a baseline for the scholarship/voucher
interventions by asking applicants to fill in a questionnaire as part of the
application. The consultant should lead the decision on this and decide, together
with the PMU and the Observatoire d‟Emploi, whether this is suitable. In any
case, the development of a baseline questionnaire for the scholarship bears
some urgency, as the intervention is about to be implemented.
o In any case, the consultant, in co-operation with the PMU, and the Observatoire
d‟Emploi (see point below), will choose a questionnaire for a baseline and a
follow-up survey of participants, to measure the impact of the evaluatable
activities of the SDCPL.
o The questionnaires should cover indicators for the intended impact of the
SDCPL, e.g. the take-up of training in priority sectors, individual labor market
success, salary and other welfare measures. It should also cover a comprehensive
array of socio-demographic background variables.
Cooperation with and instruction of the Observatoire d‟Emploi
o The consultant will draft Terms of Reference for the Observatoire d‟Emploi to
carry out the survey.
o He/she will also involve it in the dialogue on the impact evaluation and instruct it
on the SDCPL. Jointly with the Observatoire‟s statistical team, he/she and the
PMU will decide on a sample size and sampling strategy for the surveys.
A targeting mechanism and/or rollout plan for those activities of the SDCPL that can be
impact evaluated.
o The consultant will decide, jointly with the PMU, on an appropriate targeting
mechanism of the scholarship and the voucher interventions. It is currently
planed to determine eligibility according to parental income, clearly establishing
a limit that marks potential entry to the program.
o He/she will also identify if further activities could be implemented in a way that
would allow an impact evaluation. A potential candidate is the planned reform of
the enterprise-based financing mechanism for in-service training (area 2 in the
introduction). Roll-out in the form of a randomized experiment is highly
recommended, subject to the approval of the PMU.
An Identification strategy to evaluate the scholarship/voucher interventions
o Based on the chosen targeting strategy, and the consequent treatment and control
groups, the consultant will develop a strategy to arrive at a statistically sound
identification of the impact of the above interventions.
A plan to disseminate the results of the Evaluation
o Jointly with the PMU, the consultant will plan a strategy of disseminating the
results of the Impact Evaluation to the wider Government, Academia and the
public.
o The actual format of the dissemination process is subject to agreement by the
PMU and the MEF, but should include stakeholder workshops, cross-ministerial
presentations and dissemination of the data, international and national academic
publications and graduate school workshops in Tunisia.
A Manual including written procedures that will be used by the relevant institutions for
future monitoring of the SDCPL.
A contribution to the Program Document, summarizing the Results Framework,
comprising:
o the Monitoring Framework,
o the key steps of delivering the Impact Evaluation of the evaluatable components
of the SDCPL,
o The World Bank team can provide templates for this section of the Program
Document.
The World Bank team and the PMU will support the consultant at the beginning of the
assignment with all relevant information and knowledge gathered to date.
Qualifications & Requirements:
Higher level University degree either in Economics or similar quantitative discipline with
at least seven (7) years of experience in Impact Evaluation.
Proven experience and knowledge of labor economics and policy.
Proven analytical, problem-solving and quantitative skills, including statistical packages.
Proven ability to build capacity of government officials.
Excellent oral and writing skills in French;
Publication track record is desirable.
Excellent interpersonal communication skills, demonstrated ability to work cooperatively
with clients, and ability to liaise tactfully as a member of a multicultural team.
Proven ability to interact actively with clients.
Level of Effort
It is expected that the consultant perform the assignment within a period of 3 (three) months,
starting from the date of the contract signature.
The first ranked consultant should submit technical proposal containing timetable for the tasks to
be accomplished under the assignment, and should also take into consideration that the proposal
may be a subject to additional adjustment.