unicef ugandan ministry of education and sports …...mentorship approach and its potential for...
TRANSCRIPT
1
UNICEF – Ugandan Ministry of Education
and Sports BRMS Mentorship Project
Evaluation
Report
Dr Martin Prew
University of Witwatersrand School of Education
May 2014
2
Contents
Abbreviations and Acronyms ............................................................................................................ 4
1. Executive Summary ................................................................................................................... 5
2. Introduction .............................................................................................................................. 8
3. Background to the Project......................................................................................................... 9
3.1 Theory of Change ................................................................................................................... 10
3.2 Project Problem Statement .................................................................................................... 10
3.3 Project Outcomes .................................................................................................................. 11
3.4 Project Design and Implementation ....................................................................................... 11
4. Evaluation Methodology and Process ..................................................................................... 12
4.1 Review of documents............................................................................................................. 13
4.2 Use of documents and interviews to inform a cost of delivery proposition ............................. 13
4.3 Field Visits.............................................................................................................................. 13
4.4 Interviewing and Surveying Mentors ...................................................................................... 14
4.5 Stakeholder Interviews .......................................................................................................... 15
4.6 Project Metrics ...................................................................................................................... 15
4.7 Analysis of the Data ............................................................................................................... 20
5. Limitations on the Evaluation ................................................................................................. 21
6. Findings: Operational .............................................................................................................. 21
6.1 Relationship between Mentors and CCTs ............................................................................... 21
6.2 Interventions Driven by the Mentors ..................................................................................... 25
6.3 Impact on Colleges ................................................................................................................. 28
6.4 Impact on Schools .................................................................................................................. 30
7. Findings: Measuring Metrics ................................................................................................... 35
7.1 Metrics Measuring Impact on the CPTC.................................................................................. 38
7.2 Metrics Measuring Impact on Schools .................................................................................... 39
7.3 Metrics Measuring Impact on Learners .................................................................................. 41
8. Findings: Project Management and Cost Effectiveness ........................................................... 42
8.1 Preparation, Recruitment and Induction ................................................................................ 42
8.2 Project and Financial Management ........................................................................................ 44
8.3 Value for Money .................................................................................................................... 47
9. Findings: The Model ................................................................................................................ 49
10. Conclusions ......................................................................................................................... 52
11. Recommendations .............................................................................................................. 55
11.1 Recommendations for the Next Phase in Uganda ................................................................. 55
3
11.2 Recommendations for Replication of the Mentorship Model in other Countries .................. 58
References ...................................................................................................................................... 59
Annex 1: Time Lapse Analysis of Attendance at Focus Schools ....................................................... 60
Annex 2: Interview Tools ................................................................................................................ 63
Annex 3: Online Questionnaire for the Mentors ............................................................................. 68
4
Abbreviations and Acronyms
Boda boda Motorcycle taxis used across Uganda
BRMS Basic Requirements & Minimum Standards
CCT Coordinating Centre Tutor
CPD Continuous Professional Development (on-site development activities for
employees)
CPTC Core Primary Teachers’ College (colleges where CCTs are based)
DEO District Education Officer
DES Directorate of Education Standards
DIS District Inspector of Schools
EGMA Early Grade Mathematics Assessment
EGRA Early Grade Reading Assessment
MIS Municipal Inspector of Schools
MoES Ministry of Education and Sport
NAPE National Assessment in Primary Education
PLE Primary Leaving Exam
PTA Parent Teachers’ Association
PTC Primary Teachers’ College
SLA Service Level Agreement
SMC School Management Committee
TDMS Teacher Development Management System
VSO Voluntary Service Overseas (UK based human resource and development agency)
5
1. Executive Summary This formative evaluation of the UNICEF – Ministry of Education and Sport Project known as ‘the Basic
Requirements and Minimum Standards (BRMS) Mentorship Project’ was commissioned by UNICEF in
December 2013 to achieve a number of outcomes. The desired outcomes included:
Providing a formative evaluation half way through the delivery period of the project;
Providing a more summative evaluation of the work of the mentors based in the Core Primary
Teachers’ Colleges (CPTCs);
Providing baseline metrics for the future measurement of the project, as no specific project
baseline survey was conducted at the start of the project1; and
Assessing the methods and extent to which the impact of the project can be sustained and
replicated.
The outcomes expected from the BRMS mentorship programme in summary can be expressed as:
1. CCTs to improve their engagement with schools to provide professional development and
support (rather than acting as materials distributors and inspectors);
2. Improvements in teacher performance and community engagement with the school which in
the long term leads to improved completion rates and exam results;
3. Improvements in coordination of district education partners (district office, inspector, school,
teacher training college).
The evaluation used a mixed method approach. Data was collected against a number of metrics
related to impact of the mentors on the schools and CCTs as well as qualitative data exploring how
the mentors and other stakeholders experienced and learned from the project. These data are
collected from key project stakeholders including the main funder, 11 of the mentors, 37 CCTs and 8
senior managers from four of the CPTCs, and teachers and head teachers from 14 primary schools in
six education districts. The report uses the data to better understand the impact of the mentorship
model and other aspects of the project and inform the discussion on the elements of good practice
included in the model and how they can be sustained and replicated.
The report focuses particularly on the successes of the project while also discussing how the approach
could have been delivered more effectively. This helps inform and define a model for effective
implementation of a school improvement model which uses foreign and local mentors to coach and
mentor CCTs to support schools more effectively and thereby improve school performance. The
report ends with recommendations on the future implementation and sustainability of the BRMS
Mentorship approach and its potential for replication elsewhere.
The overall finding of the report is that the BRMS Mentorship Project was well conceived and, given
the constraints affecting it, had significant impact and serves as a potential model for sustained school
improvement.
This can be seen in:
• 6.9 percentage point increase in P3 NAPE literacy results in the 73 target districts by 2012 • 6.7 percentage point increase in P3 NAPE numeracy results in the 73 target districts by 2012 • Most target districts saw increases in Division 1 and 2 PLE results
1 The MoES and UNICEF were involved in a large-scale project to provide baseline data for the BRMS. This should have provided the baseline data for the project. However, the report was huge as well as not being specific to CPTCs, CCTs and their coordinating centres, and the focus schools, so it has failed to serve this purpose. Its results were not used by the colleges or mentors in their planning or delivery.
6
• 3 percentage point (4.6%) increase in primary school completion rates in the 73 target districts • 313 CCTs were able to provide improved support to over 2000 schools and 20,000 educators • 100% of sampled focus schools indicate ‘better’ or a ‘lot better’ support from their CCTs • 100% of CCTs report improved ability to support schools as a result of the project • 70% of head-teachers in sampled schools have improved practice according to their staff • 93% of sampled schools ‘happy’ with support they gets from their CCT; 0% not happy • 94% of CCTs believe relations between CCTs and between CCTs and their DPO have improved • 127 CCTs and 15 DPOs/colleges received vehicles to access schools • The total budget equates to US$1,750 being spent per school, or the training of 20,000 educators in a 3 day residential workshop.
Other key positive findings are:
The mentors were generally appropriately skilled and formed a formidable resource for the
project;
The local mentors present skills and knowledge which are different from, but complementary
to, those which were presented by the foreign mentors;
The mentors impacted on both the 313 CCTs (as of December 2013) and approximately 2000
of their schools leading to some important changes in the way that the focus schools work
and teach;
Many of these focus schools adopted key elements of child-centred learning;
Almost all the respondents expect key elements of the project’s impact on the CPTC and the
focus schools to be sustained;
The decision to make CCTs the point of entry was correct although there are concerns about
the future of this critical cadre of professional educators.
The report concludes that from the intervention a model of successful school improvement based on
colleges of education and CCTs can be defined. While this is a relatively expensive model it is judged
to be relatively cost effective given the impact it has had and can potentially have.
However, the report also finds that:
UNICEF and the MoES could have played a more structured and coordinated role in planning
and managing the project and its finances with positive effect on the project’s coherence and
impact. Had the project had a strong shared project plan supported by a specific project
baseline survey and a mid-term process evaluation, with clear project targets which were
shared by all stakeholders, the project would have achieved even more impact.
The lack of baseline data specific to the focus schools and CPTCs means that it is almost
impossible to measure the impact of the project on these schools and the CPTCs.
The report recommends that UNICEF in partnership with the MoES implements a two strand process
over the remaining two years of the project. This is detailed in Section 11. It involves support of the
intervention and impact to date in most of the colleges through a low-cost peripatetic support process,
while implementing a much more structured intervention in the remaining 8 core PTCs based on the
strengths of the model. The aim of the second strand would be to extend and explore the model
further and document it with rigour from the start so that there is reliable data to show impact on
schools, the CCTs and the CPTCs. This documentation is critical for future advocacy and replication of
the model.
7
8
2. Introduction UNICEF commissioned a formative evaluation on the implementation of a mentorship project it
designed and has been implementing in partnership with the Ugandan Ministry of Education and Sport
(MoES) in support of the MoES’s Basic Requirements and Minimum Standards (BRMS) Framework.
The four year project has been implemented in 15 Core Primary Teacher Colleges (CPTCs) across
Uganda over the last 2 years. In most regions the original model (see MoES 2010) came to an end in
December 2013. This first phase was based on the placement of an international or local teacher
education specialist as a mentor to a cluster of Coordinating Centre Tutors (CCTs), and less directly the
district and municipal inspectors of schools (DIS and MIS), while being based in CPTCs. Due to
staggered start times and extensions granted to some colleges a few of the mentors are continuing
their work in 2014.
The evaluation is meant to assist UNICEF and the MoES in determining how the second phase of the
project will be implemented over the next two years to maximise its impact and sustainability through
mainstreaming of the practices and the lessons learned. This, it is anticipated, will assist UNICEF in
refining this intervention as a model for school improvement. Finally, it is also designed to assist
UNICEF and the MoES in determining what aspects of the intervention are replicable elsewhere and,
more significantly, if the model is replicable in other parts of Uganda and possibly other countries. If
the report finds that there is value in replication elsewhere then the report needs to indicate how the
model would need to be modified to maximise adaptation and impact.
This evaluation is therefore designed to achieve four things: provide data to inform the setting of
metrics and an evaluation process for future evaluations (in the absence of a project specific baseline
survey when the project started); provide a mid-term formative evaluation of the project; present a
provisional summative evaluation of the work of the mentors on the project; and provide a basis for
the defining of the school support model. When multiple purposes are set for the same report there
is a danger that it may achieve none of them. The evaluator is aware of this danger and has
endeavoured to present data, findings, conclusions and recommendations which talk to all three of
these requirements.
To inform these analyses and recommendations a multi-pronged research process was undertaken in
order to collect data from all the main participants, recipients, partners and stakeholders of the
project in Uganda. As there was a perceived danger of these parties providing over-positive analysis
of the project it was also decided as part of the data gathering process that the evaluator should spend
time in four of the CPTCs and with two of the remaining mentors and their CCTs in undertaking school
visits. This allowed data to be collected from a sample of school stakeholders.
The data collection process was undertaken between December 2013 and April 2014. All the
respondents that the evaluator identified made themselves available for interview and over 60% of
the mentors provided detailed responses in the form of written responses to a questionnaire (see
Annex 3) and a sub-group made themselves available for both one-on-one and focus group interviews.
During the field visits the evaluator also interviewed stakeholders who were not part of the original
list of proposed respondents but who were willing to talk and had a particular story to tell or held a
particular viewpoint. One such group were district and municipal inspectors of schools working in
close proximity to a CPTC where a mentor was placed.
The report is structured in sections which present the background to the evaluation, the methodology
used to collect data and information to inform the evaluation and then presents the findings in a
structured format. Two sections look at the proposed metrics that could be used to measure the
impact of the project and the key elements of the model that is emerging from this initiative. The
9
report closes with conclusions and a recommendations section which is designed to provide UNICEF
and the MoES with the data it requires to make certain decisions about the future of the project and
the model, both in Uganda and beyond.
3. Background to the Project UNICEF, in partnership with the MoES, has been implementing an innovative model of CCT support
based on a mentorship model. It is designed to support the Ugandan government’s 13 BRMS standards
implementation, which are designed to increase child participation and survival rates in primary
schools and improve the quality of primary education through innovative projects. The success of
BRMS depends on the capacity of the primary school system to implement it and improve. This means
that the focus is on the four support systems impacting on primary schools. These are:
1. The district education office, which mainly acts as a coordination centre and liaises between
education and local government in the district;
2. The 23 colleges of education which have outreach capacity provided through the CCTs who
are based in the colleges but work in schools;
3. The Directorate for Education Standards (DES) which inspects schools and (at least in theory)
advises on improvements to the quality of the learning experience. Their inspectors are based
national levels to implement this mandate.
4. District education authorities who also conduct school inspections at municipal and district
level through the District Inspectors of Schools (DIS) and the Municipal Inspectors of Schools
(MIS).
UNICEF has focused in practice on the second of these levels – the PTCs with their cohort of CCTs. This
was a logical decision as the over 500 CCTs are a critical human resource, who have the mandate to
work with school managers, governors and staff on improving the school. However, while the
appointment of CCTs in the late 1990s was an innovative attempt to increase school effectiveness,
lack of funding and mobility has greatly reduced their impact.
It was in this context that UNICEF proposed a mentoring and coaching model which would see foreign
educators with a wide range of experience on two year VSO contracts supporting the CCTs. In time
two local mentors were added. The aim was to improve the quality of teaching and learning in selected
and later all primary schools in the catchment areas served by the 12 (and later 15) target CPTCs.
UNICEF’s intervention was rooted in a very specific assumptions which inform the Theory of Change.
These stated that:
CCTs and inspectors occupy a critical lever point in the schooling system being the nearest
level in the system to the school
In-service and pre-service training (including workshops) are costly, time consuming and of
limited or no impact on practice and development of skills particularly if they are delinked
from sustained follow-up support, so an alternative way of training CCTs is required
CCTs (and inspectors) could increase their skills and capacity and realise their potential
through being mentored
Both local and foreign educators could play the role of mentors
CCTs who have been mentored would be able to have more impact on primary schools with
more skills and capacity
10
This in turn will lead to improved quality of primary schools, performance of head teachers
and teachers and their attendance.
The basic hypothesis that the project was built on, that use of skilled mentors and coaches will have
more impact on those mentored than workshops, is well supported in the literature. This literature is
explored in detail in the ‘DES Proposal’ (MoES 2010) and again in the ‘Project Proposal to the Embassy
of the Kingdom of the Netherlands’ dated July 2011 and attributed to Margo O’Sullivan. It is not the
intention of this inception report to repeat the same review of literature.
In theory the mentor was meant to work with all CCTs and all their primary schools over a 2 year
period, as well as working with the school inspectors and liaising with the PTCs and District Education
Officers to provide mentorship and coaching at all levels and develop an intervention model which
would lead to improved primary schooling. At a later stage, when the Kingdom of the Netherlands
came in as a funder the number of focus schools included in the project was limited to 10 per CCT.
This modified the aspirations of the project to impact on all primary schools (MoES 2010).
3.1 Theory of Change The project has a specific Theory of Change underpinning its development and its implementation.
This states that:
1. Pre-service training is not considered a practical alternative to mentoring and coaching being
too long-term and expensive – and only impacting on beginning teachers – but there must be
linkages between pre-service and in-service as they must inform each other.
2. Mentoring can impact on improving CCT quality.
3. Workplace based mentoring and coaching are more cost effective than other methods of
improving teacher performance such as workshops.
4. Workshops without carefully structured follow-up support to teachers in their schools have
proven internationally and in Uganda to be ineffective in changing teacher behaviour, building
capacity and enabling the development of new skills and in enabling the successful
implementation of education reforms.
5. Improvement in CCT quality has a direct impact on teacher and school performance.
6. Working with CCTs, given their greater number, will improve primary school quality more
effectively as a first point of engagement than working with inspectors, who also have more
complex employment and reporting structures.
7. Improved teacher practice and community awareness and empowerment improves learning
outcomes.
8. Focussing on 10 schools per CCT does not have significantly adverse outcomes to non-
selected schools in their areas (and may have positive outcomes).
9. Drafting the Deputy Principals Outreach (DPOs) in each of the PTCs into the project as
counterparts for the mentors would create the basis for sustaining the project.
3.2 Project Problem Statement The UNICEF project was specifically aimed at helping to improve on the low completion rates and
achievement rates in Ugandan primary schools which have been attributed to, amongst others, high
levels of teacher absenteeism and low morale and the poor quality of the education on offer
particularly in rural primary schools. The approach, with its focus on primary schools, also indicates
11
that UNICEF was responding to the increasingly strong view that the low literacy rates and poor
Primary Leaving Exam (PLE) performance relate to poor quality teaching in early grades in primary
schools. The Project Summary documents developed by DES/UNICEF in 2010 and revised by UNICEF
in 2011 indicate that many innovatory projects have identified good and effective practice in the
Uganda context, but there is a persistent problem of replication and even of broad-based
implementation. UNICEF argues that the solutions are known and ‘BRMS standards will enable and
ensure quality education and lead to improving learning achievement particularly in lower primary’
(UNICEF 2011: 6).
The project’s problem statement also states that the standard ways that the system and others
respond to these problems through workshop training has little or no impact on classroom practice.
The second problem that the project was designed to address is low levels of community engagement
with schools and their lack of knowledge required to hold the local school and teachers accountable.
The lack of community engagement with their schools was believed to have a direct impact on the
first problem: it was argued that low community interest in schooling creates the conditions under
which large numbers of primary school learners drop-out or fail.
3.3 Project Outcomes The outcomes expected from the BRMS Mentorship Programme were that:
1. CCTs improve their engagement with schools to provide professional development and
support (rather than acting as materials distributors and inspectors);
2. Improvements in teacher performance and community engagement with the school lead in
the long term to improved completion rates and exam results.
It was reported in the original briefing for the evaluation that no detailed metrics were attached to
these outcomes in the planning process (although there are some mentioned in the original DES
planning document (MoES 2010), in the VSO SLA and three appear in the UNICEF Project Proposal to
the Netherlands Embassy in 2011) and no project-wide baseline was conducted prior to the placement
of the mentors. It is important to differentiate this requirement from the large-scale DES-led BRMS
baseline survey which examined school, teacher and head teacher BRMS capacity, and the needs
assessments which the mentors undertook as part of their induction on arrival in Uganda. These latter
needs assessments were mainly aimed at guiding the mentors’ work and introducing them to the
schools and college rather than providing comprehensive data on the state of primary schools and the
support systems available to the schools.
This report is therefore aimed at providing a formative evaluation of the project, a summative
evaluation of the work of the mentors and a foundation of data for future evaluations.
The intention from UNICEF is that if this project shows that it has impacted positively on the CCTs and
the schools that it will possibly be adapted and a recommendation will be made that it could be
replicated in other countries by UNICEF.
3.4 Project Design and Implementation The project was designed by UNICEF (Uganda) with the MoES. It was deliberately designed as a pilot
for UNICEF, while servicing a need the MoES had for implementation of a country-wide programme in
support of the revised 13 Basic Requirements and Minimum Standards.
The mentors were to be based in 15 Core Primary Teacher Colleges (CPTCs) and impact on the 73
districts in which the BRMS operates across Uganda, with particular focus on teaching and learning
12
practices and school management techniques. They were conscious of the 13 standards and were
allowed to impact on any of them that they wished to target within an agreed framework which was
narrowed early on in the project to focus on four main indicators: teaching and learning; school
management; community involvement; and learner safety and school security.
The mentors determined with the DPO and CCTs on their particular focus and approach after
conducting an environmental scan designed to identify needs. This meant that while all the mentors
worked within a broad school improvement framework, they all implemented different projects.
Each mentor was instructed to work with those CCTs based at the core PTC where she or he was
placed, who wished to engage with the project. However, UNICEF and the mentors reported that all
the CCTs engaged and so the mentors worked with all their CPTC CCTs. The number of CCTs varies
between colleges but no college had under 20 CCTs and some had considerably more. The total
number of CCTs involved in the project was meant to be 352 however, as some of the posts were
empty, the actual number of the mentors worked with had fallen to 313 by December 2013. The
empty coordinating centres (CCs) are serviced by the remaining CCTs or caretakers who are either Pre-
service tutors or retired CCTs. So as to limit the volume of work and the challenge each CCT was to
nominate 10 of their schools for the intervention, making a target for the project of working in 3,520
schools. This meant when the mentor and CCTs were mounting school-focused workshops these
schools would be invited to participate and the CCT was meant to implement the lessons learned from
working with the mentor in these ten schools. However, the CCTs remained responsible for support
to all the other schools in their cluster so would in all likelihood introduce some of the innovations to
these schools as well.
The method used by each mentor to work with their college and CCTs were not stipulated but all the
mentors used a combination of workshops (some for CCTs, some for CCTs and pre-service tutors and
some for CCTs and focus schools) and co-support visits to schools to implement or support the
implementation of the workshop lessons. All the mentors also had to work closely with their DPO,
who was designated as their college counterpart. They also worked with the college principal in
respect of the work plan and the management of the project finances.
In response to the need to improve school accountability to its community the project also aimed to
improve the performance of the School Management Committees (SMCs) and Parent Teachers’
Associations and empower parents and other community members to report on the attendance and
performance of the teachers, as well as becoming more engaged with their schools.
4. Evaluation Methodology and Process This evaluation is designed to both provide a formative evaluation of the Project, a provisional
summative evaluation of the work of the mentors on the project and a basis for setting metrics and
evaluation process for future evaluations (in the absence of a baseline survey when the project
started).
In the process of undertaking this review the evaluator:
Reviewed the Theory of Change and the assumptions that underpin it
Reviewed the programme design and implementation process against the original objectives
Developed some draft metrics for interim outcomes and measured them
Reviewed the programme plan and budget to indicate cost of delivery.
13
The evaluation was conducted in early 2014. The approach included a review and analysis of the
available documentation, observations and interviews in the field, and the collection of data to inform
a set of metrics.
While the evaluator is interested in failure and challenges faced by the mentors and the colleges in
implementing the programme, the main focus was on conditions where the project has succeeded
and achieved real impact. This allows for the description of a proposed model in section 9 of this
report. However, this positive approach does not mean that problems faced in the planning and
implementation of the project have been ignored as this would serve no purpose and potentially lead
to poor value for money in replicating the project elsewhere.
4.1 Review of documents
UNICEF (Uganda) provided the evaluator with a range of documents related to the BRMS Mentorship
Project.
The documents included:
The initial project proposal developed with DES
The final project proposal
BRMS needs assessment and baseline documents
MoES BRMS implementation and quality improvement in primary schools documents
A booklet on good practice coming from the BRMS Project
VSO documents related to the placement of mentors
UNICEF, VSO and Embassy of the Kingdom of the Netherlands (EKN) documents related to the
cost of the project and unit costs of each mentor
Sample of internal communications between UNICEF and the mentors
A 10 year old evaluation of the Primary Education Reform Programme and TDMS
A set of documents generated from two of the mentor placements
Reflections by mentors on their experiences and lessons learned
Presentations made by mentors and DPOs at the project reflection workshop held in
December 2013 in Kampala.
These documents were reviewed using and basic documentary analysis tools. These documents
helped in gaining an understanding of the intentions behind the project and its particular design
features, the model being developed as well as, more prosaically, the planning and implementation
processes.
4.2 Use of documents and interviews to inform a cost of delivery proposition It is not possible within the parameters of this evaluation to undertake a full value for money analysis
of the project and the model it uses. However, it was possible to use data from some of the interviews
and the financial and mentor placement documentation to come to some tentative conclusions on
the costs of the intervention which could be compared to the costs of alternative implementation
approaches to developing teachers. This is a very rough measure, but maybe useful to the MoES and
UNICEF in planning for the outer years of this project.
4.3 Field Visits The researcher undertook field visits to four CPTC sites to review the past and present delivery of the
project and to better understand the impact the project has had to date. The visited colleges are those
at Mityana (Busubizi CPTC), Hoima (Bulera CPTC), Kabale-Buhinda (Kabale CPTC), and Kitgum CPTC.
These CPTCs were selected in collaboration with the UNICEF project manager as they exhibit a range
14
of college and CCT conditions, school conditions, responses by colleges and CCTs to the programme,
and contrasting mentorship approaches. The evaluator spent 2 to 3 days in each CPTC except Busubizi.
The intention behind the first college visit to Busubizi CPTC in Mityana was to field test the data
collection instruments and the design of the evaluation, so only one day was spent in that CPTC. The
field test was also used to check if key informants at CPTC or school level had been overlooked in the
evaluation design and ensure that data is available to inform the proposed project metrics. The latter
was hard to ascertain from the field test process, so the original set of proposed metrics were retained.
The pilot was successful. It became clear that all possible informants had been included in the
evaluation design and the field test indicated that the instruments designed to collect data at both
college and school levels were fit for purpose. As this is the case the findings from this CPTC and set
of CCTs are included in the analysis as they are considered reliable and valid using standard research
criteria.
At each CPTC site the researcher:
lnterviewed the mentor (if still in place);
Interviewed between 4 and 5 CCTs who worked with the mentor. These were selected in
advance purposively by the mentor (where still available) and/or the DPO based on those that
the mentor/DPO believe have adopted the methods and approaches being advocated in their
work most effectively, and from those who were in the college on the day;
Surveyed a further 3 or 4 CCTs in each college from the same purposive sample as above. The
group of 8 or 9 CCTs was split in two on a simple randomised basis;
Interviewed both the CPTC Principal and the DPO;
Interviewed the head teacher and a focus group of teaching staff in a sample of 3 – 5 schools
per PTC. A purposive selection of schools was taken from those which have been impacted by
the mentor and CCTs and a smaller number which were outside the group of focus schools to
assess whether the methods being advocated in the core schools are filtering out to other
schools in the vicinity. The schools included both rural and urban sited primary schools. The
14 schools were selected by the mentor (where available), the DPO or a CCT based on their
accessibility and involvement in the project.
In total data was collected from 37 CCTs (20 through focus group interviews and 17 using a survey
instrument) and from 14 schools. In addition all four of the DPOs and PTC principals were interviewed.
The researcher also spent a day travelling with two of the mentors who are still on site to observe how
the mentors spend their day, how they relate to the CCTs and to schools they work in, and how they
engage with the Principal, DPO and other members of the college management team. The purpose of
this exercise was to develop a picture of how the mentors are implementing their mentorship role in
practice.
This data was collected as planned between February and early April 2014.
4.4 Interviewing and Surveying Mentors The researcher conducted a focus group interview with five of the departing international teacher
mentors in December 2013. This followed attendance at a reflection workshop with MoES, UNICEF,
mentor, CCT, DPO and partner involvement at which a number of presentations were made. During
the workshop the researcher held a number of informal discussions with mentors and partners in
order to gain background information and deeper understanding of the basics of the project.
15
During the field visits the two mentors who were still in place were interviewed as well as two of the
mentors who are now working back in East Africa, and who were known to have strong opinions on
the project.
These data were enriched with data collected from mentors through a questionnaire which was
administered to all the former and present mentors. Basic statistical analysis of their responses has
helped inform this report. Eleven responded out of a possible 18 former and present mentors. This
data was collected in December 2013, and March 2014.
4.5 Stakeholder Interviews Many of the key players and stakeholders who have been involved in planning and implementing the
project are based in Kampala.
The researcher interviewed the following:
A senior manager in the MoES who has been directly involved in the planning and delivery of
the BRMS mentorship project. This official was identified by UNICEF based on knowledge of
the project.
The BRMS project staff and management in UNICEF. The researcher interviewed those who
have been directly involved.
The project liaison who is the Financial Controller in the Embassy of the Kingdom of the
Netherlands.
The VSO director and the relevant project manager who were directly involved in recruiting
and inducting the overseas mentors and have played a support role throughout the period of
the mentors’ being on the project.
These six interviews provided information on the thinking behind the establishment of the project,
the challenges faced during its implementation, the extent to which these key stakeholders believe it
has achieved its intended outcomes and their thinking on how the project can be further
mainstreamed, sustained and replicated. This data was collected in early 2014.
4.6 Project Metrics The project documentation indicates that this UNICEF and MoES intervention was meant to achieve
three objectives which can generate metrics that can be measured as part of the evaluation. These
are:
Head teacher performance
Teacher attendance
Quality of education and performance of schools
In addition, it is important to relate some metrics to the performance of the CCTs and the mentors.
This is because the relationship between the CCTs and the mentors and the relationship between the
CCTs and schools are key to the model. As UNICEF wants to understand how and why the model
worked and what the variables and conditions are for it to work, metrics related to the CCTs’
relationship with their mentors and the schools are going to be critical.
Data can be accessed for most of these metrics, however the report provides discussion on their utility.
This report also attempts to identify other metrics for which data could be collected should this
approach be replicated elsewhere (see Tables 1 and 2 below). These are likely to particularly relate to
the impact of the intervention on the CCTs and the colleges. The lack of a project specific baseline is
particularly limiting in respect of these aspects of the project.
16
The first of the two tables which follow relates the indicators or metrics to be used in the evaluation
to the theory of change and the assumptions made in the relating of the ToC to the indicators. The
second expands on the metrics and looks at how they will be measured. The two tables should be read
together as they complement each other.
17
Table 1: Theory of Change and Metrics Framework for BRMS Mentorship Programme
Theory of Change
Training Assumptions Indicators Mentoring Assumptions Indicators
Needs Most training of educators and CCTs is of low quality and dependent on trying to drive improvement in schools and performance through workshops
That workshops will bring about change in the classroom and in the work of CCTs
Improved test scores. Reduced learner drop-out rates in schools. Improved job performance by CCTs
Performance by CCTs, inspectors and District Education Officers (DEOs) is inadequate to transform schools into high performing centres of learning.
Mentoring CCTs, inspectors and DEOs will improve the performance of educators in schools including head teachers.
Improved test scores using PLE results. Reduced learner drop-out rates calculated for sample of schools from school data. Improved job performance by CCTs, inspectors and DEO and improved head teacher performance as measured by a satisfaction survey.
Inputs Model workshops for CCTs to model effective workshop practice
CCTs can learn from observing and being part of effective workshops
No. of workshops held. Workshops are evaluated as more effective than traditional workshops by CCTs and teachers
Mentors based in PTCs and working with CCTs
Access to mentoring helps CCTs and other PTC staff to do their jobs better. Attaching one mentor to each PTC is adequate to provide support to sufficient numbers of CCTs to impact large numbers of schools.
No. of mentors in place based on UNICEF documents. No. of CCTs linked to each mentor, based on mentor reports and UNICEF/VSO project documents. Amount of time each CCT spends with each mentor based on self-reporting questionnaire for CCTs and head teachers.
Outputs Model workshops. Attendance of CCTs at workshops. CCTs improve school support model
Workshops are held and are better than in past. CCTs attend and learn from them. CCTs can transfer what they learn in workshops to their school support
Attendance records. Workshop agenda. Post-test of CCTs with improvement over pre-test scores. CCT improvement in practice and regularity of schools visits as measured by head teachers.
Mentor mentoring sessions with CCTs and other PTC staff. CCTs provide guidance to teachers and head teachers and support schools.
Basing mentors (foreign and local) at PTCs and mandating them to mentor CCTs will improve CCT support of schools. CCTs will learn from mentors and apply the knowledge to schools.
No. of CCT visits to schools based on CCT reports, mentor reports. Nature and quality of the support provided by CCTs to schools as assessed by mentors and CCTs in interviews and questionnaires and head teachers based on interviews and questionnaires.
18
Table 1: Theory of Change and Metrics Framework for BRMS Mentorship Programme
Theory of Change
Training Assumptions Indicators Mentoring Assumptions Indicators
Intermediate outcomes
CCTs visit schools more regularly. Teaching and school management improve
CCTs can learn improved school support approaches from training. CCTs provide more school support. More school support by CCTs impacts positively on school performance
Records of CCT visits to schools. Reports from school personnel on quality of CCT support.
CCTs, inspectors and DEOs do their jobs better. Improved head teacher performance. Teaching improves in schools. Pockets of effective CCTs able to support schools effectively. Improve teacher and learner enjoyment and commitment.
CCTs and inspectors with support of DEOs and PTC management able to provide support and guidance after mentors have completed their placement. Intensity of mentorship adequate to lead to sustained change. Improved enjoyment and support of teachers translates into improved attendance by learners and teachers.
No. of CCTs and inspectors mentored, using mentor reports and PTC/DPO records. Intensity of access to mentor for each CCT and inspector, using mentor and CCT reports. Assessment of the quality of mentorship provided and CCT/ inspector response to mentors based on site observations by evaluator, mentor and CCT reports. Improved performance of head teachers measured through satisfaction survey and improved attendance of teachers and decreased absentee rates for teachers.
Long-term outcomes
Learning improves. Test schools improve. Student drop out declines.
Better support by CCTs has sustained positive impact on teaching and school management. Better teaching and school management leads to improved learning and a more enjoyable learning experience.
Improved test scores. Lower learner drop-out rate.
Learning improves, with spread of child-centred learning. Reduced student drop-out rates. Cadre of skilled CCTs able to support schools effectively.
Critical number of mentored CCTs and inspectors able to influence behaviour and work of other CCTs leading to cadre of effective CCTs and inspectors.
Test scores using PLE school results. Reduced learner drop-out rates calculated for sample of schools from school data. Behaviour and professionalism of CCTs and inspectors as assessed by evaluators during field visits and satisfaction survey from head teachers.
19
Table 2: Proposed Interim Metrics for Measuring the BRMS Project Impact
Focus area and assumptions
Question to be answered by metrics
Metric to be measured Method to collect data for metric
Head teacher performance
How do we know if the head teacher performance improved?
CCTs and teachers report that the head teachers have improved management skills over the last 2 years/as a result of the project.
Satisfaction questions for teachers in schools and for CCTs on the performance of head teachers
Teacher Attendance
How do we know if teacher attendance has improved?
Select a week in 2011, 2012 and 2013 and compare attendance of teachers (no of absentee days as a percentage of all days) and number of teachers absent for more than 2 days in any of these weeks.
Teacher attendance registers of target schools.
Reduce teacher absenteeism in the target 77 target districts by 5 percentage points.
Use BRMS sample baseline study figures as the basis for comparison and conduct similar review of absenteeism in same or similar schools using the same methodology.
Improved retention of learners in focus schools
How do we know if the project has impacted on learner retention?
Increase in primary school learner completion rates over the period of the project by 2 percentage points in districts where the project is operational
Analysis of EMIS data for the 73 districts impacted by the project
Performance of the school as a useful proxy for quality of the school.
How do we know if the performance of the education received in the schools has improved?
3 percentage point increase in Division 1 and 2 passes and 3 percentage point decrease in the number of fails in the Primary Leaving Exam (PLE) in sample schools between 2011 and 2013.
As performance is best measured by exam results in a random sample of 50 schools from among the schools where the target CCTs have been working will be selected from the national EMIS data base and their exam results for 2011, 2012 and if available 2013 evaluated and trends identified. Another 20 schools will be selected from other neighbouring districts where there was no impact from the mentored CCTs.
Improved P3 and P6 NAPE national literacy and numeracy test results by 3% points.
Use 2010 NAPE results for the 73 districts as a baseline and compare with 2013 results for same districts. (UNICEF has already collected, tabulated and analysed this data).
CCT performance
How have the professional habits of CCTs
(i)The number of days 20 randomly selected mentored CCTs have spent in schools and the number of schools they visited compared to the
Select 20 CCTs at random (10 who no longer have a mentor and 10 from among those who still have a mentor) and get them to record the number of days spent in schools and the number
20
Focus area and assumptions
Question to be answered by metrics
Metric to be measured Method to collect data for metric
changed as a result of the work of the mentors?
numbers for a randomly selected group of 10 non-mentored CCTs.
of schools visited and what they did in each school over a particular week. Do the same exercise with 10 non-project CCTs in neighbouring districts.
(ii)The number and type of activities these 20 mentored CCTs did in these schools compared to the number and type recorded by the non-mentored CCTs
Select 20 CCTs at random (10 who no longer have a mentor and 10 from among those who still have a mentor) and get them to record the number of days spent in schools and the number of schools visited and what they did in each school over a particular week. Do the same exercise with 10 non-project CCTs in neighbouring districts.
4.7 Analysis of the Data Both qualitative and quantitative data were collected although the majority of the data is qualitative.
The quantitative data was analysed using basic Excel tools. This was made possible by the relatively
small amount of quantitative data that was collected.
The qualitative data was managed in a range of ways in order to triangulate the data. The project
planning and delivery related documents were examined using various standard documentary analysis
tools, including use and repetition of key words and phrases. There was a particular focus on the way
that the concept of mentorship is presented and how that has informed the model of mentorship that
has been implemented. The words and writing of the mentors were also examined for interpretation
of their role and their reflection on how that role has been fulfilled.
The interview transcripts were either entered directly onto the computer by the evaluator or written
out long-hand at the time of the interview as experience shows that respondents in Uganda are often
less confident speaking into a recorder and prefer to be able to see their words being written down
or typed in front of them. This also saved time in transcribing recorded interviews. The printed out or
copied pages with responses to the interviews were annotated using different colour markers to
identify key categories of responses (Vulliamy et al, 1990). These categories were influenced by a
review of the questions asked, the responses and by the evaluation questions (Carspecken, 1995).
Similarly coloured responses were be grouped together. Coding was not be used as it tends to reduce
the responses to a limited number of codes, with the consequent dilution of their depth and richness.
Similar responses were counted and noted to inform very basic quantitative analysis. The narrative
was then drawn out from the categories and the information contained in the statements attached to
each category.
The approach also involved a level of ‘backward mapping’ (Elmore 1980) as I have started the data
collection and analysis from the mentors up through the system. Backward mapping allows for the
participant’s experience of the innovation to be foregrounded in a ‘bottom up’ approach. It assumes
that the success of an innovation is best assessed through the involvement and understanding of the
change held by those directly involved. This data will be triangulated with the data from the college
21
based staff, particularly the CCTs, as it is acknowledged that many of the mentors may have a
particular cultural lens through which they may see and judge the project. The testament of the CCTs
and DPOs in particular helped endorse and in some cases balance that of the mentors. Observations
during field visits, as well as interviews with UNICEF, VSO and the funder also provided alternative
perspectives which assisted in developing as valid and reliable analysis of the implementation and
impact of the model as possible within the circumstances.
Cost of delivery was assessed simply by looking at the unit cost per college, school and CCT of the
mentorship model and looking at those figures in the context of the impact that the programme had
on the colleges and schools.
5. Limitations on the Evaluation The main limitation in the evaluation is that there was no project specific baseline conducted at the
start of the intervention. Drawing a direct causal relationship between an intervention of this nature
– particularly one which is one stage removed from school sites – and improvements in the complex
environment of a school where there are so many variables is highly problematic. This is made
infinitely more problematic when no baseline survey was conducted in the schools. The environmental
scans that the mentors conducted as part of their induction are inconsistent and so cannot serve in
the place of a coherent rigorous baseline survey. While one could construct a baseline through
examining similar schools where the mentor is not impacting the fact that the mentors were working
with all CCTs in the PTC and that the CCTs were mandated to spread the lessons they learned in the
ten target schools to their other schools means that there are no comparable schools to draw any
baseline conclusions from. This means that the measurement of the intervention’s impact on schools
is highly impressionistic and speculative.
Other limitations relate to the lack of a complete record of the project through the available
documentation generated by UNICEF as well as wide variation in the quality and depth of information
generated by the mentors while in their posts.
A further limitation is the fact that most of the mentors had already left their post before the start of
the evaluation. This meant that some mentors could only be reached by electronic means for interview
and there were limitations on the selection of colleges available for observation of the project in
operation.
6. Findings: Operational
6.1 Relationship between Mentors and CCTs In all four CPTCs where the research was undertaken the 37 CCT respondents were positive about
their relationship with their mentor and they all – without exception – believed that their ability to do
their core work of supporting schools had been improved as a result of working with the mentor. The
mentors in their responses to a survey (see Annex 3) agreed that the relations with the CCTs were the
most productive and satisfying part of the work. However, they made clear that the CCTs were not
properly prepared for the project, which meant that at the start of the project there was some
misunderstanding and confusion around their role. One CCT confirmed this when he said that they
thought the mentor had come not to ‘mentor’, but to ‘monitor’ them. The data collection process to
inform the needs assessment which each mentor conducted with the CCTs and DPO immediately on
22
arriving at the college helped allay these fears. The mentors reported that while most CCTs gradually
warmed to the project and became enthusiastic participants this was not the case with all CCTs,
particularly male CCTs nearing retirement. In at least two colleges the resistance of the management
to the project and the mentor made it difficult for the CCTs to work with the mentor.
It was anticipated that the surveyed group of CCTs would indicate an increased intensity of visits to
schools as a result of the project. However, most of the CCTs reported that this had not changed much
(see Figure 1), because it is a fairly inelastic relationship influenced more by workload, distance, access
and funds provided for transport, than by CCT commitment. Where CCTs believed the intensity of
visits had increased they related the change to the presence of the motorbikes which UNICEF had
supplied them with, which allowed for more schools to be visited each day, as the alternative is
travelling by taxi or boda boda. However, as one CCT stated baldly “fuel remains a challenge” as the
CCTs are only allocated Ugx 80,000 (US$32) per month for fuel and maintenance. One group of CCTs
also pointed out that the motorbikes are aging and maintenance costs are growing.
In fact the intensity of visits by some CCTs may have even reduced as one impact of the mentor in
some of the colleges was to link the work of the CCTs and the pre-service training of teachers more
closely. In two colleges the CCTs indicated that this means that they are lecturing trainee teachers
more than in the past using their experience in the field to enrich the training. While this is a positive
development it does mean that the time the CCTs have available for school visits is reduced.
Figure 1: Regularity of CCT Visits to Schools (n=17)
When the CCTs, who were surveyed, were asked to list the main activities they engaged in with schools
before the intervention and the main ones they are conducting this year there was found to be some
change. Although nearly half the respondents (47%) put the same activity at the top of both lists, and
for 41% (n=7) of the respondents this was classroom observation or support supervision, the
remaining 53% (n=9) indicated that their main activity in schools is related to the work they did with
the mentor. However, most of the rest of the activity list for all CCTs was similar pre and post
intervention. This was confirmed in the focus group interviews with CCTs. The reasons for this relative
stability in activities conducted by the CCTs is probably due to most of the mentors choosing areas of
focus which complemented what CCTs were already doing, and all involved some form of support
0
1
2
3
4
5
6
7
8
9
Weekly Monthly Not regularly Occasionally Rarely
23
supervision, and the relatively low intensity of the time the mentor was able to spend with each CCT
(see figures 2 and 3). Most of the surveyed CCTs worked with the mentor at best monthly and most
of these interactions saw the mentor working with CCTs in a pair or group.
Figure 2: Regularity of work with Mentor (n=17)
Figure 3: Mode of CCT Engagement with Mentor (n=17)
Interestingly the CCTs in all four colleges visited held that the level of engagement they had with the
mentor was adequate. This was a key concern of the evaluation as mentorship and coaching models
normally involve much greater intensity of engagement between the mentor or coach and the mentee
than the UNICEF model included. That the CCTs felt satisfied with the intensity is probably based on
the following factors:
0
2
4
6
8
10
12
Weekly Fortnightly Monthly Termly Less than termly
0
2
4
6
8
10
12
14
Alone withmentor
With one otherCCT
With group ofCCTs
Sometimesalone/sometimes
other CCTs
24
(i) Most CCTs are in the latter part of their careers and so are relatively well-skilled and
knowledgeable such that they claimed little that the mentors introduced to them was
totally new, but supported what they already knew;
(ii) Most of the CCTs are in their 50s and so possibly their capacity for absorbing many new
ideas is limited;
(iii) Each CCT has a heavy workload already which the work of the mentor might support but
did not replace so greater intensity would reduce their ability to service their schools and
attend numerous workshops being held regularly by the MoES and NGOs for which CCTs
are paid ‘facilitation’.
However 100% of the interviewed CCTs argued for the period of time that the mentor was based in
their college to be extended, with between one and three extra years mentioned. The reasons cited
for this relate to the belief that the mentor was just beginning to make a real impact at the point
where their contract ended and that they felt as CCTs that they wanted to spend more time with the
mentor. This may appear to contradict their view that the intensity of engagement was adequate, but
in fact endorses those reasons. CCTs could not give more of their time during the two years of the
placement but could go on engaging with the same level of intensity for the following years.
The level of engagement and the ability of the mentors to impact on the CCTs was further limited by
the sheer number of schools for which some CCTs are responsible. This number has been rising in
recent years as the original cohort of CCTs, who were employed as CCTs in the late 1990s in most cases
from tutor posts, reach retirement or die. Only in one college visited were the vacant posts filled
promptly with younger CCTs. In the other three colleges the cohort of CCTs was of a fairly uniform age
and were covering for the fact that some of the CCT posts are vacant. This leaves most of the CCTs
responsible for over 50 schools and some for over 300 schools. When the distances between schools
in the rural areas and the limited access the CCTs have to fuel are factored in it is surprising that some
schools are visited at all. However, in two ways this overloading of the CCTs worked to the advantage
of the mentors. Firstly, CCTs were receptive to new ideas and particularly ones which would make
their workload seem more manageable; and secondly they seem to have welcomed increasing focus
to their work, which the mentors brought. This was most clearly illustrated in targeted lesson
observations, or ‘support supervision’ as it is called in all the colleges. The CCTs reported that they
used to observe lessons without any real idea of what they were looking for, but after working with
their mentor, they now limit their supervision to particular aspects of the lesson, such as use of
instructional materials (IMs) in the lesson. After the lesson they advise the teacher on this specific
aspect of the lesson. The CCTs also said they now ask learners what they have learned in the lesson
and feel more capacitated to debrief the teacher at the end of an observed lesson than before in a
way which “appreciates the teachers’ efforts rather than blaming them” as one CCT put it. They also
indicated that after working with the mentor they are more competent in advising teachers in how to
improve their teaching practice, particularly in relation to teaching literacy (see table 3). This relates
to the increasing confidence with which CCTs, who engaged with the project, approached their work.
Their raised confidence levels was commented on by the mentors, their managers, VSO and the CCTs
themselves.
Table 3: Responses to question asking ‘Can you list what (if anything) you do differently in schools
as a result of the mentor working with you?’
Activity mentioned by CCTs Number of CCTs citing this activity
Advise teachers on improved ways of teaching/improved teaching of literacy/reading
13
25
Activity mentioned by CCTs Number of CCTs citing this activity
Improved lesson observation/support supervision and/or feedback to teachers
12
Mobilising and involving community/stakeholders in planning and improving schools
6
Joint planning with school staff/improve assessment of school and learning needs/generation of data
6
Creating condition under which learners can talk more / express selves
4
Assist teachers breakdown and interpret curriculum 4
Conduct continuous professional development including school management training
4
Production of IMs 2
Keeping of records of school visits and activities 2
Advising teachers on positive discipline 2
Advising teachers on the organisation of classrooms 2
Visit schools more often 1
The weakest element in the planned intervention appears to have been on involving the community
more effectively in their schools. Most foreign mentors made little impact on this objective, but the
local mentors made substantial progress in this focus area. This may represent the relatively complex
nature of the relationship between a community and its school and the difficulty of an outsider making
real improvements in that relationship. However, by making sure that the focus schools are managed
better, that teachers are teaching more and better, that corporal punishment is used less, that CCTs
visit the schools more often and that the classrooms look more attractive the children are likely to be
keener to attend. This is likely to impact in time on the relationship between the school and its
community, particularly if the PLE results rise. There are signs that this is happening, if inconsistently.
Given the number of CCTs at 313 and the relatively low intensity of engagement between individual
CCTs and the mentor across all four colleges it is impressive that the mentors impacted on the work
and thinking of the CCTs. The next section looks at this aspect of the work of the mentors.
6.2 Interventions Driven by the Mentors The mentors, as their main modus operandi, combined workshops held in the college and in
coordinating schools with school support visits. The workshops were mainly focused on introducing
the CCTs and often the pre-service tutors to new or refined ideas or for collaborative development of
such documents as support supervision protocols, lesson plans and schemes of work, breakdown of
the themes in the curriculum to inform teaching plans, and instructional materials (IMs). In a couple
of the visited CPTCs the power of these workshops led to the mentor being asked to play a role in
informing pre-service training in the college. This was an important value add of the project as the
linking of pre-service training of teachers to school reality is weak in many education systems. In three
of the four PTCs visited the mentors assisted the pre-service tutors with teaching practice assessment,
development and use of appropriate low cost IMs using local materials and, in one college, the
deconstructing of the thematic curriculum into teachable chunks of knowledge and skills. The
principals and DPOs in the PTCs where the mentor and the CCTs had worked with pre-service students
and their tutors indicated that the benefits had been noted. One principal related the work of the
mentor and the CCTs with student teachers to a rapid and pronounced improvement in their end of
course results.
26
The level of impact that the mentors had on CCTs and schools came out in the interviews. Each of the
four mentors had a somewhat different focus. These were:
Early grade reading, general literacy and management of the library and the library period.
Deconstructing the thematic curriculum subject by subject into digestible schemes of work.
Assessment and improving teaching and learning by use of VARK (Visual, Aural, Read/Write,
Kinesthetic) analysis to assist in lesson preparation, with a focus on P2 and P4 English.
Improved classroom and school planning practice with a focus on early grade reading.
There was a high level of overlap between the focus areas of these four mentors. The decision to focus
on particular professional areas of practice was informed by the initial needs assessment undertaken
by each of the mentors with the CCTs and DPO in their college at the start of their placement. The final
decision on what to focus on though was also informed by three other considerations: the competence
and knowledge levels of the mentor; the centrality of BRMS and particularly the importance afforded
four of the BRMS pillars by UNICEF in initial briefing sessions with the mentors; and the interaction
between mentors. While the areas of focus in three of these four colleges were similar the mentors
introduced a range of activities which included learning games, development and use of IMs using
local resources, deconstruction of the thematic curriculum, improved methods of conducting and
assessing lesson observations. As one of the former mentors explained the recruitment process by
VSO was based on the submission of CVs and not detailed interviews, so in the words of this mentor,
“Those [mentors] recruited came from a wide variety of backgrounds in the education sector – this
made the programme complicated. So UNICEF was forced to manage with what they got. So they had
to leave [mentors] to do the best they can. So this led to individual needs assessment. They naturally
got a wide range of projects as people find what they want to find and the questions they ask are
driven by background and interests. So the project never had national coherence”.
While this is a personal view – and not one shared by other former mentors who were interviewed –
it does highlight the problem of recruitment for such a complex role, and it also may indicate the
problem of having an intervention which lacks a unifying theme. However, it could be argued that
while the interventions were specific to particular sites and environments there was an overall
unifying strategy and that a specific model of CCT support and school improvement emerged. This is
explored later.
Those interviewed were asked about whether it was significant that their mentor was either local or
foreign. The general response was that it was not important as long as the mentor was able to do the
assigned work effectively, had innovative ideas and could work with the college. The principal of the
college where the local mentor was placed indicated that they had expected a foreign mentor but
once they realised that they had a Ugandan educator who had been a PTC college principal coming as
mentor they saw the advantages: no environmental adjustment or language issues; thorough
knowledge of the system and its weaknesses; and immediate credibility with staff and schools.
Generally it was acknowledged in the other colleges that the foreign mentors took time to settle in,
get used to the Ugandan education system and work culture, and in one college the mentor struggled
to be understood at first. These colleges indicated that they were happy having foreign educators as
mentors as they brought new and innovative ideas particularly in solving old problems in the
education system and in schools. This was not lost on the principal of the PTC which received the local
mentor who said “We would have benefited by a foreign mentor as he or she would have dealt with
problems like corporal punishment [in schools] in a different way”. At the same time he indicated
pleasure at the way this mentor had been able to assist him with management issues at the college
and particularly helped resolve a tough staff issue while also making sure that the CCTs were doing
27
their work and agreed, “a foreigner couldn’t have done this”. On the other hand one of the CPTCs with
a foreign mentor indicated that it would have helped to have a mentor who understood the Ugandan
situation better and had been more flexible in the interpretation of her role and the management of
the vehicle and the UNICEF project funding.
The project managers as well as the college management and the mentors mentioned a period of
some months when project funds dried up. This was due to late transfer of funds from the main funder
to UNICEF, as well as a new financial system in UNICEF which apparently lead to the temporary delay
of the funds transfer. This impacted delivery to some extent in each college. In all the four CPTCs the
college stepped in and made sure that college funds were available to keep at least a bare minimum
of programmes running. The mentors and the college principal acknowledged in two of the colleges
that this created frustrations for the mentor and delayed development of the rollout of the project. In
the other two PTC where the relationship between the mentor and the college management was
particularly strong the impact was minimal as ways were found to make sure that the disruption was
kept to a minimum, although field delivery was scaled down. This showed the importance of the
relationship between the mentor and the college principal.
A number of the principals and mentors pointed out that the relationship between principal and
mentor could have been strengthened from the start of the project had UNICEF ensured greater role
clarification and information sharing through more workshops before and as the project started and
more specific documentation defining the relationships. They pointed out that more information
would have prevented tension over who had the right to use the project vehicle and for what
purposes, management of the project funds and the amount of time they would need to commit to
the project. UNICEF argues, with proof, that the use of the vehicle after initial problems was made
very clear to all the parties, and that this was conveyed in documentary and verbal form in a specially
convened meeting to discuss the vehicles in mid-2011. The MoES reiterated the rules on use of the
vehicles to the colleges. As with the management of the funds more detailed documentation and
signed commitments by the CPTC managers and the mentors to observe the rules might have helped.
It should be noted that although the generous project funding and the project vehicles caused most
of the management problems that affected the project, without these funds and vehicles this project
could not have delivered and been successful.
The project plan built in a role for the college DPOs as counterparts for the mentors. This had the
advantage of providing the mentor with immediate access to the college management and providing
the project with institutional status. However, the DPOs and mentors admitted that the DPOs rarely
collaborated with the mentors as much as envisaged. This was explicitly based on the other
responsibilities that DPOs have, but many respondents in two of the colleges implied that the DPO
may have found it somewhat demeaning to be working in schools and being ‘mentored’. However,
one CCT commented that, “Our DPO used not to monitor [what we do in schools], but by the mentor
making him move, he started also to work”. The wisdom of DPOs playing this role was questioned at
the start of the project by UNICEF and the main funder, but the MoES was apparently adamant that
as the DPOs are the line manager for the CCTs they alone could legitimately play the line management
and counterpart role.
Every respondent indicated that they believe that at least some of what the mentor was doing with
the colleges and schools would be sustained. The reservations expressed by the PTC principals
concerned whether when they retire (and most PTC college principals are approaching retirement
age) their successor will have the same dedication to the innovations. However, most of the CCTs and
PTC managers saw the greatest strength of the work that the mentors did being that it (i) supported
the work that the CCTs are meant to be doing in schools; (ii) almost universally supported government
28
policy and involved improved implementation of policy; and (iii) often helped CCTs conceptualise their
work better and improve their impact on schools and the appreciation of their work at school level.
This is true of the bolstered support supervision with improved documentation, the development and
use of low cost IMs, improved school management transparency leading to improved head-teacher
and staff relations, improved use of the library period, collection of school level data to inform school
planning, conducting school-based CPD, helping teachers manage large classes and discipline without
using corporal punishment, helping teachers to implement school health and social related
government initiatives, and improving early grade literacy and numeracy. It is striking how little
innovation the mentors introduced. This was a strength of their approach as it meant that everything
they did do was accessible to the CCTs and schools.
Where the mentors added real value was in the approach they took to schools and teachers and in
the rigour and effectiveness of the changes they introduced to practices and processes. So, the
adapted support supervision documents and processes were more teacher-friendly and easier to use
than the instruments being used and they were made accessible to head-teachers who were given the
modelling and support to implement lesson observations in their own schools effectively. Similarly,
teachers saw effective use of group work and positive discipline and realised that they could do it
themselves. Where innovation was introduced it was in such areas as the collection of relevant data
at school level to inform better interventions, initiating a process of discussing with learners what they
have learned in the lesson, use of group-based reading to enhance reading skills in early grade classes,
breaking down the topics in the syllabus to assist teachers in their planning, varying and improving
learner assessment methods, and innovative ways of using the environment to enhance learning. All
of these activities added real value to existing processes in schools and helped CCTs gain more
confidence and enjoyment in their work. At the same time schools saw the CCTs adding more value
through their work, particularly as their focus shifted from pseudo-inspection to mentoring and
coaching. Overall the impact of the more successful mentors was to increase the collegial nature of
the relationship between the colleges and their CCTs and the schools.
6.3 Impact on Colleges The Core Primary Teacher Colleges where the mentors were based were meant to be the primary
beneficiaries of the project, along with the district and municipal inspectors. The inspectors in practice
were rarely included in the project, except where the mentor sought their involvement, as was the
case in two of the sample of four sites visited.
To ascertain the impact of the intervention on these CPTCs the college principals, DPOs, CCTs and
mentors were interviewed. The over-riding response was positive. The respondents were in no doubt
that the placement of the mentor in the colleges for two years had a positive impact on the college
and its programmes. This was expressed in terms of improvements in the way that the CCTs go about
their work, the energy and new ideas and approaches that had been brought to the college by the
mentor and the impact in some of the colleges on pre-service training. On the last point one principal
explained,
“Only one first year student failed [the first year of the teacher training course] last year while we used
to get 15% failing every year, with 270 students that is a big change, from about 40 failing to 1! We
have no doubt it is partly due to the work we were doing with the mentor”.
The principals recognised that the need to plan with the mentor meant overall they were planning
college activities better than before. In all of the colleges the improved ability to access schools using
the UNICEF double-cab pick-up truck provided to each mentor and the motorbikes provided to 127
CCTs was seen as a major benefit.
29
It is significant that most of the college managers stressed that the mentors allowed the college to
fulfil its existing roles more effectively. As has already been observed there was little sense that the
mentors had brought new concepts with them. The principals and DPOs pointed out that support
supervision, teaching of literacy and numeracy in schools, the library and the library period, concern
over corporal punishment, use of group-work and development of IMs were all in place prior to the
arrival of the mentor. This may be significant for sustainability. If the managers feel that the mentors
were simply adding value to existing processes then they were more likely to own these changes and
sustain them. Even the more novel work that mentors did, such as introducing innovative ways of
teaching reading in the early grades, were easily absorbed into existing practices. However, the
argument that the project was bringing nothing new was seen by at least one mentor as a way of
avoiding changing,
“I was treated with a great deal of respect but the motivation to change things was limited … My DPO
was always promising to visit schools and workshops with me but never did”.
Perhaps at a management level the three main areas of impact were seen in the backwash impact of
the changes made in CCT work to the pre-service training of teachers, the assistance two of the
mentors gave to managers in running the college and in the same two colleges the focus on
networking key stakeholders into the work of the CCTs. These were done most easily and naturally by
the local mentor, He had been a principal of a PTC and found that the lack of management
responsibility as a mentor allowed him time for reflection. This in turn led to his assisting the
management team of his college refine their management approaches and particularly assisted in
solving a particularly complex staff issue. The principal made it clear that this peer support was hugely
significant and valuable. The foreign mentors tended to upset the management when they tried to
improve management of finances and administration. Only one of the three surveyed foreign mentors
managed to build such a strong relationship with the college management that her management
advice was accepted. One of the others irritated the management with her suggestions about how
the project fund and the project vehicle should be managed.
The same two mentors – one local and the other foreign – who were able to work with the
management successfully on improving management systems also impacted even more significantly
on repositioning their CPTC and the CCTs in relation to other education stakeholders. The relationship
between the DEO, municipal education officers, and municipal and district inspectors and CCTs and
their CPTCs is often remote. This creates problems for schools as these various officials tend to visit
and assess the schools using different tools and processes. Both of these mentors identified this as a
key barrier to providing schools with sustained and seamless support. In resolving it they got all the
stakeholders planning school visits and reviewing them together, using the same classroom
observation instruments and often visiting schools as a single team. In addition the officials were
invited to the workshops that the CCTs and mentor ran so that they could understand what the CCTs
were working on improving in schools. One of the CPTC principals stressed that these officers, such as
the inspectors, would still use their own instruments and undertake their statutory processes with
schools alongside using the shared instruments, but now these visits were planned with the other
officers and the CCTs so that they no longer appeared as stand-alone activities.
As one focus group of senior district based officials with responsibility for education explained,
“[The mentor] developed a stakeholder guide. He has conducted coordination meetings between all
stakeholders - we have had regular meetings. Now we share challenges and achievements and draw
up work-plans together for improvement across the district. We were not coordinating before, as
facilitation for this was not there. Also, previously CCTs would not share reports with us so we now
30
know more and it improves our performance and now we all mentor each other on how to deal with
challenges and this makes an impact on the ground. We even make joint monitoring visits. [The
mentor] frequently visits our offices for follow up. [The mentor] checks the CCTs’ work and their reports
come to us”.
This approach was acknowledged and clearly appreciated by these officers during interviews as they
had formerly felt isolated and often lacked transport to get to schools. Travelling together meant
access to schools for all the officials and CCTs was improved. Innovations such as this are likely to
survive as it has improved the working lives of senior officials without creating much extra work. As
one of the focus group of district education officials said,
“We are really happy as it has improved our working relations and our work. Training to teachers has
greatly improved their performance and on our own performance as we learn from one another. We
will go on meeting and working closely together”.
6.4 Impact on Schools While the mentor was primarily responsible for impacting on the CCTs, the project planning
documents, the theory of change and the objectives all assume that there will be a direct causal
relationship between the work that the mentors will undertake with their CCTs and the improved
performance of teachers and schools in the focus schools, greater retention and achievement of
learners as well as more engagement of communities with their schools.
Each CCT that a mentor worked with was meant to replicate the innovations in 10 of their schools.
However, in at least one PTC visited, the mentor reduced the focus to just 2 target schools per CCT for
the first year. She argued that this allowed for the work that she was doing with the CCTs greater
chance of impacting positively and substantially at school level. This mentor believed that 10 schools
would reduce the impact of the CCTs and so her ability to show them what an improved school looks
like. Once the two schools were functioning well the focus was expanded to the other eight schools.
As the 15 mentors, based in 15 CPTCs, worked with a total of 3522 CCTs then the impact of their work
was potentially felt in 3520 primary schools in 73 districts scattered across the country. However, as
we have seen, the number of schools each of the CCTs worked with alongside their mentor varied
from the 1:10 norm. The mentors who were surveyed through the questionnaire indicating that they
worked with 2, 3, 4, 6, 10 and 20 schools per CCT. This makes it very difficult to reach any clarity on
how many schools the project actually impacted. However, it is at least 2,000 schools. This may be an
under-estimation as a number of the mentors noted that schools neighbouring ones where they
worked or through the CCT were copying the practices in the focus schools.
The finance controller of the main funder shared his understanding that the project was meant to
impact on a large number of schools and argued that,
“We thought from the plan that this [project] was good value for money as they [mentors] are
impacting so many schools across so many districts. They were meant to impact on thousands of
schools”.
As explained earlier drawing a direct causal relationship between an intervention of this nature and
improvements in such complex organisations as schools, on which so many other variables are acting,
is highly problematic. The lack of a project specific baseline survey compounds this challenge.
2 As some CCTs have retired, died or left the service and not been replaced the actual number of CCTs the mentors worked with had fallen to 313 by the end of this phase of the project, however as some were covering two Coordinating Centres, in total 352 clusters of schools were included in the project.
31
Therefore, the findings in this section and the conclusion which are rooted in those findings are mainly
based on qualitative evidence provided by the CCTs and the school-based interviews and
questionnaires filled in by the school head-teachers. These makes causal assertions with evidence
which gains some validity by repetition and recognition.
What can be said with certainty is that the fourteen schools which were visited for the evaluation
process had in all but two cases experienced the project directly and all these school management
teams asserted that the project had a positive impact on their school in various ways. The schools
report that the project was experienced and impacted at school and classroom level in a fairly similar
way. This indicates that although the mentors introduced somewhat different programmes and
approaches there were commonalities. The common threads were the implementation of child-
centred methods combined with support supervision, the presentation of the basic requirements and
minimum standards to schools while making these requirements and standards accessible and
implementable, and the interpretation and repackaging of the thematic curriculum or aspects of it.
These are seen across all the schools visited in the four core PTC catchment areas visited. The
commonalities in improvements that the schools noted include:
Improvements in support supervision by the CCTs and in some instances by the school
management using enhanced user-friendly instruments;
Much greater use of instructional materials and particularly a growth in the ability to develop
teaching aids based on local materials;
Improved and varied forms of learner assessment with more use of continuous assessment;
Reduction in the use of corporal punishment;
Improved management capacity exhibited in various ways;
Improved capacity to plan at cluster, school and classroom levels.
Some, but not all, the schools also experienced:
Improved capacity to manage and teach early grade literacy – particularly reading – and
numeracy;
Improved organisation, use and management of their libraries and effective use of the library
period;
The introduction and use of learning games in the classroom;
Increased community involvement in schools and greater willingness to support school
development activities;
Increased stakeholder (district executives, DEOs, DIS/MIS, religious authorities) knowledge
about how to support their schools and teachers;
Ability of the district, PTC and education authorities to plan and work together and support
each other;
Ability to unpack the thematic curriculum and repackage it as schemes of work and lesson
plans that the teachers can use in class;
Improved record keeping in schools and in PTCs;
Improved school and learner safety.
The CCTs report that there was a generally positive response to the changes introduced (see Figure 4)
although nearly half of the CCTs report that some of their schools were not enthusiastic about the
changes introduced through the innovations the mentor was introducing.
32
Figure 4: Reaction of Schools to Changes (n=17)
The school head-teachers were asked about their relationship with their CCT. A remarkably large
percentage (50%) of the CCTs had visited the school in the last week. This probably relates to the CCTs
nominating the schools and visiting to ‘warn’ them of the impending evaluation visit. In addition all
but two of these schools were focus schools for the project so inevitably see their CCT a lot more
regularly than the other schools in the cluster probably do.
Figure 5: When did your CCT last visit your school? (n=14)
More useful is the head-teacher assessment of the regularity of the visits of the CCT to their school
over time (figure 6). Again this is influenced by the status of the majority of the schools being
0
1
2
3
4
5
6
7
8
9
10
Schools like changes Schools do not likechanges
Some schools do/somedo not
0%
10%
20%
30%
40%
50%
60%
In last week In last month In last term
33
coordinating centre and focus schools. The vast majority of the head-teachers reported that their CCT
had been visiting regularly or very regularly (monthly or weekly).
Figure 6: Regularity of CCT visits to schools (n=14)
In the satisfaction questionnaire the head-teachers were asked about how happy they and their staff
are with the support their CCT provides their school and whether the CCTs’ interventions meet the
needs of the school. Interestingly most of the head-teachers indicate that the CCTs while meeting
their needs, do so only to ‘some extent’ (figure 7). Given the intensity of support that these schools
have had from the CCTs over the last two years we might have expected this response to be more
positive.
Figure 7: Does the CCT meet the needs of your school? (n=14)
0%
10%
20%
30%
40%
50%
60%
Very regularly(weekly)
Regularly(monthly)
Not regularly Occasionally
0%
10%
20%
30%
40%
50%
60%
70%
80%
Yes No To some extent
34
Significantly, although they may not feel that the CCT meets all their needs, a very high proportion
(93%) of head-teachers indicate that they and their staff are happy with the support they get from the
CCT (figure 8). This apparent contradiction may be due to the realisation by the head-teachers that
although the CCT does not meet all their needs as a school their CCT does his or her best given the
circumstances. Another reason for a positive response represented in figure 8 may be explained by
figure 9.
Figure 8: Are the school staff happy with the support your school gets from your CCT? (n=14)
When asked about whether the service they get from their CCT has changed over the last two years
(since the start of the mentorship intervention) a significant 86% of the head-teachers asserted that
the support has got ‘better’ or ‘a lot better’. It is even more significant if we remove the responses of
the two schools which were not focus schools. Then 100% of head-teachers report better or a lot
better service. It is important to note that even though the CCTs were targeting the focus schools they
did not ignore their other schools – hence the two non-focus schools report that the service they get
from their CCT has ‘stayed the same’. It would be a serious concern if the project had advantaged the
focus schools at the expense of their other schools.
Figure 9: Changes in the service from your CCT over the last two years (n=14)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Yes No To some extent Some are, othersare not
35
It is interesting to note that the CCTs have seen a change in the behaviour and commitment of their
head-teachers. There is widespread reporting of head-teachers doing support supervision of their
teachers and checking of schemes of work and lesson plans which they either used not to do, or did
in a way that as one CCT put it, “harassed the teachers”, and often lead to “warning letters”. This, the
CCTs noted, has reflected the generally more relaxed, collegial and improved relationship between
head-teachers and their staff. It was noted that head-teachers are now more likely to teach than
before and are also more likely call the CCT and request support with specific school-based CPD
events. Such changes in professional and personal behaviour should be sustained. However, it was
pointed out that the constant rotation of staff and particularly head-teachers which many district
education offices effect may impact on and undermine even this impact.
The CCTs also noted that most of the head-teachers are able to talk to learners and parents with more
confidence and in a more transparent way. Perhaps most interesting is that a few CCTs commented
that some of the head-teachers have so gained in confidence that they allow their teachers to do
lesson observations of their lessons, and can engage in robust discussions on their own teaching. This
goes with a notable improvement in the preparedness of some head-teachers to adopt a pedagogical
leadership role, whereas in the past they left this to the senior teachers. The CCTs put this down to
head-teachers, having had no specific training for their role, not being clear until now of what role
they can play in relation to classroom teaching and lesson planning. However, one group of CCTs noted
that some of their head-teachers had resisted the changes and even after being trained refused to
undertake classroom observations. This was in part explained by teachers who one mentor reported
told her after a workshop that it “reminded them of what they learned at college and should be doing”.
This could be the premise on which to go on failing to change.
7. Findings: Measuring Metrics The ability to measure metrics in this project has been compromised. The lack of a project baseline
and clearly enunciated and widely shared numerical targets guiding delivery make it impossible to
ascertain the impact of the project with any level of certainty. This is further compounded by the level
of autonomy provided to each mentor and CPTC in determining what elements of the BRMS to
implement as part of the project. Finally, some of the earlier data collection processes which were
needed to track PLE results of the 73 target districts are compromised.
0%
10%
20%
30%
40%
50%
60%
70%
Stayed same Worse Better A lot better
36
However, notwithstanding all of the above, an attempt will be made to try and identify some
relationships between the intervention and observed and measurable changes in the colleges and
schools. Unfortunately, much of this section relies on those involved in the project making associations
and attribution. This, from a research point of view, is inadequate. However, such associations may
assist UNICEF in determining the value of the project and how to roll it out. They may also allow
UNICEF to identify likely metrics to collect in future when implementing a version of this project
elsewhere, or in measuring the impact of the extension of this project.
The first table, table 4, explores the metrics agreed prior to the evaluation with UNICEF. The original
table is included in Section 4.6. The last column, which described the proposed method by which the
data would be collected, has been combined with the third column and a new column added which
explains how the data were collected or the challenges faced in collecting those data.
Table 4: Status of Proposed Project Metrics
Focus area Question to be answered by metrics
Metric to be measured and method used to collect data
Status of this data
Head teacher performance
How do we know if the head teacher performance improved?
CCTs and teachers report that the head teachers have improved management skills over the last 2 years/as a result of the project using satisfaction questionnaires and interviews with CCTs and teachers.
Data collected using satisfaction questionnaires and interviews. The results are presented below.
Teacher Attendance
How do we know if teacher attendance has improved?
Select a week in 2011, 2012 and 2013 and compare attendance of teachers (no of absentee days as a percentage of all days) and number of teachers absent for more than 2 days in any of these weeks.
CCTs and mentors claimed following the strike collecting teacher attendance data would be too provocative. Learner attendance data was therefore collected as a possible proxy for teacher attendance. However, the data is inconclusive as described below.
Reduce teacher absenteeism in the target 73 target districts by 5 percentage points.
See above.
Improved retention of learners in focus schools
How do we know if the project has impacted on learner retention?
Increase in primary school learner completion rates over the period of the project by 2 percentage points in districts where the project is operational
EMIS indicates that completion rates, in the target districts, which were 64% in 2011, rose to 67% (for both boys and girls) in 2013. This is a three percentage point increase.
Performance of the school as a useful proxy for quality of the school.
How do we know if the performance of the education received in the schools has improved?
3 percentage point increase in Division 1 and 2 passes and 3 percentage point decrease in the number of fails in the Primary Leaving Exam (PLE) in sample schools between 2011 and 2013. A random sample of 50 schools from among the schools where the target CCTs have been working will be selected from the national EMIS data base and their exam results for 2011, 2012 and if
Of the 49 districts for which we have data 29 saw an increase in Division 1 and 2 passes, while 19 saw a decline and 1 stayed the same between 2010 and 2013. Division 1 passes in the 4 districts where the sampled CPTC’s are sited were compared. 3 had seen increases in Division 1 passes by 13%, 24%, and 82%, while the 4th saw a decline of 66%. Most of the target districts saw increases in Division 1 passes with some increasing dramatically. However,
37
Focus area Question to be answered by metrics
Metric to be measured and method used to collect data
Status of this data
available 2013 evaluated and trends identified. Another 20 schools will be selected from other neighbouring districts where there was no impact from the mentored CCTs.
attribution of any increases to the project would be inappropriate.
Improved P3 and P6 NAPE national literacy and numeracy test results by 3% points. Use 2010 NAPE results for the 73 districts as a baseline and compare with 2012 results for same districts.
Of the 73 districts between the 2010 NAPE and 2012 NAPE 62 districts recorded an increase in P3 literacy scores with an average increase of 6.9 percentage points across all 73 districts, while 53 districts had an increase in P3 numeracy scores with an average increase of 6.7 percentage points across all 73 districts. The P6 literacy and numeracy scores were broadly stable over the same period. As most of the focus of intervention was on P1-3 the P3 results are significant. However, it is impossible to claim causation.
CCT performance
How have the professional habits of CCTs changed as a result of the work of the mentors?
(i) The number of days 20 randomly selected mentored CCTs have spent in schools and the number of schools they visited compared to the numbers for a randomly selected group of 10 non-mentored CCTs. (ii) The number and type of activities these 20 mentored CCTs did in these schools compared to the number and type recorded by the non-mentored CCTs.
CCTs indicated in interview and questionnaire responses that many extraneous demands impact on the time CCTs spend in schools. Where they visit more this was attributed to the receipt of a motorbike from the project and not from working with the mentor. Therefore, this metric was replaced by a measure of perceived changes in the service the CCT provides as part of a questionnaire administered to the head-teachers in the 14 school sample, including 2 schools which are not part of the focus schools. This shows a marked increase in satisfaction in the support and services provided by the CCTs over the last 2 years (see above), while there has been no increase (or decrease) in satisfaction in the 2 schools which are not part of the project.
It should be noted also that – as the example of figure 10 below indicates – metrics tend to be limited
and uni-dimensional. Knowing that relationships have changed between the 17 CCTs and their
colleagues does not tell the reader how they have changed. Such details are provided through
qualitative information.
When CCTs, DPOs and the CPTC principals were asked during the interviews to provide two
measurable indicators to prove the impact on the CPTC and the schools, they provided the list in table
5.
Table 5: Measurable Indicators of Project Impact Listed by CPTC Based Stakeholders
38
Indicator: Measured by: No. identifying this:
Improved planning Presence of lesson plans of agreed format and quality in schools and PTCs
8
Improved CCT support to schools
Number of CCT visits to schools and quality of what they do in class based on teacher satisfaction
7
Improved learner performance
Scores in word recognition and reading skills in lower school; PLE results in upper school
5
Improved ability of schools to use IMs
Count of appropriate IMs on walls; Count of IMs used during observed lessons; Evidence of improved school ability to develop and store IMs.
4
Improved teacher performance/ effectiveness
Timing of learner talk compared to teacher talk in lesson observations; Teacher use of a variety of methods to check understanding in lesson; Improved formulation and use of lesson plans.
3
Improved head teacher performance / management
Holding of meetings outside school time; PLE and other indicators exhibited in head teacher office (talking office/talking school); Monitoring reports on teaching done more often and quality of reports more supportive; Improved filing; Relationships with staff and learners more positive based on satisfaction surveys conducted among learners and teachers; Increased number of CPD activities in school with measurable impact; Timetable in place and observed.
3
Improved attendance at school
Counting of head-teacher, teacher and learner attendance regularly.
2
Increased learner safety
Schools with no reports of using corporal punishment; Learner reports of less use of cane.
1
This list was endorsed by the responses in a survey of the eleven mentors who responded.
7.1 Metrics Measuring Impact on the CPTC No specific metrics were planned for the CPTCs. The interviews with CPTC managers and the responses
from the mentors indicate that there are potential metrics that could have been measured at the start
of the project and then monitored during the delivery process. These include:
The performance of student teachers. The perception in one college, as reported above, that
the number of 1st year students failing their inaugural year had dropped from about 15% to
well below 0.5% is significant. Obviously this assumes some backwash effect of what the
mentors are doing in the schools to the pre-service tutors. This can we assumed as the CCTs
also tutors in all the colleges so may take the excitement that they feel over the new
approaches and learnings they have experienced into the lecture room. This was compounded
in three of the colleges where project workshops were open to the tutors and in two of the
colleges where workshops were held by the mentor and CCTs for student teachers on such
topics as developing low cost IMs.
Improved quality of CCT engagement with schools. This could be measured by the presence of
a programme of visits with clear objectives to be achieved through the visits; from the number
of schools each CCT visits in a set time period; and the length of time the CCT spends on
average in a school and the range of activities undertaken when in each school.
39
Improved CCT job satisfaction. This can be measured by administering satisfaction
questionnaires to samples of CCTs and calibrating the satisfaction scores on a range of items
concerning motivation, commitment, enjoyment and preparedness to work extra hours and
commit personal funds to access schools. This could be triangulated by monitoring trends in
days taken off by CCTs due to illness and general absenteeism.
Improved sharing of information and approaches among CCTs and with DPO. This could be
measured by the number of information sharing sessions and the quality of the input.
Improved relationship between CCTs and between CCTs and the DPO. As these are critical
relationships which have an impact on the planning of activities and the potential impact of
the CCTs on their schools this is a critical metric (see Figure 10). As in the example provided
this can be measured through a question in a CCT satisfaction questionnaire. The follow up
response made clear that for all but one of the CCTs this was related to improved joint
planning and team work. This is significant as processes for joint planning and team work are
easy to implement and cost little.
Improved planning of activities and interventions. This can be measured through the level of
coordinated travel plans, the quality of plans and the level of adherence to plans. It can also
be seen through the involvement of the DPO in planning and monitoring of CCT support to
schools.
Figure 10: CCT Assessment of Improved Relationship with other CCTs and DPO as Result of the Work
with the Mentor (n=17)
7.2 Metrics Measuring Impact on Schools The project is designed to impact on the 3,520 primary schools which the 352 CCTs are working with
as part of this project. Given the focus areas that the mentors and CPTC management teams identified
for the project the BRMS Mentorship intervention should impact on the management capacity of the
head-teachers, the use of IMs in the classroom and improved support and monitoring of teachers in
the classroom. These can be measured in terms of proxy indicators. Improved management capacity
can be measured through satisfaction questionnaires filled in by teachers in which they are asked to
0
2
4
6
8
10
12
14
16
18
Yes with bothother CCTs and
DPO
No, I see nochange with either
CCTs or DPO
With some but notall CCTs
With DPO but notother CCTs
40
rate changes in the performance of the head-teacher over the period of the intervention. When this
is combined with the assessment by the head-teacher on whether the mentor helped him or her we
have some useful data. Of the 14 schools visited 12 of the focus groups of teachers provided a usable
response to the question ‘Have you seen any changes in the way that your head-teacher works and
engages with you and the SMC as a result of the BRMS project/work of the mentor and CCT? What
are those changes?’ Two groups of teachers said that they could not tell as the head-teacher is new
to the school, three said they had seen no change (two of these are non-focus schools) and seven said
there had been a change. These 7 groups of teachers identified the changes identified in figure 11.
Some schools produced a number of responses.
In these schools the impact was also measured in improvements in the school.
Figure 11: Changes noted by teachers in their head-teacher as a result of the project
Such data provides substance to the stark numerical data. Given that the 15 mentors were set the
ambitious task through the CCTs and DPOs of impacting on over 3000 schools the fact that the project
was seen to impact on the sample of 12 schools visited, which were project focus schools, and in a
more limited way on one of the two non-focus schools positively, is impressive. Even more so is that
they impacted in a marked way on seven of the ten head-teachers who had been in the school for the
period of the project.
More specifically in a number of schools the impact of the mentor could be measured on the way
reading and mathematics are taught, on school safety and cleanliness, on lesson planning and
textbook utilisation, and on the way the library is managed as well as on the use of the library period.
As these are not common to all the PTCs they do not help in measuring the impact of the project in
global quantitative terms. However, they provide supporting data that the decision to allow each
0
1
2
3
4
41
mentor and PTC to set their own focus areas allowed impact on a range of areas over and above the
broad-based improvements in the majority of schools.
The local and foreign mentors impacted on schools in rather different but complementary ways. The
local mentors tended to link their inputs to the thematic curriculum, have a focus on community
involvement in the school, build political and practical school support alliances between the college
personnel and the officials based at district level, and overall dealt with systemic issues. This allows
more coordinated support to schools and helps schools manage the new curriculum and policy, but
the impact on the classroom is mainly tangential. In contrast the foreign mentors generally seemed
most at home in schools running workshops and observing classrooms. Their impact at class level was
often tangible, but they were rarely impacting on the overall system supporting schools. The metrics
used need to reflect these differences.
7.3 Metrics Measuring Impact on Learners There is a lack of data on the impact of the project on the learners in the focus schools. UNICEF tried
to collect a range of data for all the focus schools and to that end hired a consultant to collect such
data. Unfortunately the consultant failed to deliver and as a result the project lost the opportunity to
collect considerable useful data real-time. As a result during the evaluation I attempted to collect
relevant data that would indicate the impact of the project.
It was considered by the CCTs as too sensitive to gather data – where it existed – on teacher
attendance at school. However, the same sensitivity does not apply to learner attendance. Learner
attendance is likely to be related to a number of factors, only a few of which are susceptible to impact
by a project of this nature. However, if improved learner attendance and retention can be shown
across a number of schools where the project is operational it may indicate that the head-teachers
and teachers are attending more regularly, teaching more rigorously and effectively and enjoying their
work more. Certainly these were indicators related to the impact of the project that were regularly
cited by CCTs, head-teachers and teachers during the evaluation exercise. To see what reliability there
is in these assertions time-lapse data on learner attendance was collected from 9 schools in the area
served by one CPTC for the same two weeks in 2012 and 2014 (see Annex 1). Analysis of these school
attendance data indicates that there are no clear patterns emerging. In four of the 9 schools the
learner attendance has dropped between 2012 and 2014. In the other five schools the enrolment has
increased. Discussions with teachers during the research indicate that the main driver of changes in
enrolment is the presence and opening of low-cost private schools in the area. Parents move their
children between schools where there is the choice between private and state schools with
bewildering regularity based on access to funds. When they have some money after the harvest or
during periods of improved retail sales they enrol their children in private schools only to move them
to the fee-free state schools during leaner periods. While this would also be expected to impact on
retention of learners the districts where the project was operational saw a 3 percentage point increase
in the proportion of learners completing P7 between 2011 and 2013. Significantly EMIS shows that in
2013 equal percentages of boys and girls (67%) completed P7. The target was an increase of two
percentage points to 66%.
In some schools the impact could also be measured through monitoring learner performance in
literacy and numeracy. This was one of the proposed performance targets. However, while most
mentors saw improving teaching of early grade literacy and to a lesser extent, numeracy, as a key pillar
of their work, this was not the case in at least one of the colleges visited where the mentor did not
focus on literacy. The impact of improved literacy and numeracy teaching in lower grades may be
measured through internal tests and in certain grades through the use of the NAPE results. However,
42
these tests are considered inconsistent. Therefore, UNICEF could consider investing in undertaking
Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA). These
tests would add value and allow the impact of the intervention on learners to be calibrated accurately
and understood. However, for this project these tests should have been undertaken with foundation
phase learners at the start of the project and then again after two years of intervention. The other
method is to monitor the PLE results in English and local language and also in mathematics, to see
whether the changes wrought in the early grades translate some years later into improved PLE results.
The disadvantage of using this measure is that the effect is only measured four years and more after
the initial input. This means that direct causal relations between these results and the project will be
impossible to establish, even if there were no other external programmes impacting on the schools
being measured. Then significant changes in performance at PLE, controlling for the effects related to
other variables in the school and in the learner population through the use of statistical software, can
be associated with the intervention. Such analysis is beyond the scope of this study.
8. Findings: Project Management and Cost Effectiveness The evidence for this section of the report is drawn from a questionnaire administered to all the
mentors (see Annex 3) past and present who had spent at least a year as a BRMS Mentor (there was
a 61% response rate), and interviews conducted with seven of the 18 mentors, the main funder, VSO
and UNICEF managers, a senior MoES manager, as well as the interviews conducted with school and
college staff and district education officials during the field visits as previously detailed.
8.1 Preparation, Recruitment and Induction The mentors and college managers indicated that the CPTCs where they were placed were not
involved in the planning of the project. However, all of the CPTCs had been represented at a UNICEF-
MoES workshop explaining the purpose of the project before the mentors arrived and received
documentation on the purpose of the project in a series of workshops. Notwithstanding this
preparation some of the CPTC managers seemed to have been largely ignorant – or played ignorant –
of the purpose of the project and the mandate provided to the mentor. MoES chose the CPTCs that
would be involved in the project. They were not required to apply for involvement or even plan in
advance for the project. Possibly as a result, at least three of the college management teams were
clearly not keen on hosting the project or reportedly showed little interest in the professional aspects
of the project but looked for opportunities to exploit the project’s funding. This also meant that most
of the college principals had not prepared their DPO and CCTs for the role they would be expected to
play in respect of the project.
There is disagreement among the respondents as to the wisdom of making the mentors report to the
DPO. The MoES and UNICEF stressed that as the DPO is the line manager of the CCTs no one else could
have played this role. Further, the director of VSO argues that the mentors needed access to the
college management and they needed the authority that went with that reporting line, but if they had
reported directly to the principal they might have side-lined the CCTs and DPO. He argued that the
problem lay in not preparing the DPOs adequately for this role nor preparing the mentors so they
appreciated the relationship between the DPO and principal. As he said, “DPOs lack real power as the
Principal has that – mentors expected the DPOs to take decisions but they cannot and therefore
don’t”. This does talk to the need to manage the project in a different way at college level. The DPO
was also meant to be the counterpart to the mentors. This role does not seem to have been fully
43
thought through in the planning. Most of the mentors found the DPO unprepared to play the role of
counterpart. Some faced complete resistance and one reported having no professional contact with
the DPO for the whole period of her placement. Given the age and gender of most DPOs it is perhaps
not surprising that some reacted negatively to being mentored. They may have seen this as an implied
criticism of their job performance and ability. This evaluation agrees that DPOs should have been the
mentors’ line managers but should not have been assigned the role of counterpart. This could have
been better fulfilled by a senior CCT.
There is general agreement among most of the mentors that the process through which they were
recruited and inducted was not well conceived. Many of the mentors reported that the information
which they received prior to arriving in Uganda was either limited or confusing. VSO had responsibility
for providing this information. A few reported that they thought they were coming out to take up a
role in an early childhood development project (which was a parallel MoES-UNICEF project for which
VSO was also recruiting) and a couple of others assumed they would be working as pre-service tutors
or ‘teacher educators’. The remaining mentors received accurate information – or read the
information more carefully – and were happy that their role and their skills had been well-matched.
However, all of the eleven mentors who responded to the survey indicated that no one had prepared
them for the financial planning and management role that they ended up playing and they were not
prepared for dealing with the highly charged issues which developed in many of the CPTCs around the
access and use of the project vehicle. UNICEF’s project management readily agreed that this financial
management and oversight role was not planned for but it became increasingly necessary for delivery
that the mentors were able to plan and budget with the college management and ensure funds were
spent on the project.
The main problem with the induction was that the mentors did not arrive in one group, and
experienced different induction processes, with some reportedly only getting one day of induction in
Uganda, before going to their CPTC. The most satisfied were those who arrived in a group at the start
of the project and those inducted by a serving mentor. These got the full induction as planned between
VSO and UNICEF, which was a week with VSO and three days with UNICEF. Some mentors argued that
they found that the induction ill-prepared them for their role. In fact many of the mentors who arrived
during the course of the project reported that they arrived at the college without a clear idea of what
their role would entail. This may be in part due to VSO respondents admitting that they had not been
fully briefed on the project’s genesis and on the definition of mentoring that UNICEF was using.
Further, some mentors claimed that the lack of a defined project offering left them confused and
wasted time, while others “found that liberating and not confusing” as it meant they could be
informed by local need and their own interests. One former mentor argued that the lack of a binding
theme led to fragmentation and failure to knit the projects together in a programme and wasted
money, while allowing lazy or ill-prepared mentors to coast through the placement with little evidence
of delivery and impact.
A number of the mentors further criticised the lack of a contractual requirement, imposed by UNICEF
on the CPTCs, for the management of their CPTC to induct them on arrival at the college. One mentor
asserted that such an induction could have included the CCTs, pre-service tutors as well the district
level education officials. Such a process would have alerted the foreign mentors to the important role
that the district officials play in schools while alerting the district officials to the project. One mentor
confessed that she only realised how critical these officials are to accessing and working in schools in
the latter part of her placement. The mentors further argued that such an induction would have been
the ideal moment to agree on: the respective roles of the mentor, principal, DPO and CCTs; the
intentions behind the project; a protocol for the use of the vehicle; and written procedures for
44
accounting for project funds. Mentors felt that such clarity at the start of the project would have
assisted in ensuring all college managers understood what the project was all about and would have
provided a semi-public space for them to declare their professional commitment to the project. They
believed this might have prevented later contestations in their colleges.
8.2 Project and Financial Management
Project management
The representative of the main funder stated,
“Getting the right people in the right place was key and [UNICEF] got good people – very experienced
people. These mentors were left to find out a lot of things for themselves. There was a lot of differences
between different people – [UNICEF allowed a] lot of freedom for mentors, this could have been fine,
you don’t want very experienced people and then script them. However, I believe that UNICEF should
have helped shepherd this and ensure quality across the field. But was there a system in UNICEF to
make it into one programme – how has UNICEF steered the process?”
There is general agreement by the main partners, the mentors and funder that there was little
coherence between interventions, which they claim made it very difficult for them to inform each
other and to measure impact. The reality is that there was significant overlap between interventions.
For example most of the mentors included support supervision, improved lesson and school planning,
the development of low cost IMs and particularly the promotion of child-centred teaching in their
offering, with a limited focus on assisting teachers in dealing with implementing positive discipline.
This indicates that the BRMS framework and the decision to focus on four of its pillars, and the level
of interaction that mentors had with UNICEF and each other led to some commonality. However, there
is some truth in the claim that management could have been stronger in providing pedagogical and
project management support to the mentors and spreading knowledge of the different interventions
being implemented to all mentors timeously, while providing the glue that would have made the
mentors see the synergies in the different approaches they were taking. This, along with a strong
baseline survey and process evaluation at the end of the first year, would have allowed the project to
be easier to manage and present as a coherent programme of articulated activities and outcomes. The
mentors and partners are correct in arguing that they could have been knitted together effectively
and if so would have been able to inform the system better at national level.
From the interview at national MoES level it was clear that this project has had limited impact, unlike
the parallel ECD mentorship project. There seems to have been too little advocacy of the model and
its successes, which would have emanated from more networking of the mentors with each other, key
MoES officials and other projects doing similar work in Uganda. The general view was that the regular
review meetings met UNICEF’s reporting requirements rather than providing added substantive
pedagogical and intellectual value. However, UNICEF pointed out these meetings were instituted on
a termly basis purely for the mentors and colleges to share their progress and approaches and were
not part of the UNICEF’s reporting requirements to head office or to the funders.
The main funder asked how impact can be measured with no baseline and also said that from the start
they warned UNICEF that the plan was very ambitious and depended too much on the quality of the
mentors and their ability to work in what would likely be a difficult and lonely role. He didn’t think
that the project planning team had adequately factored in these issues into the planning otherwise
there would have been more professional support built in for the mentors. Some, but not all, of the
mentors reflected this view.
45
The project was mainly the responsibility of the education specialist who took over as UNICEF project
manager in September 2010, after the project had been conceptualised by the national UNICEF office
chief of education. For a period of 8 months (August 2012 to April 2013) the education specialist was
not involved with the project. When she resumed her duties she took up her previous project
management role. During the period of her absence the chief of education took over more direct
project oversight and more junior UNICEF employees based in regional UNICEF sub-offices managed
the operational support. This experiment in more accessible support appears to have caused more
harm than benefit. This was because the staff in these offices were said to lack the knowledge and
authority that the central UNICEF team possessed. As a result various mentors said they were left
frustrated. A UNICEF senior manager asserted that the project required a dedicated stable staff
complement on at least two year TA contracts. The relatively high turnover of UNICEF staff on the
project certainly made consistent delivery difficult. This combined with concurrent financial problems
which are discussed below undermined confidence of some of the mentors in UNICEF and its
management capacity.
Financial management
The funds were drawn down from UNICEF each term into each college account. The amount was
determined by the workplan for the BRMS related activities which the mentor and the college
planned, and included allowable expenses and allowances. UNICEF applied its standard procedures of
checking the proposed budget and spending for the next term before transferring the funds. The
mentors accessed the funds for their work in accordance with the college procedures and on
instruction from the DPO who signed off on the activities. The principal approved and then the college
bursar was instructed to release the funds from the account. At the end of each term the mentor and
DPO conferred and compared their spending records and then the DPO submitted a financial report
to UNICEF detailing how the drawn-down funds had been utilised.
While this process seems sound and allows for due diligence to be applied at various stages in the
process some of the mentors indicated problems in the process. They criticised the way that in their
colleges the lack of strong financial systems put the mentor on a collision course with the college
management. One mentor critiqued how UNICEF deals with financial discrepancies, pointing out that
the UNICEF system of ‘blocking funds’ after six months of failing to get accounts approved, impacts on
the work of the mentor; no one else suffers. She argued that if UNICEF instituted spot checks and
more regular audits on their funds and if colleges were forced to keep accurate financial records based
on more robust systems of allocations and receipting and were clearer as to what is allowable under
the funding, and were held personally liable for funds which are misused, abuses would be reduced.
Perhaps, more realistically, a number of the mentors argued that UNICEF needs a strategy to deal with
perceived financial mismanagement by colleges. One of the mentors also argued that sometimes she
found that UNICEF talked directly to the college management and sometimes to her about project
finances, without sharing the same messages with the other party. She explained that this provided
the college management with wriggle room in relation to the finances and the budget and created
tension between herself and college’s management. She argued for more transparency in the
information that UNICEF management shares with the college managers and the mentor. Importantly,
the main funder was critical of UNICEF’s financial systems, “UNICEF’s financial management is weak”.
He argued that the fact that he could remove $150,000 from the budget with no obvious impact on
the project and the final under-spend of well over 10% of budget indicated poor initial project planning
and budgeting “the budget was not thought through well”. However, given the decisions UNICEF and
this funder made in relation to reducing the number of motorbikes provided to CCTs and providing
46
none of the planned 50 motorbikes (with budgeted fuel allowance) to DISs as well as other
modifications to the delivery model, such an under-spend it not hard to explain.
There was considerable reporting of a period of over five months when, during a key stage in the
project’s delivery process, no project funds were available due apparently to UNICEF implementing a
new financial system which saw the New York UNICEF office apparently misallocating the money. This
compounded a problem caused by the tardy transfer of funds from the main funder to UNICEF3. The
response to this problem in the various colleges was largely dependent on the relationship between
the mentor and the college principal. Where the relationship was strong the college advanced the
money to sustain basic delivery, however all mentors saw a curtailing of activities. For most of the
mentors this was part of a pattern of late payment and withholding or blocking of funds where
misappropriation was suspected. The latter is a standard UNICEF policy where funds remain
unaccounted for after six months, and is implemented automatically. Often the cause of the problem
lay with the college management who would submit their reports and drawdown requests late, even
if the mentor had apparently supplied the data on time, or would struggle to account for spending.
Six of the 11 mentors who completed the survey (see Annex 3) for this evaluation reported that they
had used their own money to fund the gaps in funding, either caused by the delay in the transfer from
the donor or “blocking” of the partner (in case of unaccounted funds for over 6 months). This is
completely unacceptable. As the director of VSO and the main funder both pointed out an
organisation such as UNICEF should be able to advance funds to cover such shortfalls and not rely on
the goodwill of the mentors and the college management to cover the funding gap and keep the
project running. UNICEF should realise that such situations inevitably strain relations between the
mentors and the college management and undermine trust between the CCTs, districts and their
schools and the mentor. One mentor summarised the dilemma neatly when she said,
“This delay [in receiving money] has had a detrimental effect on the reputation and therefore the
importance of the programme with the districts and the schools … I think without the issue of money
the relationships would have been more open – but without the money the college would not have
considered being involved”.
While a few of the mentors were convinced their college management were misusing project funds,
all the colleges accounted for all the funds transferred by UNICEF for the project. Even where funds
seemed to have been misused UNICEF audits uncovered no evidence of such. In the minds of some of
the mentors imposing a well-funded project of this nature with, what they argued, were too few
checks and balances on the cash-starved colleges and college managers was asking for trouble. One
suggested that the colleges should have had to bid for the project and the funds based on deliverables
which they would then be held accountable for. She argued this would have ensured commitment to
the project and set measurable targets with managers accountable for their attainment.
Although the project management team in UNICEF, the MoES and the local mentors defended the
management of the project, arguing that it was responsive and did its best to ensure the project
delivered, this view was not shared by other project partners – including the most foreign mentors,
the main funder and VSO. They pointed out that the way that UNICEF had managed the illness,
absence and movements of key project managers and the impact this had on the start-up process,
which took much longer than planned, and on continuity, as well as the sharing of information and
the quality of reporting, impacted negatively on the delivery and management of the project. VSO was
3 UNICEF requested the third budget tranche of nearly US$1.5 million on 31 August 2012, but it was not transferred to the UNICEF account until 10 October 2012.
47
particularly critical of the way that they felt the value they and the early mentors could have added,
to the project during the planning and early delivery stages, was never sought. The organisation also
argued that the project would have benefited by the project being managed by a steering committee
including all the partners. However, UNICEF management pointed out that during the planning the
VSO office was not structured to play such a role. All agreed that if this had been possible it would
have led to more involvement and commitment by the partners including MoES, and also, in VSO and
the funder’s view, to UNICEF having to clarify the project targets. VSO also argued that, with such an
approach, a proper project ‘logframe’ with clear targets and a baseline survey, or at least a national
needs assessment, would have been put in place, which would have helped create articulation and
cohesion across the project. Most of the mentors agreed with these criticisms. UNICEF provided such
documents during the evaluation – they existed but apparently had not been used by the main
participants during the project to guide delivery and had not been used by UNICEF to hold the mentors
and colleges accountable or measure impact.
It does seem that as a UNICEF flagship project more attention could have been paid by the UNICEF
country office to hand-over and continuity to ensure that the funder, partners and mentors felt
confidence in the management of the project. While some of the criticisms of the management of the
project are somewhat ill-informed and can be contested there is no doubt that those partners most
impacted by the management of the project perceived that there were weaknesses in the financial
and project management which had material impact on the ability of the project to make as much
impact as it perhaps could have done. This was perhaps because some, but by no means all, of the
mentors and some of the partners felt that UNICEF, the MoES, the CPTCs and VSO, “did not share a
coherent and coordinated approach to an undertaking which was of strategic national importance”.
Perhaps most worrying is that a number of the mentors shared the view that “no one in the [UNICEF]
office in Kampala was really interested in our experiences from the field … the Good Practices booklet
was a hoop-jumping exercise”. It should be noted that UNICEF took particular exception to this view,
arguing the Best Practices booklet was a sincere attempt to document and replicate best practice
which emerged through the project.
Finally, there is a general belief from the main stakeholders in the project that the weaknesses detailed
above have had a negative impact on the planning and sustaining of the main thrusts of the project in
the immediate aftermath of the departure of many of the mentors at the end of their contracts. They
also argue that the lack of clarity on what the next stage of the intervention will look like has
undermined the institutionalisation of impact.
8.3 Value for Money The model is an expensive one to implement as it involves considerable capital expenditure and uses
foreign mentors. VSO calculates that the annualised costs for each foreign mentor comes to $26,000.
This includes their air tickets, medical insurance, volunteer allowances, housing/furnishing
allowances, pension contributions, and basics for life in rural Uganda including gas cylinder, water
cylinder, and a mosquito net. The funder for the project in 12 of the colleges4 estimates the overall
cost per college for the project averages out at $330,000. This includes the cost of the mentor, the
project vehicle (each mentor and college had use of a project vehicle which is a double-cab 4x4 pick-
up), the purchase of 127 motorbikes for CCTs, and other material and support costs. The unit cost for
each of the colleges with a local mentor is less, but not substantially so, as most of the costs were
4 Three funders funded the project, with the main funder being the Embassy of the Kingdom of the Netherlands with 12 colleges, Irish Aid with one college, and UNICEF has funded the intervention in the 2 colleges with local mentors.
48
fixed, such as the project vehicle and driver, the motorbikes, and most support costs. Although the
recruitment costs were much lower for a local mentor they are receiving higher incomes than the VSO
recruited foreign mentors.
The finance director at the funder asserted that from the original project plan the funder was
convinced that the project, if delivered as planned, would represent good VfM as the plan involved
impacting thousands of schools across half the districts in the country. Although the original budget
allocation was $4,178,000 only about 84% of this budget was drawn down by UNICEF for the project
- $3.5 million in total.
The assessment of whether the project has provided good VfM cannot be calculated using technical
VfM methodology for the reasons explained earlier. However, we can draw some conclusions which
talk to the value for money of the approach taken in this project. The project cost some US$3.5 million
for 12 of the colleges. Of this some US$700,000 was spent getting the mentors recruited, placed and
paid as detailed above. A larger amount was budgeted for the purchase of project vehicles and
motorbikes for 149 of the CCTs and 50 DISs. In the end, no motorbikes were provided to the DISs and
a reduced number of 127 motorbikes were provided to CCTs. The remaining funds were used for the
drivers’ salaries, workshops, materials, as well as the project’s running and support costs. As we saw
earlier, overall the project cost about US$330,000 per college, or US$10,000 per CCT, including the
motorbikes. Assuming the mentors and their CCTs impacted on about 2000 schools as discussed
earlier this works out at expenditure of US$1,750 per school. While this is not cheap, given the impact
that this research has found that the mentors have had on the CCTs and the schools, this is a
reasonable investment.
However, the value for money of this first investment will be much greater if the impact is sustained
through a well-planned second phase with the impact of the first phase sustained in schools and
colleges. The funder argued that this is only going to happen if the CCTs are funded adequately to be
able to visit schools more and undertake a continuation of the work that they have been doing with
the mentor. This clearly requires deep MoES commitment and support, as well as some coordination
at college level following the departure of the mentors. The next phase should be much cheaper as it
could rely much more on local personnel and would not need the provision of new vehicles. However,
the danger of not putting such a phase in rapidly is that the greater the gap in time between the
mentor leaving and the next phase starting the harder it is going to be for the new phase to build on
the first phase. Any further delay in implementing the next phase therefore impacts negatively on the
value for money of the project to date. The VfM will be further enhanced in the next two years if the
lessons from the project are used to inform MoES policy and thinking in relation to teacher support
and school development. At the very least the lessons from this project should give the MoES pause
for thought about supporting projects which use workshops as their core means of spreading
improved school practice.
The other way of looking at the VfM proposition is whether the US$3.5m could have had more impact
on teacher development and schools if spent in another way. This is obviously speculative. Given that
there is a unanimous view amongst the project stakeholders that using CCTs as the entry point to
improve schools is the most cost effective leverage point to bring about change in schools then we
need to examine if CCTs could have been supported in other ways which would have had more impact
or cost less. There is no doubt that the key to increasing the impact of the CCTs is (i) improved mobility
which is sustained, (ii) providing CCTs with the skills they need to impact more effectively on their
schools and pre-service students, (iii) filling all the vacant CCTs posts and improving the conditions
under which the CCTs work, and (iv) protecting them from multiple workshops and other distractions
from their core work. The project impacted on the first and second of these, although lack of funds
49
for long-term maintenance of the motorbikes may create problems in the future. Impacting on the
third point would require subsidising their salaries (which would be highly problematic and
unsustainable) or developing their offices and the conditions in their coordinating centre schools. The
last point requires political will. As suggested by one of the mentors, UNICEF could have made its
funding conditional on the MoES appointing new CCTs and investing some funds in the improvement
of the CCT’s offices, provision of basic supplies for the CCT and possibly even making improvements
to the coordinating centre school. If this had been done, or is done in the second phase, the
investment that UNICEF is making would undoubtedly have more impact on both the CCTs and the
schools.
If the same funds had been spent on workshops for teachers, which has traditionally been the main
approach to school and teacher improvement in Uganda, as in most other countries, the $3.5m would
have covered a three-day training for just under 20,000 teachers. This is based on the average cost of
a workshop for USAID funded projects in Uganda being $60.00 per teacher per day of workshop. As
already noted the BRMS Mentorship project impacted on over 2000 schools directly. If we assume an
average of 10 educators per school (a fair estimate) then we can assume that the project has impacted
on a similar 20,000 teachers. While spending the money on a workshop for these teachers would have
been simpler, we know from experience that a three day workshop is unlikely to impact on the practice
of more than 10% of those teachers, and without school-based support even these teachers are
unlikely to sustain the training. In contrast, the BRMS Mentorship project has trained and supported
a cadre of 313 CCTs, provided nearly half of them with motorbikes and each college with a vehicle to
undertake school support work. This evaluation shows that in the sampled focus schools real changes
have taken place. No workshop could have achieved this. With limited on-going support there is a very
high probability that these CCTs will go on impacting the focus schools and spread the learned
approaches to their other schools. The final value-add in some CPTCs is that CCTs are feeding this
knowledge on to the pre-service students and are sharing improved methodology with district
education officers.
9. Findings: The Model The model as observed in the evaluation is presented in Figure 12. It should be noted that the
relationships varied somewhat between different colleges based on the relationship that the mentor
had with the college authorities and the district authorities.
Figure 12: Model of BRMS Mentorship Project as Implemented 2011 - 2013
50
As UNICEF is interested in the potential of the BRMS Mentorship approach to be used as a model that
could be replicated it is important to reflect on what the stakeholders believe works and should be in
the model and what they believe could be improved in the model.
There was complete agreement that the key entry points to schools are CCTs and so they were
the correct target of the intervention.
There was agreement among the stakeholders that the mentors were generally well selected,
although a few of the mentors themselves argued that the selection process should have been
more rigorous.
There was a high level of agreement that the induction process and the start of the project
could be improved. UNICEF, the college principals and some of the mentors argued the impact
would have been greater if the mentors had all arrived at the same time, allowing more
concerted preparation of the college staff and district officials. National and college level
workshops involving all stakeholders at the project’s start would have ensured better
communications and more transparency on how funding and the vehicle would be managed
and clarity on everyone’s role in the project. Signed documents would have reinforced these
duties and relationships.
Such workshops at the start of the project would have assisted the colleges, districts officials
and the mentors understand how the project fitted within policy and particularly the BRMS.
All the college stakeholders asserted that the Ministry funding for the CCTs is inadequate –
both for maintaining the coordinating centre and for fuel or transport to visit their schools.
Replication within Uganda therefore requires this concern to be addressed.
The CCTs, the mentors and the college principals see the access to motorbikes supplied
through the project as a key element in the CCTs’ ability to do their job better, impact more
effectively on schools and sustain that impact. This facility should be extended to all CCTs.
51
More CCTs need to be appointed as some have responsibility for over 100 schools, and a few
are responsible for over 300 schools, a ratio which does not allow them to provide each school
with constructive professional support. Many of the present cadre of CCTs are nearing
retirement and those who have already died or retired have generally not been replaced.
Some of the mentors mentioned that this was a key weakness in the mentorship model; they
saw little value in mentoring men who were on the cusp of retirement and had little interest
in changing the way they work. The present CCTs need to train the next generation of CCTs.
The consensus among CCTs and mentors was that except in the larger colleges with over 30
CCTs a single mentor could work effectively with all the CCTs. Where the number of CCTs made
the work of the mentor harder the three solutions offered were for a second mentor to be
appointed, for the number of schools which each CCT had to impact on being reduced below
10, and a longer placement for the mentor. The one solution that was dismissed was that of
the mentor working with a subset of all the CCTs. This was seen as likely to create jealousy
and so problems for the principal.
A suitably senior CCT should be appointed to play the project counterpart role. This CCT would
need to be relieved of their coordinating centre responsibilities. The role of the DPO as the
counterpart to the mentor appears to have been wrongly conceived and developed. It is
accepted that the mentors needed to have a point of contact and line manager in the college
management team and the DPOs, by virtue of their position, should play this role. However,
to play these roles effectively in future the DPOs and the counterparts need to be much better
prepared for it, and they must understand the methods and approaches being advanced and
commit to them.
There is need for a much stronger link between the project and the pre-service training of
teachers. A two-pronged approach whereby educators in schools and future teachers are both
exposed to the child-centred innovations is likely to lead to these innovations being sustained
in schools.
Some CCTs and district officials argued that a key element in the model should be the
institutionalisation of coordinated planning between CCTs and district school stakeholders
including the DEO, inspectors and municipal education officers. This requires a small
budgetary allocation for the meetings and in order for joint school visits the stakeholders
argued for the model to cover some of the DEOs’ and inspectors’ transport costs.
One college management team suggested that the focus on head-teachers in the project
planning should be improved as they are key to the adoption of changes in schools. They
suggested more sustained management training with school visits and use of locally filmed
videos to allow head-teachers and deputies to be able to see what good school management
looks like and as the basis for CPD and sustained school-level support supervision of teachers.
This was being done by some of the mentors with satisfactory results.
During the evaluation a number of the respondents engaged with the thorny issue of whether
a Ugandan mentor could be as effective as a foreign one. As one college principal pointed out
there was a scheme in the late 1990s to provide colleges with local technical advisers but as
he stressed, “they had no new things [to offer]”. This was the main concern raised by the
colleges who had a foreign mentor. There is a realisation that foreigners can convince teachers
that something can work, as one DPO said, “Many Ugandans don’t believe in the thematic
curriculum [but] when a foreigner says it can work then teachers can believe it”. This concern
was not shared by the colleges with a local mentor, as they saw the advantages of having a
mentor who understands the environment and the politics of the education field. Perhaps the
wisest statement on this issue was made by one of the college principals who said,
52
“The person coming from overseas is coming with specific skills and approaches – a local would
not have these, so I suggest a foreigner can start the thing as they will move fast, then the
project can be taken over by a local who would need to work closely with the mentor. It must
not be the DPO”.
Networking mentors with different backgrounds towards the end of the project worked well,
allowing them to share best practice and experiences. This benefited both the local and
foreign mentors as they approached the project and the possibilities of the role differently.
There was some criticism that this networking was not organised earlier by UNICEF. However,
there was also a sense that the mentors did not want such distractions at the start of the
placement. Those that sought more support or to network were assisted by UNICEF in visiting
each other.
There was some disagreement among key stakeholders about the model focusing on mentors
working in schools directly. It was argued that the model should focus on mentors working
directly with CCTs and only using the ‘model’ school attached to the college if there is need to
demonstrate in situ. As the head of VSO in Uganda said,
“Mentors were meant to impact via 20 or 30 CCTs and then to a thousand schools sometimes,
this is too long a line, and contains too many assumptions about impact … There is not much
of a role for [foreign volunteers] in schools as the aim is broad based impact … It should have
been based on the needs of the PTC and CCTs’ needs – this was a missed opportunity”
However, this misses the point that CCTs were most impressed by seeing the mentor
improving reading, the library, the management and so on in their schools. It also provides
both a potential critical mass in schools pushing for change and real pressure on the CCT to
carry on supporting them in the same way they did while the mentor was with them. It is an
integral part of the model. In reality most of the mentors and their CPTC managers agreed on
a target number of schools for each CCT to work with during the placement, which was often
not the suggested ten per CCT. It is important that the CCTs and colleges have that flexibility.
When modelling an important innovation of this nature it is critical that the management is
strong and that the documentation is above criticism. This project would have benefited by
stronger and broader-based initial planning, a stronger management structure, a consistent
and dedicated project team, signed documents defining key protocols with financial
implications, and a credible specific project baseline survey followed by a process evaluation
conducted during the early stages of the project.
Finally, if UNICEF is to promote this project design as a model it needs a clearer definition of
what a UNICEF ‘mentor’ is and does. VSO, for instance, has a clear definition of a ‘mentor’ and
it is not the same as the one used in the project. Even though this may appear as semantics,
it is core to the project and its replication. A lack of clarity on this combined with a lack of a
clear thread or cohesive theme for the project allowed, as one mentor put it, for “innovation
and hard work on one side and laziness and wriggle room on the other side” – both for the
mentors and the CCTs.
10. Conclusions “The programme has impacted beyond what we expected” (VSO respondent).
There is no doubt that the approach used in the BRMS Mentorship project was broadly well-conceived
and has had a significant impact on both CPTCs and the focus schools. While this evaluation cannot
53
provide adequately reliable data on the impact of the project on CCTs and schools due to difficulties
of causation and attribution I believe that an adequate case has been made from the data that is
available and the testimonies of the main stakeholders and recipients of the project to determine that
the intervention has made deep impact in many colleges and schools. Because much of this impact
relates to behaviour change and fundamental changes in the way that educators go about their jobs
there is good reason to think that many of the changes will be sustained. The recipients of the project
believe that many of the innovations will be sustained as they make their teaching and school
management more fulfilling and they are enjoying the sense of success.
This confidence in the sustainability of much of the project’s practices is also based on the role of the
CCT in both understanding the innovations and implementing them in their schools. They are the key
link between the system and schools. Their involvement and commitment to take the innovations
forward is based on practical experience of them. The intervention particularly impacted on CCT
practice where the mentor and CCT developed a trusting and supporting relationship and the CCT
wanted to learn and change. This happened in most colleges with most CCTs.
The good practices collected by UNICEF from the mentors which have been compiled into a booklet
illustrate the richness of the experience and the ideas that have emanated from the time that the
mentors have spent working across the fifteen colleges with the 313 CCTs in 352 school clusters. They
show that what the CCTs and school-based educators are committing to sustaining are important
innovations which have impact on classrooms, schools and the education system. Such interventions
included:
Literacy improvements in early grades leading to teachers seeing benefits of improved learner
ability to engage with content in all subjects.
Use of drama, the ‘Learning Arch’, pair and group work and innovative ways of approaching
special needs education – to name a few – to enhance learning in workshops and classrooms.
Innovative projects to enhance community involvement in schools and the involvement of the
community in improving the nutrition of learners.
Building capacity of all stakeholders including head-teachers to do school support supervision
using standardised instrumentation effectively and positively.
Assisting teachers with breaking down the thematic curriculum into teachable topics and
building the teachers’ ability to adapt and develop appropriate instructional materials.
Building capacity of schools to undertake cooperative learning with child-centred
methodology and building of management capacity.
Networking and building capacity of the various stakeholders (CCTs, DEOs, DIS/MIS and
municipal education officials) to support schools while planning and reviewing together, so
that they can share approaches and perceptions as well as providing mutual support.
Impacting on pre-service training of new teachers so that there is a constant supply of new
teachers coming into schools with child-centred ideas and improved teaching skills.
While the project managed to recruit strong local and national educators it is interesting to note that
the foreign mentors tended to focus on the classroom and school and few of them impacted on the
systems within which schools work, while the local mentors tended to focus on these and the
community involvement in schools and undertook much less work in classrooms and schools. Clearly
a combination of both approaches is what is needed. This also goes for the innovations. On their own
they are not transformative – it is the combination of these innovations with advancing child-centred
practices which provided real impact.
54
However, the Good Practices booklet while exuding good ideas and innovation, appears to graphically
illustrate a weakness of the project’s approach. It has been argued by some stakeholders that the
model as implemented was too dependent on individual mentors and their interests and passions,
which although filtered by the BRMS pillars, the needs assessment process and the college
management’s priorities, shine through all aspects of this project and the booklet. Where the mentor
was innovative, built trust with the college authorities, had a good working relationship with the CCTs,
and found receptive head-teachers in the target schools then the model appeared to work well. For a
model which UNICEF intends to replicate there appears to be too much reliance on personality and
assumptions. It could also though be argued from the findings in this report that while the
interventions promoted by the mentors, with their CCTs, were specific to particular sites and
environments there was an overall unifying strategy and that a specific model of CCT and school
support and development emerged. The use of the guiding BRMS pillars provided an overall focus on
child-centred and cooperative learning while the additional focus on developing safe child-friendly
schools reinforce that unifying theme.
While the commitment and engagement by most of the CCTs in the project was apparent this was not
the same with the CPTC principals and DPOs. Their engagement with the project varied hugely. While
about half the principals were supportive and interested the other half were often obstructive, or
largely indifferent. Few of the DPOs actively engaged with the project even though in the design they
were the counterparts for the mentor and therefore meant to be the key to sustaining the project.
They will not do this. Most mentors saw this as a design fault.
This evaluation found the management of the project and the financial systems were not as strong as
required for a project of this complexity and cost, and which also serves as a flagship project. The
financial and management processes were often open to interpretation. Every mentor responded that
the key challenge they faced was related to their responsibilities in relation to the project finances. A
project of this nature requires a number of detailed and very specific procedural manuals covering the
project management, and the management of project resources and finances. The mentors, DPOs and
college bursars need to be trained to use these manuals and made aware of their responsibilities,
which if possible, they should be contractually bound to perform. This did not happen. As a result the
DPOs in most colleges were able to avoid playing the role of counterpart to the mentor, with
implications for the sustainability of the project at college level. The project required the
implementation of these guidelines and auditing of the project books at college level regularly. This
did not happen. Perhaps not surprisingly given the lack of such systems and limited checks and
balances disputation over the project funds and assets were common between some of the college
management teams and their mentor.
A weakness of the first phase of the project has been the limited impact it has had on national thinking
and policy. The management of the project was slow to network the mentors with each other and
with similar or complementary education projects. The networking of mentors improved during the
project and possibly UNICEF was correct in implementing such meetings later in the two year
placements to avoid distracting the mentors. However, more networking and overt engagement in
the products of the mentors’ work at an earlier stage by UNICEF might have led to a stronger profile
for the project with MoES and nationally, with other projects being informed by the approach being
used.
The danger of the present gap in the delivery is that the impact will be further dissipated as other
projects with funds attached are likely to come along and take the attention of the CCT before the
lessons and practices from this project are lodged fully in the system. The MoES is seen to compound
this problem with constant flow of workshops and demands on the CCTs as the main way of accessing
55
schools for new policies and practices. CCTs do not complain as they supplement their incomes
through these workshops but they negatively impact on the ability of the CCTs to sustain input and
improvements into their schools. UNICEF could assist MoES manage the impact on the system of their
demands.
The evaluation concludes that there are elements of an important model contained within the BRMS
Mentorship Project. These have deep relevance to Uganda and could be transferred with obvious
modifications to other countries. This is discussed below in the section on recommendations.
11. Recommendations It could be argued that this evaluation is being conducted 8 months too late. The main activities on
the project at present represent ad hoc extensions granted to specific colleges and mentors. The
recommendations for delivery over the next two years would have been different if the mentors were
still in place as the focus would have been on handover and making sure that the interventions and
underlying thinking was shared with the next level of designated project coordinators in the colleges.
However, given that most of the mentors have departed and the colleges have had 6 months without
the presence of the mentor the recommendations focus on how to take the process forward
independent of most of the original mentors.
The evaluation explored the possibilities of the intervention and the model being sustained where it
has been implemented as well as replicated in other districts. During the field visits and interviews
with CCTs, mentors and college staff as well as the UNICEF staff, MoES managers and funders, this was
explored. Consideration has also been given to replication of the model by UNICEF in other developing
countries.
11.1 Recommendations for the Next Phase in Uganda In the Ugandan context it appears that the future of the project is very much tied up with the future
of the CCTs. The next phase should go on working with and through CCTs, with certain reservations.
While they form the natural link between the schools and the education system and have a focus on
pedagogical support and so are in a perfect position to impact on teaching and learning in schools,
most of the present cadre of CCTs are nearing retirement. Investing considerable sums in their
transformation may not be the best use of funds, unless there is a commitment by MoES to recruit
more CCTs before the present CCTs retire so that the present cadre can train them. This evaluation
has shown that a motivated, better resourced and mentored cadre of CCTs can make a significant and
positive impact on schools. Therefore, adequate funding needs to be secured for both the
continuation of the UNICEF input and the necessary improvements in the resourcing of the CCTs.
While this report has noted that too many of the interventions were dependent on the interests of
individual mentors the report finds that beneath the variety there are strong commonalities. The
approach used in each site was much the same creating a unified intervention model. One possible
way forward is to pull the various strands into a unifying strategy, which seemed to have been best
conceptualised by one of the mentors under the title ‘Cooperative Learning’. This approach would
need to be adapted for Ugandan schools. However, it captures the foundations needed for the
implementation of the good practices that UNICEF collected from the mentors. This needs to be
informed by the observation that the most successful interventions have been ones with systemic
impact. This means that they have three features: they support national policy; they build capacity in
56
state institutions with a strong and clear role for state employees; and they impact at school and
classroom level.
A combination of these elements with an overarching strategy would have most impact. It is the
weaving together of these various strands that provides strength and sustainability. Having literacy
improvement without a school-wide focus on implementing child-centred methods and putting in
place strong systems of support supervision is unlikely to be sustained. Equally, making the school
child-centred without introducing strong interventions which advance reading skills may not lead to
real improvements in student learning. Ultimately, no changes are likely to be sustained if they are
not understood and supported by the DPO, the CCTs, and the district and municipal level education
officials.
Underpinning this is the need to tighten up on the central administration of the project. However
unified the approach and well implemented if the funding processes are not tight and transparent
tensions will appear. The management of the project requires tight, standardised and documented
protocols for access and use of the project vehicle and funds.
If the model is to continue to be implemented in CPTCs then the college management is key. The
management needs to be properly prepared to manage the project and an effective counterpart is
needed to co-ordinate the project in each college. This should not be the DPO but could be a senior
CCT who is relieved of other duties. The link between the project and pre-service training of teachers
must be made much more strongly. As one DIS said “the [child centred learning and literacy]
programme needs to be part of the PTC preset course”. It is significant that VSO, one of the partners
in this project, is implementing a pre-service training intervention triggered by their experience with
this project. The outer two years of the UNICEF project need to link with this VSO project. Indeed it
needs to link more closely with a number of similar initiatives impacting on schools, and particularly
those which, like the UNICEF model, work with CCTs and PTCs.
Lastly, the second phase of the project needs to see substantial funds used for the measurement of
impact and a stronger advocacy process to ensure that the project’s impact informs thinking in the
MoES and in other NGOs.
This report therefore recommends that for the next two years UNICEF (Uganda) implements a two
strand process.
Recommendation 1:
Continue to support the 15 CPTCs which were included in Phase 1 in line with the BRMS project, which
means:
Employment of two national mentors with expertise in curriculum and materials
development as well as a track record in co-ordinating stakeholder engagement with
schools and employment of one of the former international mentors to collaborate with
the national mentors on a rotational basis across the 15 CPTCs. These three mentors
would:
o Conduct workshops in CPTCs and schools to spread the good practice identified in
phase 1, and to deepen understanding and management of the thematic curriculum
at school level;
o Support the CCTs to gradually extend the project elements to all the schools the CCTs
in the 15 CPTCs serve;
57
o Provide funds to ensure that the CCTs in these colleges can get to schools and attend
workshops and meetings, supplementing their government fuel allowance;
o Model school support supervision involving CCTs and DISs and other
district/municipal education officials involving co-evaluation and supervision of
schools, head teachers and teachers. Head teacher performance instruments need to
be added to the CCT/DIS toolkit during this phase;
Meanwhile MoES and UNICEF could advocate across government around the value of
CCTs and the impact of mentorship in education.
The aim of this recommendation is to ensure that the good work that has so far been implemented is
reinforced and supported but at minimal cost, while being spread to all the schools in the 73 districts
covered by the 15 CPTCs.
Recommendation 2:
Implement a smaller scale one year intervention in the 8 CPTCs originally not included in the
intervention, with a strong research component. This intervention would involve:
The employment of one national mentor in each college collaborating with some experienced
international mentors delivering a similar model as in phase 1, with each national mentor
working with all CCTs attached to the CPTC;
A central thrust or strand binding the delivery. This would use the four BRMS themes
encapsulated in a focus on child-centred learning. Each mentor and CCT team could agree on
their specific projects within that focus;
An explicit thrust in feeding back the methodology into the pre-service training of the
teachers, using many of the methods and instruments contained in the Good Practices booklet
A senior CCT appointed in each college as project coordinator with a project steering
committee established in each college which would include the principal, the DPO, the local
and foreign mentor, a DEO and DIS/MIS, and the MoES and/or UNICEF;
An appraisal system which holds mentors accountable for the achievement of agreed
measurable outcomes at CPTC and school levels – with the use of a credible learner
assessment tool such as UWEZO or EGRA;
Regular liaison meetings with district education authorities to ensure co-planning and support
to schools;
A set of procedural manuals used in the colleges to inform the management of the project
and the use and accounting for funds and resources including the project vehicle;
Regular audits of the project funds at college level;
Regular exchange and networking meetings between the 3 colleges, the other 15 CPTCs, other
similar projects, and the MoES;
A strong and properly funded strategy for evaluation (including a baseline survey setting the
metrics and a process evaluation after one year), documentation and advocacy. Impact on
literacy should be measured by use of EGRA testing at the start and end of the year in sampled
schools.
This recommendation would require commitment from the three CPTCs and by the MoES and local
authorities:
o To fill all CCT posts in the three colleges which are vacant during the first six months
of phase 2;
o To provide extra funds to maintain all the existing motorbikes in the colleges;
58
o To increase the fuel allocation for each CCT involved in the project to Ugx 120,000 per
month for the duration of the project so the impact on the intensity of school visits
can be monitored;
o To limit transfers of teachers and head-teachers in the three regions as much as
possible for the two years;
o To serve on the project steering committees at both national and college level.
Such a hothouse extension of the project should prove whether a relatively low cost mentorship
model with a predominantly local cadre of mentors can impact in a sustainable way, as seems likely,
on schools and the practice of the CCTs.
11.2 Recommendations for Replication of the Mentorship Model in other Countries The hardest aspect of envisaging how this project would be replicated in other countries is working
out who the mentors would work with in the education system. It is unusual for education systems to
have a cadre of state-funded senior educationists based in schools and teacher training colleges with
a professional role of supporting teaching and learning. Without that CCT cadre the model will be
difficult to implement. This report suggests that UNICEF focuses on further testing the model in
colleges in Uganda as suggested above. Once that is fully documented with the use of a baseline survey
to set the project metrics and an evaluation to gather data to attempt to prove causation and
attribution, then UNICEF should consider how to replicate the model elsewhere.
In the meantime this report recommends that where UNICEF has influence on national governments
it presents the CCT concept and advocates the adoption of such a system in their education systems
with the support of the Ugandan MoES. This may have a double impact: replicating an important
innovation in countries where it could have real impact on teaching and learning; and convincing the
Uganda government that this is a critical and uniquely Ugandan innovation and a key element in the
education system which requires adequate funding to be sustained and have full impact.
UNICEF could also advocate through its networks that a combination of local and foreign mentors with
deep skills working in concert in education systems can make real impact. The foreign mentors could
provide support across a number of local mentors or a strong networking process could be created.
This would allow the foreign mentors to benefit from local mentors understanding of the politics and
the potential for teaming while the foreign mentors add experience of using cooperative child-centred
methodology and of working positively with teachers and learners as well as providing specific skills
such as developing early grade reading and use of group work.
59
References Carspecken, P.F. (1995). Critical Ethnography in Educational Research: A Theoretical and Practical
Guide. New York: Routledge.
Elmore, R.F. (1980). Backward Mapping: Implementation of Research and Policy Decisions. Political
Science Quarterly, 94(4).
Uganda Ministry of Education and Sports (2010). BRMS Implementation and Quality Improvement in
Primary Schools through CCT and DIS/MIS Mentoring/Coaching. DES and ITIED: Kampala
UNICEF (2011) Project Summary: To Improve the Quality of Education, especially of Teaching and
Learning, in all Primary Schools Attached to 12 Selected Core Primary Teachers’ Colleges.
Project Proposal to the Embassy of the Kingdom of the Netherlands. Authored by Margo
O’Sullivan. Unpublished.
Vulliamy, G., Lewin, K. & Stephens, D. (1990). Doing Educational Research in Developing Countries:
Qualitative Strategies. London: Falmer Press.
60
Annex 1: Time Lapse Analysis of Attendance at Focus Schools
Attendance for the first two weeks of March 2012 and 2014
School A 2012 week1 boys
Week1 girls
Week2 Boys
Week2 girls
2014 Week1 boys
Week1 girls
Week2 boys
Week2 girls
Monday
392
379
450
410
500
502
490
420
Tuesday
305
380
452
408
500
480
480
430
Wednesday
309
381
400
390
450
490
400
410
Thursday
400
405
398
400
502
501
492
482
Friday
401
400
403
407
490
420
490
480
School B 2012 week1 boys
Week1 girls
Week2 Boys
Week2 girls
2014 Week1 boys
Week1 girls
Week2 boys
Week2 girls
Monday
353
331
336
342
317
327
326
324
Tuesday
345
353
313
340
320
320
323
332
Wednesday
355
347
331
349
323
297
313
342
Thursday
329
341
Women’s day
318
304
318
329
Friday
319
321
303
318
318
302
303
311
School C 2012 week1 boys
Week1 girls
Week2 boys
Week2 girls
2014 Week1 boys
Week1 girls
Week2 boys
Week2 girls
Monday
259
271
323
341
393
376
405
391
Tuesday
314
311
333
335
395
394
408
405
Wednesday
322
330
370
358
402
399
409
403
Thursday
325
339
338
340
404
405
408
388
Friday
303
332
308
319
400
366
367
394
61
School D 2012 week1 boys
Week1 girls
Week2 boys
Week2 girls
2014 Week1 boys
Week1 girls
Week2 boys
Week2 girls
Monday
191
224
221
235
211
212
221
236
Tuesday
197
224
224
239
202
216
214
214
Wednesday
218
226
214
235
187
184
215
199
Thursday
210
216
221
239
219
226
197
210
Friday
199
208
217
239
219
225
206
220
School E 2012 week1 boys
Week1 girls
Week2 boys
Week2 girls
2014 Week1 boys
Week1 girls
Week2 boys
Week2 girls
Monday
245
261
281
300
162
187
243
238
Tuesday
276
294
285
312
217
217
256
248
Wednesday
287
306
294
298
219
227
253
249
Thursday
307
262
291
291
221
224
260
248
Friday
285
293
298
296
224
215
248
218
School F 2012 week1 boys
Week1 girls
Week2 boys
Week2 girls
2014 Week1 boys
Week1 girls
Week2 boys
Week2 girls
Monday
104
108
155
175
152
151
186
146
Tuesday
94
174
200
223
154
165
175
156
Wednesday
143
140
196
221
140
146
142
156
Thursday
135
141
207
214
167
165
160
128
Friday
137
144
181
186
152
134
147
148
62
School G 2012 week1 boys
Week1 girls
Week2 boys
Week2 girls
2014 Week1 boys
Week1 girls
Week2 boys
Week2 girls
Monday
370
362
375
376
421
449
431
455
Tuesday
344
370
365
381
417
438
438
457
Wednesday
369
348
364
385
399
427
417
438
Thursday
366
337
355
291
413
441
397
432
Friday
360
326
372
370
446
460
398
437
School H 2012 week1 boys
Week1 girls
Week2 boys
Week2 girls
2014 Week1 boys
Week1 girls
Week2 boys
Week2 girls
Monday
1006
930
977
900
974
933
984
942
Tuesday
1001
928
997
919
974
931
992
949
Wednesday
998
920
998
921
982
944
994
931
Thursday
992
912
1009
936
990
942
992
945
Friday
1002
910
1000
924
993
935
979
934
School I 2012 week1 boys
Week1 girls
Week2 boys
Week2 girls
2014 Week1 boys
Week1 girls
Week2 boys
Week2 girls
Monday
432
388
439
401
339
293
320
271
Tuesday
463
410
470
396
342
300
331
278
Wednesday
424
385
433
391
334
301
335
272
Thursday
431
386
421
391
330
315
330
284
Friday
446
394
417
403
321
298
328
271
63
Annex 2: Interview Tools
Interview Questions for Mentors on UNICEF’s BRMS Project
Position:
Length of time in position:
Former employment and experience:
Gender:
Approximate age:
--------------------------------------------------
1. Please tell me what role you have played, where and for how long on the BRMS Project
2. Can you tell me how you were recruited?
3. When recruited and inducted by VSO what were you told you would be doing?
4. How did the project vary from what you were told?
5. What do you feel were the main strengths of the (i) project design; (ii) project
implementation; (iii) project management and organisation.
6. What do you feel were the main weaknesses of the (i) project design; (ii) project
implementation; (iii) project management and organisation.
7. What could UNICEF have done differently to make the whole project more effective/
what would you do differently if you were in charge of this project?
8. What would you say have been the main achievements of the project?
9. It seems the relationships between mentor and CCTs/DPOs were particularly critical.
Do you agree? Describe how they played out.
10. What examples of good practice were you involved with developing/implementing?
11. (if not covered) What are the particular challenges of being a mentor in the context of
this project? Is it really a mentoring role? How much was it coaching and how much
mentoring?
12. What have you learned from this project?
13. What do you think will be sustained and what will not be sustained do you think?
14. What could UNICEF do to increase the chances of sustaining the inputs? What should
UNICEF do with CCTs and colleges?
15. Is there anything else that you think an evaluator of this project should look at? What
would you measure if you were doing the evaluation?
64
Interview Tool for CCTs
Length of time as CCT:
Gender:
Approximate age:
-----------------------------------------------
Please tell me how you became part of the BRMS mentorship programme?
Can you describe the role your mentor played during the project?
Can you describe exactly what the mentor did with you?
Can you describe how regularly you worked together?
Do you think you spent enough time with the mentor?
Has the way you do your work changed at all as a result of the project? If so can you tell me
exactly how it has changed?
What challenges did the mentor face in working in the college and local schools?
What challenges did you face in working with the mentor?
The mentor was here for 2 years. Do you think that this is the right length of time for such a
placement – or should it be longer/shorter? Why?
Can you describe the innovations or changes which the mentor introduced in schools and the
college?
Did you see any changes in the behaviour, skills and professionalism of your head teachers as
a result of the mentorship programme?
Do you think that any of the innovations you have described will continue/will be
implemented after another two years? Why/why not?
If I was to use two measures/indicators to prove the impact of the mentor on you, the college
and your schools what would they be? (these must be measurable)
Do you think that the other CCTs the mentor worked with in the college would share your
views on working with the mentor? If so why/why not?
If the mentorship model is used elsewhere what do you think are the critical lessons that
should be learned from this programme (possibly prompt with transport, access to funds, use
of foreign mentor, number of CCTs per mentor, nature of work programme, timeframes)
As you look back over the time that the mentor was here with you what are your main feelings
about the work you did together?
65
Interview Tool for DPOs
Length of time as DPO:
Gender:
Approximate age:
--------------------------------------------------
Please describe for me the role that you played in relation to the BRMS programme mentor.
Were you involved in planning this project and your role in it with the college? UNICEF? The
MoES? If so how and with what results?
How would you describe the relations between you and the mentor?
What challenges did the mentor face in working in the college and local schools?
What challenges did you face in working with the mentor?
Were there any issues that arose in relation to the funding of the project and the mentor’s
work that you can share with me?
The mentor was here for 2 years. Do you think that this is the right length of time for such a
placement – or should it be longer/shorter? Why?
Can you describe the innovations or changes which the mentor introduced in schools and the
college?
Do you think that any of the innovations you have described will continue/will be
implemented after another two years? Why/why not?
If I was to use two measures/indicators to prove the impact of the mentor on you and the
college what would they be? (these must be measurable)
Do you think that the other CCTs the mentor worked with in the college would share your
views on working with the mentor? If so why/why not?
If the mentorship model is used elsewhere what do you think are the critical lessons that
should be learned from this programme (possibly prompt with transport, access to funds, use
of foreign mentor, number of CCTs per mentor, nature of work programme, timeframes)
As you look back over the time that the mentor was here with you what are your main feelings
about the work you did together?
Interview Tool for School-based Personnel
66
Position:
Length of time in position:
Gender:
Approximate age:
-----------------------------------------------
Please describe the work that the mentor and the CCTs did with you while the mentor was at
the local PTC.
How was the mentor introduced to you? Did you understand what her/his role would be?
Were your expectations of the work that the mentor would do with you achieved?
Can you describe how the mentor worked with your CCT?
What challenges did the mentor face in working with you?
What challenges did you face in working with the mentor?
The mentor was here for 2 years. Do you think that this is the right length of time for such a
placement – or should it be longer/shorter? Why?
Can you describe the innovations or changes which the mentor introduced in your school and
in other schools in this sub-county? Are these good changes?
(For teacher focus group) Have you seen any changes in the way that your head teacher works
and engages with you and the SMC as a result of the BRMS project/work of the mentor and
CCT? What are those changes?
Do you think that any of the innovations/changes you have described will continue/will be
implemented after another two years? Why/why not?
If I was to use two measures/indicators to prove the impact of the mentor on your school
what would they be? (these must be measurable)
If the mentorship model is used elsewhere what do you think are the critical lessons that
should be learned from this programme (possibly prompt with transport, use of foreign
mentor, number of CCTs/schools per mentor, nature of work programme, timeframes)
As you look back over the time that the mentor was here working with you what are your
main feelings about the work you did together?
67
Interview Tool for Stakeholders
Position:
Length of time in position:
Gender:
Approximate age:
---------------------------------------------------
Can you describe in detail your role in relation to the BRMS mentorship programme?
What are your main impressions of the programme?
Can you tell me what the thinking was behind using mentors in such a way?
What challenges were you aware of that the mentors faced in working in the college and
schools?
The mentor was here for 2 years. Do you think that this is the right length of time for such a
placement – or should it be longer/shorter? Why?
Can you describe the innovations or changes which the mentor introduced in schools and the
colleges where they were based?
Do you think that any of the innovations you have described will continue/will be
implemented after another two years? Why/why not?
If the mentorship model is used elsewhere what do you think are the critical lessons that
should be learned from this programme (possibly prompt with transport, access to funds, use
of foreign mentor, number of CCTs per mentor, nature of work programme, timeframes)
As you look back over the time that the mentor was here with you what are your main feelings
about the work they did?
On reflection do you think that the mentorship approach is more or less likely to bring about
improvement in schools and the way that CCTs support schools than other approaches to
improving schools?
Do you think that the mentorship approach shows good value for money? If so why/why not?
Would you fund/sponsor another mentorship programme in education after your
experiences with this one?
68
Annex 3: Online Questionnaire for the Mentors
As part of the evaluation of the BRMS Mentorship Programme UNICEF has commissioned a formative
review. It is hoped that this will help us better understand the impact of using mentors to improve
schools and the work of CCTs and adopt elements of emerging good practice in other projects. You
are asked to complete the enclosed questionnaire as fully and openly as you can. Your responses will
be consolidated and analysed with those of the other mentors and no names will be used in the report.
Please fill in the top portion and then provide details to the questions below:
Present Job: ……………………………………..
Job before being BRMS mentor: ………………………………………….
Age: ……………………..
1. Can you describe in detail how you heard of the BRMS Mentorship Programme and how you
were recruited and inducted onto it?
2. Can you describe any differences between what you understood the role to be and what you
actually ended up doing in the job as mentor?
3. Describe in detail the roles you played in your college and with the schools
4. What would you say were the greatest successes you had as a mentor?
5. Please describe any challenges you faced in that role and how you resolved them (if you did)
6. What advice would you give UNICEF on improving the project and its impact?
7. Are there elements of the work you did in your college and with your schools which you think
will be/could be mainstreamed and sustained? What are they and why do you think these
elements will be/could be sustained?
8. On reflection what are your main feeling and thoughts about the time you spent as a mentor
in Uganda?
Thank you for filling in this questionnaire and sharing your reflections.